2023-12-01T21:53:13.591 INFO:root:teuthology version: 0.0.1.dev214+gf2de108 2023-12-01T21:53:13.592 DEBUG:teuthology.run:Teuthology command: teuthology --description rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} --archive /home/teuthworker/archive/yuriw-2023-12-01_16:53:58-rados-wip-yuri8-testing-2023-11-29-0706-pacific-distro-default-smithi/7474315 --verbose --name yuriw-2023-12-01_16:53:58-rados-wip-yuri8-testing-2023-11-29-0706-pacific-distro-default-smithi --owner scheduled_yuriw@teuthology -- /home/teuthworker/archive/yuriw-2023-12-01_16:53:58-rados-wip-yuri8-testing-2023-11-29-0706-pacific-distro-default-smithi/7474315/orig.config.yaml 2023-12-01T21:53:13.644 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-01T21:53:13.737 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-12-01_16:53:58-rados-wip-yuri8-testing-2023-11-29-0706-pacific-distro-default-smithi/7474315 branch: wip-yuri8-testing-2023-11-29-0706-pacific description: rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} email: null first_in_suite: false job_id: '7474315' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'setup ceph/pacific v16.2.4 ' name: yuriw-2023-12-01_16:53:58-rados-wip-yuri8-testing-2023-11-29-0706-pacific-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-yuri8-testing-2023-11-29-0706-pacific ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: mon pg warn min per osd: 0 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding sha1: 0dc074ce0b0b3b4f393f56dd32ef36884ebf45a9 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 0dc074ce0b0b3b4f393f56dd32ef36884ebf45a9 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri8-testing-2023-11-29-0706-pacific sha1: 0dc074ce0b0b3b4f393f56dd32ef36884ebf45a9 owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - host.a - client.0 - osd.0 - osd.1 - osd.2 - - host.b - client.1 - osd.3 - osd.4 - osd.5 seed: 8528 sha1: 0dc074ce0b0b3b4f393f56dd32ef36884ebf45a9 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri8-testing-2023-11-29-0706-pacific suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_0dc074ce0b0b3b4f393f56dd32ef36884ebf45a9/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 0dc074ce0b0b3b4f393f56dd32ef36884ebf45a9 targets: smithi029.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE/Gu7jodpkoe5qWBuiwaFjNwhjul3l3KPVahRuQC55JMjyXx24DHCmCGW02tLw76baQ3mep7UA/wI5eV8bv24M= smithi073.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDvGhdhX/gmwksEZ0RDbX9usvX71/SkQPZMnaOpgQuILm1Bcrctk4U7dn2bknhIe8bsWP/eN604nGENK8O4gMZ8= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - pexec: clients: - sudo modprobe -r ceph - sudo modprobe ceph disable_send_metrics=on - install: exclude_packages: - ceph-volume tag: v16.2.4 - print: '**** done install task...' - cephadm: avoid_pacific_features: true cephadm_branch: v16.2.4 cephadm_git_url: https://github.com/ceph/ceph image: docker.io/ceph/ceph:v16.2.4 roleless: true - print: '**** done starting v16.2.4' - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: host.a: - ceph fs volume create cephfs --placement=4 - ceph fs dump - cephadm.shell: host.a: - ceph fs set cephfs max_mds 1 - cephadm.shell: host.a: - ceph fs set cephfs allow_standby_replay true - cephadm.shell: host.a: - ceph fs set cephfs inline_data true --yes-i-really-really-mean-it - cephadm.shell: host.a: - ceph fs dump - ceph --format=json fs dump | jq -e ".filesystems | length == 1" - while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done - fs.pre_upgrade_save: null - kclient: null - print: '**** done client' - parallel: - upgrade-tasks - workload-tasks - cephadm.shell: host.a: - ceph fs dump - fs.post_upgrade_checks: null teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: f2de108beb0a6dde7f3e6c96039071828a1f6269 timestamp: 2023-12-01_16:53:58 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.232157 workload-tasks: sequential: - workunit: clients: all: - suites/fsstress.sh 2023-12-01T21:53:13.737 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_0dc074ce0b0b3b4f393f56dd32ef36884ebf45a9/qa; will attempt to use it 2023-12-01T21:53:13.738 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_0dc074ce0b0b3b4f393f56dd32ef36884ebf45a9/qa/tasks 2023-12-01T21:53:13.738 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-12-01T21:53:13.740 INFO:teuthology.task.internal:Checking packages... 2023-12-01T21:53:13.774 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '0dc074ce0b0b3b4f393f56dd32ef36884ebf45a9' 2023-12-01T21:53:13.775 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-12-01T21:53:13.775 INFO:teuthology.packaging:ref: None 2023-12-01T21:53:13.775 INFO:teuthology.packaging:tag: None 2023-12-01T21:53:13.775 INFO:teuthology.packaging:branch: wip-yuri8-testing-2023-11-29-0706-pacific 2023-12-01T21:53:13.775 INFO:teuthology.packaging:sha1: 0dc074ce0b0b3b4f393f56dd32ef36884ebf45a9 2023-12-01T21:53:13.775 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri8-testing-2023-11-29-0706-pacific 2023-12-01T21:53:13.997 INFO:teuthology.task.internal:Found packages for ceph version 16.2.14-328.g0dc074ce 2023-12-01T21:53:13.997 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-12-01T21:53:14.004 INFO:teuthology.task.internal:no buildpackages task found 2023-12-01T21:53:14.004 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-12-01T21:53:14.011 INFO:teuthology.task.internal:Saving configuration 2023-12-01T21:53:14.030 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-12-01T21:53:14.034 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-01T21:53:14.060 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi029.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-01_16:53:58-rados-wip-yuri8-testing-2023-11-29-0706-pacific-distro-default-smithi/7474315', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-01 21:47:57.719678', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE/Gu7jodpkoe5qWBuiwaFjNwhjul3l3KPVahRuQC55JMjyXx24DHCmCGW02tLw76baQ3mep7UA/wI5eV8bv24M='} 2023-12-01T21:53:14.081 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi073.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-01_16:53:58-rados-wip-yuri8-testing-2023-11-29-0706-pacific-distro-default-smithi/7474315', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-01 21:47:57.721648', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDvGhdhX/gmwksEZ0RDbX9usvX71/SkQPZMnaOpgQuILm1Bcrctk4U7dn2bknhIe8bsWP/eN604nGENK8O4gMZ8='} 2023-12-01T21:53:14.081 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-12-01T21:53:14.085 INFO:teuthology.task.internal:roles: ubuntu@smithi029.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2023-12-01T21:53:14.085 INFO:teuthology.task.internal:roles: ubuntu@smithi073.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2023-12-01T21:53:14.085 INFO:teuthology.run_tasks:Running task console_log... 2023-12-01T21:53:14.149 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f5b5774ba60>, signals=[15]) 2023-12-01T21:53:14.150 INFO:teuthology.run_tasks:Running task internal.connect... 2023-12-01T21:53:14.157 INFO:teuthology.task.internal:Opening connections... 2023-12-01T21:53:14.157 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi029.front.sepia.ceph.com 2023-12-01T21:53:14.159 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-01T21:53:14.227 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi073.front.sepia.ceph.com 2023-12-01T21:53:14.227 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi073.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-01T21:53:14.296 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-12-01T21:53:14.303 DEBUG:teuthology.orchestra.run.smithi029:> uname -m 2023-12-01T21:53:14.327 INFO:teuthology.orchestra.run.smithi029.stdout:x86_64 2023-12-01T21:53:14.328 DEBUG:teuthology.orchestra.run.smithi029:> cat /etc/os-release 2023-12-01T21:53:14.391 INFO:teuthology.orchestra.run.smithi029.stdout:NAME="CentOS Stream" 2023-12-01T21:53:14.392 INFO:teuthology.orchestra.run.smithi029.stdout:VERSION="8" 2023-12-01T21:53:14.392 INFO:teuthology.orchestra.run.smithi029.stdout:ID="centos" 2023-12-01T21:53:14.392 INFO:teuthology.orchestra.run.smithi029.stdout:ID_LIKE="rhel fedora" 2023-12-01T21:53:14.392 INFO:teuthology.orchestra.run.smithi029.stdout:VERSION_ID="8" 2023-12-01T21:53:14.392 INFO:teuthology.orchestra.run.smithi029.stdout:PLATFORM_ID="platform:el8" 2023-12-01T21:53:14.392 INFO:teuthology.orchestra.run.smithi029.stdout:PRETTY_NAME="CentOS Stream 8" 2023-12-01T21:53:14.392 INFO:teuthology.orchestra.run.smithi029.stdout:ANSI_COLOR="0;31" 2023-12-01T21:53:14.392 INFO:teuthology.orchestra.run.smithi029.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-12-01T21:53:14.392 INFO:teuthology.orchestra.run.smithi029.stdout:HOME_URL="https://centos.org/" 2023-12-01T21:53:14.393 INFO:teuthology.orchestra.run.smithi029.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-12-01T21:53:14.393 INFO:teuthology.orchestra.run.smithi029.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-12-01T21:53:14.393 INFO:teuthology.orchestra.run.smithi029.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-12-01T21:53:14.394 INFO:teuthology.lock.ops:Updating smithi029.front.sepia.ceph.com on lock server 2023-12-01T21:53:14.420 DEBUG:teuthology.orchestra.run.smithi073:> uname -m 2023-12-01T21:53:14.443 INFO:teuthology.orchestra.run.smithi073.stdout:x86_64 2023-12-01T21:53:14.443 DEBUG:teuthology.orchestra.run.smithi073:> cat /etc/os-release 2023-12-01T21:53:14.504 INFO:teuthology.orchestra.run.smithi073.stdout:NAME="CentOS Stream" 2023-12-01T21:53:14.504 INFO:teuthology.orchestra.run.smithi073.stdout:VERSION="8" 2023-12-01T21:53:14.505 INFO:teuthology.orchestra.run.smithi073.stdout:ID="centos" 2023-12-01T21:53:14.505 INFO:teuthology.orchestra.run.smithi073.stdout:ID_LIKE="rhel fedora" 2023-12-01T21:53:14.505 INFO:teuthology.orchestra.run.smithi073.stdout:VERSION_ID="8" 2023-12-01T21:53:14.505 INFO:teuthology.orchestra.run.smithi073.stdout:PLATFORM_ID="platform:el8" 2023-12-01T21:53:14.505 INFO:teuthology.orchestra.run.smithi073.stdout:PRETTY_NAME="CentOS Stream 8" 2023-12-01T21:53:14.505 INFO:teuthology.orchestra.run.smithi073.stdout:ANSI_COLOR="0;31" 2023-12-01T21:53:14.505 INFO:teuthology.orchestra.run.smithi073.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-12-01T21:53:14.505 INFO:teuthology.orchestra.run.smithi073.stdout:HOME_URL="https://centos.org/" 2023-12-01T21:53:14.505 INFO:teuthology.orchestra.run.smithi073.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-12-01T21:53:14.505 INFO:teuthology.orchestra.run.smithi073.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-12-01T21:53:14.505 INFO:teuthology.orchestra.run.smithi073.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-12-01T21:53:14.505 INFO:teuthology.lock.ops:Updating smithi073.front.sepia.ceph.com on lock server 2023-12-01T21:53:14.528 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-12-01T21:53:14.538 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-12-01T21:53:14.543 INFO:teuthology.task.internal:Checking for old test directory... 2023-12-01T21:53:14.544 DEBUG:teuthology.orchestra.run.smithi029:> test '!' -e /home/ubuntu/cephtest 2023-12-01T21:53:14.546 DEBUG:teuthology.orchestra.run.smithi073:> test '!' -e /home/ubuntu/cephtest 2023-12-01T21:53:14.567 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-12-01T21:53:14.605 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-12-01T21:53:14.606 DEBUG:teuthology.orchestra.run.smithi029:> test -z $(ls -A /var/lib/ceph) 2023-12-01T21:53:14.608 DEBUG:teuthology.orchestra.run.smithi073:> test -z $(ls -A /var/lib/ceph) 2023-12-01T21:53:14.639 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-12-01T21:53:14.693 INFO:teuthology.run_tasks:Running task kernel... 2023-12-01T21:53:14.707 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-12-01T21:53:14.707 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro'}, 'host.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-12-01T21:53:14.707 DEBUG:teuthology.orchestra.run.smithi029:> test -f /run/.containerenv -o -f /.dockerenv 2023-12-01T21:53:14.708 DEBUG:teuthology.orchestra.run.smithi073:> test -f /run/.containerenv -o -f /.dockerenv 2023-12-01T21:53:14.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-01T21:53:14.729 DEBUG:teuthology.orchestra.run.smithi073:> uname -r 2023-12-01T21:53:14.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-01T21:53:14.730 DEBUG:teuthology.orchestra.run.smithi029:> uname -r 2023-12-01T21:53:14.790 INFO:teuthology.orchestra.run.smithi073.stdout:4.18.0-489.el8.x86_64 2023-12-01T21:53:14.790 INFO:teuthology.task.kernel:Running kernel on smithi073: 4.18.0-489.el8.x86_64 2023-12-01T21:53:14.790 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum install -y kernel 2023-12-01T21:53:14.793 INFO:teuthology.orchestra.run.smithi029.stdout:4.18.0-489.el8.x86_64 2023-12-01T21:53:14.793 INFO:teuthology.task.kernel:Running kernel on smithi029: 4.18.0-489.el8.x86_64 2023-12-01T21:53:14.793 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum install -y kernel 2023-12-01T21:53:19.862 INFO:teuthology.orchestra.run.smithi073.stdout:CentOS Stream 8 - AppStream 13 MB/s | 34 MB 00:02 2023-12-01T21:53:21.248 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 8 - AppStream 16 MB/s | 34 MB 00:02 2023-12-01T21:53:37.363 INFO:teuthology.orchestra.run.smithi073.stdout:CentOS Stream 8 - BaseOS 4.9 MB/s | 55 MB 00:11 2023-12-01T21:53:43.489 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 8 - BaseOS 3.6 MB/s | 55 MB 00:15 2023-12-01T21:53:45.379 INFO:teuthology.orchestra.run.smithi073.stdout:CentOS Stream 8 - Extras 85 kB/s | 18 kB 00:00 2023-12-01T21:53:45.971 INFO:teuthology.orchestra.run.smithi073.stdout:CentOS Stream 8 - Extras common packages 17 kB/s | 6.9 kB 00:00 2023-12-01T21:53:46.265 INFO:teuthology.orchestra.run.smithi073.stdout:Copr repo for python3-asyncssh owned by ceph 24 kB/s | 3.5 kB 00:00 2023-12-01T21:53:52.151 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 8 - Extras 81 kB/s | 18 kB 00:00 2023-12-01T21:53:52.719 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 8 - Extras common packages 18 kB/s | 6.9 kB 00:00 2023-12-01T21:53:53.038 INFO:teuthology.orchestra.run.smithi029.stdout:Copr repo for python3-asyncssh owned by ceph 22 kB/s | 3.5 kB 00:00 2023-12-01T21:53:53.975 INFO:teuthology.orchestra.run.smithi029.stdout:Extra Packages for Enterprise Linux 21 MB/s | 16 MB 00:00 2023-12-01T21:53:58.073 INFO:teuthology.orchestra.run.smithi029.stdout:lab-extras 196 kB/s | 24 kB 00:00 2023-12-01T21:53:58.531 INFO:teuthology.orchestra.run.smithi073.stdout:Extra Packages for Enterprise Linux 1.3 MB/s | 16 MB 00:12 2023-12-01T21:54:02.010 INFO:teuthology.orchestra.run.smithi029.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2023-12-01T21:54:02.010 INFO:teuthology.orchestra.run.smithi029.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2023-12-01T21:54:02.332 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-12-01T21:54:02.333 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-12-01T21:54:02.333 INFO:teuthology.orchestra.run.smithi029.stdout: Package Architecture Version Repository Size 2023-12-01T21:54:02.333 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-12-01T21:54:02.333 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2023-12-01T21:54:02.333 INFO:teuthology.orchestra.run.smithi029.stdout: kernel x86_64 4.18.0-526.el8 baseos 10 M 2023-12-01T21:54:02.333 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2023-12-01T21:54:02.333 INFO:teuthology.orchestra.run.smithi029.stdout: kernel-core x86_64 4.18.0-526.el8 baseos 43 M 2023-12-01T21:54:02.333 INFO:teuthology.orchestra.run.smithi029.stdout: kernel-modules x86_64 4.18.0-526.el8 baseos 36 M 2023-12-01T21:54:02.334 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-01T21:54:02.334 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2023-12-01T21:54:02.334 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-12-01T21:54:02.334 INFO:teuthology.orchestra.run.smithi029.stdout:Install 3 Packages 2023-12-01T21:54:02.334 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-01T21:54:02.334 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 90 M 2023-12-01T21:54:02.335 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 96 M 2023-12-01T21:54:02.335 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2023-12-01T21:54:02.339 INFO:teuthology.orchestra.run.smithi073.stdout:lab-extras 497 kB/s | 24 kB 00:00 2023-12-01T21:54:05.938 INFO:teuthology.orchestra.run.smithi073.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2023-12-01T21:54:05.938 INFO:teuthology.orchestra.run.smithi073.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2023-12-01T21:54:06.237 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2023-12-01T21:54:06.238 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2023-12-01T21:54:06.238 INFO:teuthology.orchestra.run.smithi073.stdout: Package Architecture Version Repository Size 2023-12-01T21:54:06.238 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2023-12-01T21:54:06.238 INFO:teuthology.orchestra.run.smithi073.stdout:Installing: 2023-12-01T21:54:06.238 INFO:teuthology.orchestra.run.smithi073.stdout: kernel x86_64 4.18.0-526.el8 baseos 10 M 2023-12-01T21:54:06.238 INFO:teuthology.orchestra.run.smithi073.stdout:Installing dependencies: 2023-12-01T21:54:06.238 INFO:teuthology.orchestra.run.smithi073.stdout: kernel-core x86_64 4.18.0-526.el8 baseos 43 M 2023-12-01T21:54:06.238 INFO:teuthology.orchestra.run.smithi073.stdout: kernel-modules x86_64 4.18.0-526.el8 baseos 36 M 2023-12-01T21:54:06.238 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T21:54:06.238 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction Summary 2023-12-01T21:54:06.238 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2023-12-01T21:54:06.238 INFO:teuthology.orchestra.run.smithi073.stdout:Install 3 Packages 2023-12-01T21:54:06.238 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T21:54:06.239 INFO:teuthology.orchestra.run.smithi073.stdout:Total download size: 90 M 2023-12-01T21:54:06.239 INFO:teuthology.orchestra.run.smithi073.stdout:Installed size: 96 M 2023-12-01T21:54:06.239 INFO:teuthology.orchestra.run.smithi073.stdout:Downloading Packages: 2023-12-01T21:54:09.370 INFO:teuthology.orchestra.run.smithi029.stdout:(1/3): kernel-4.18.0-526.el8.x86_64.rpm 1.7 MB/s | 10 MB 00:05 2023-12-01T21:54:16.281 INFO:teuthology.orchestra.run.smithi073.stdout:(1/3): kernel-4.18.0-526.el8.x86_64.rpm 1.1 MB/s | 10 MB 00:08 2023-12-01T21:54:16.421 INFO:teuthology.orchestra.run.smithi029.stdout:(2/3): kernel-modules-4.18.0-526.el8.x86_64.rpm 2.8 MB/s | 36 MB 00:12 2023-12-01T21:54:17.805 INFO:teuthology.orchestra.run.smithi029.stdout:(3/3): kernel-core-4.18.0-526.el8.x86_64.rpm 3.0 MB/s | 43 MB 00:14 2023-12-01T21:54:17.805 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2023-12-01T21:54:17.805 INFO:teuthology.orchestra.run.smithi029.stdout:Total 5.8 MB/s | 90 MB 00:15 2023-12-01T21:54:18.516 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2023-12-01T21:54:19.002 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2023-12-01T21:54:19.002 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2023-12-01T21:54:19.947 INFO:teuthology.orchestra.run.smithi073.stdout:(2/3): kernel-core-4.18.0-526.el8.x86_64.rpm 3.5 MB/s | 43 MB 00:12 2023-12-01T21:54:20.273 INFO:teuthology.orchestra.run.smithi073.stdout:(3/3): kernel-modules-4.18.0-526.el8.x86_64.rpm 2.8 MB/s | 36 MB 00:12 2023-12-01T21:54:20.273 INFO:teuthology.orchestra.run.smithi073.stdout:-------------------------------------------------------------------------------- 2023-12-01T21:54:20.273 INFO:teuthology.orchestra.run.smithi073.stdout:Total 6.4 MB/s | 90 MB 00:14 2023-12-01T21:54:20.291 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2023-12-01T21:54:20.317 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2023-12-01T21:54:20.954 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction check 2023-12-01T21:54:21.380 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction check succeeded. 2023-12-01T21:54:21.380 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction test 2023-12-01T21:54:22.612 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction test succeeded. 2023-12-01T21:54:22.635 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction 2023-12-01T21:54:23.267 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2023-12-01T21:54:24.270 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : kernel-core-4.18.0-526.el8.x86_64 1/3 2023-12-01T21:54:25.431 INFO:teuthology.orchestra.run.smithi073.stdout: Preparing : 1/1 2023-12-01T21:54:25.911 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : kernel-core-4.18.0-526.el8.x86_64 1/3 2023-12-01T21:54:28.016 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: kernel-core-4.18.0-526.el8.x86_64 1/3 2023-12-01T21:54:28.551 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : kernel-modules-4.18.0-526.el8.x86_64 2/3 2023-12-01T21:54:29.487 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: kernel-core-4.18.0-526.el8.x86_64 1/3 2023-12-01T21:54:29.860 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : kernel-modules-4.18.0-526.el8.x86_64 2/3 2023-12-01T21:54:32.228 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: kernel-modules-4.18.0-526.el8.x86_64 2/3 2023-12-01T21:54:32.569 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : kernel-4.18.0-526.el8.x86_64 3/3 2023-12-01T21:54:33.284 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: kernel-modules-4.18.0-526.el8.x86_64 2/3 2023-12-01T21:54:33.624 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : kernel-4.18.0-526.el8.x86_64 3/3 2023-12-01T21:54:55.493 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: kernel-core-4.18.0-526.el8.x86_64 3/3 2023-12-01T21:54:55.515 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: kernel-modules-4.18.0-526.el8.x86_64 3/3 2023-12-01T21:54:55.867 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: kernel-4.18.0-526.el8.x86_64 3/3 2023-12-01T21:54:55.867 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : kernel-4.18.0-526.el8.x86_64 1/3 2023-12-01T21:54:55.867 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : kernel-core-4.18.0-526.el8.x86_64 2/3 2023-12-01T21:54:56.630 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : kernel-modules-4.18.0-526.el8.x86_64 3/3 2023-12-01T21:54:56.631 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T21:54:56.631 INFO:teuthology.orchestra.run.smithi073.stdout:Installed: 2023-12-01T21:54:56.631 INFO:teuthology.orchestra.run.smithi073.stdout: kernel-4.18.0-526.el8.x86_64 kernel-core-4.18.0-526.el8.x86_64 2023-12-01T21:54:56.631 INFO:teuthology.orchestra.run.smithi073.stdout: kernel-modules-4.18.0-526.el8.x86_64 2023-12-01T21:54:56.631 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T21:54:56.631 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2023-12-01T21:54:56.775 DEBUG:teuthology.orchestra.run.smithi073:> rpm -q kernel | sort -rV | head -n 1 2023-12-01T21:54:56.989 INFO:teuthology.orchestra.run.smithi073.stdout:kernel-4.18.0-526.el8.x86_64 2023-12-01T21:54:56.989 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-526.el8.x86_64 2023-12-01T21:54:56.989 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-526.el8.x86_64 2023-12-01T21:54:56.989 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-12-01T21:54:56.989 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-12-01T21:54:56.989 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2023-12-01T21:54:56.989 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi073.front.sepia.ceph.com, path=None, version=distro) 2023-12-01T21:54:56.989 DEBUG:teuthology.orchestra.run.smithi073:> rpm -q kernel | sort -rV | head -n 1 2023-12-01T21:54:57.197 INFO:teuthology.orchestra.run.smithi073.stdout:kernel-4.18.0-526.el8.x86_64 2023-12-01T21:54:57.197 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-526.el8.x86_64 2023-12-01T21:54:57.197 DEBUG:teuthology.orchestra.run.smithi073:> sudo rpm -qi grub2-tools 2023-12-01T21:54:57.231 INFO:teuthology.orchestra.run.smithi073.stdout:Name : grub2-tools 2023-12-01T21:54:57.231 INFO:teuthology.orchestra.run.smithi073.stdout:Epoch : 1 2023-12-01T21:54:57.231 INFO:teuthology.orchestra.run.smithi073.stdout:Version : 2.02 2023-12-01T21:54:57.231 INFO:teuthology.orchestra.run.smithi073.stdout:Release : 129.el8 2023-12-01T21:54:57.231 INFO:teuthology.orchestra.run.smithi073.stdout:Architecture: x86_64 2023-12-01T21:54:57.231 INFO:teuthology.orchestra.run.smithi073.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2023-12-01T21:54:57.231 INFO:teuthology.orchestra.run.smithi073.stdout:Group : System Environment/Base 2023-12-01T21:54:57.231 INFO:teuthology.orchestra.run.smithi073.stdout:Size : 9265586 2023-12-01T21:54:57.232 INFO:teuthology.orchestra.run.smithi073.stdout:License : GPLv3+ 2023-12-01T21:54:57.232 INFO:teuthology.orchestra.run.smithi073.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-12-01T21:54:57.232 INFO:teuthology.orchestra.run.smithi073.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-12-01T21:54:57.232 INFO:teuthology.orchestra.run.smithi073.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-12-01T21:54:57.232 INFO:teuthology.orchestra.run.smithi073.stdout:Build Host : kbuilder.bsys.centos.org 2023-12-01T21:54:57.232 INFO:teuthology.orchestra.run.smithi073.stdout:Relocations : (not relocatable) 2023-12-01T21:54:57.232 INFO:teuthology.orchestra.run.smithi073.stdout:Packager : CentOS BuildSystem 2023-12-01T21:54:57.232 INFO:teuthology.orchestra.run.smithi073.stdout:Vendor : CentOS 2023-12-01T21:54:57.232 INFO:teuthology.orchestra.run.smithi073.stdout:URL : http://www.gnu.org/software/grub/ 2023-12-01T21:54:57.232 INFO:teuthology.orchestra.run.smithi073.stdout:Summary : Support tools for GRUB. 2023-12-01T21:54:57.232 INFO:teuthology.orchestra.run.smithi073.stdout:Description : 2023-12-01T21:54:57.232 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T21:54:57.232 INFO:teuthology.orchestra.run.smithi073.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-12-01T21:54:57.232 INFO:teuthology.orchestra.run.smithi073.stdout:customizable bootloader with modular architecture. It supports a rich 2023-12-01T21:54:57.232 INFO:teuthology.orchestra.run.smithi073.stdout:variety of kernel formats, file systems, computer architectures and 2023-12-01T21:54:57.233 INFO:teuthology.orchestra.run.smithi073.stdout:hardware devices. 2023-12-01T21:54:57.233 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T21:54:57.233 INFO:teuthology.orchestra.run.smithi073.stdout:This subpackage provides tools for support of all platforms. 2023-12-01T21:54:57.234 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-12-01T21:54:57.234 INFO:teuthology.task.kernel:Updating grub on smithi073 to boot 4.18.0-526.el8.x86_64 2023-12-01T21:54:57.234 DEBUG:teuthology.orchestra.run.smithi073:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-12-01T21:54:57.338 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: kernel-core-4.18.0-526.el8.x86_64 3/3 2023-12-01T21:54:57.363 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: kernel-modules-4.18.0-526.el8.x86_64 3/3 2023-12-01T21:54:57.738 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: kernel-4.18.0-526.el8.x86_64 3/3 2023-12-01T21:54:57.738 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : kernel-4.18.0-526.el8.x86_64 1/3 2023-12-01T21:54:57.738 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : kernel-core-4.18.0-526.el8.x86_64 2/3 2023-12-01T21:54:58.578 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : kernel-modules-4.18.0-526.el8.x86_64 3/3 2023-12-01T21:54:58.578 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-01T21:54:58.578 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2023-12-01T21:54:58.578 INFO:teuthology.orchestra.run.smithi029.stdout: kernel-4.18.0-526.el8.x86_64 kernel-core-4.18.0-526.el8.x86_64 2023-12-01T21:54:58.578 INFO:teuthology.orchestra.run.smithi029.stdout: kernel-modules-4.18.0-526.el8.x86_64 2023-12-01T21:54:58.578 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-01T21:54:58.578 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-12-01T21:54:58.758 DEBUG:teuthology.orchestra.run.smithi029:> rpm -q kernel | sort -rV | head -n 1 2023-12-01T21:54:58.937 INFO:teuthology.orchestra.run.smithi073.stderr:Generating grub configuration file ... 2023-12-01T21:54:58.985 INFO:teuthology.orchestra.run.smithi029.stdout:kernel-4.18.0-526.el8.x86_64 2023-12-01T21:54:58.986 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-526.el8.x86_64 2023-12-01T21:54:58.986 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-526.el8.x86_64 2023-12-01T21:54:58.986 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-12-01T21:54:58.986 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-12-01T21:54:58.986 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2023-12-01T21:54:58.986 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi029.front.sepia.ceph.com, path=None, version=distro) 2023-12-01T21:54:58.986 DEBUG:teuthology.orchestra.run.smithi029:> rpm -q kernel | sort -rV | head -n 1 2023-12-01T21:54:59.205 INFO:teuthology.orchestra.run.smithi029.stdout:kernel-4.18.0-526.el8.x86_64 2023-12-01T21:54:59.205 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-526.el8.x86_64 2023-12-01T21:54:59.206 DEBUG:teuthology.orchestra.run.smithi029:> sudo rpm -qi grub2-tools 2023-12-01T21:54:59.239 INFO:teuthology.orchestra.run.smithi029.stdout:Name : grub2-tools 2023-12-01T21:54:59.240 INFO:teuthology.orchestra.run.smithi029.stdout:Epoch : 1 2023-12-01T21:54:59.240 INFO:teuthology.orchestra.run.smithi029.stdout:Version : 2.02 2023-12-01T21:54:59.240 INFO:teuthology.orchestra.run.smithi029.stdout:Release : 129.el8 2023-12-01T21:54:59.240 INFO:teuthology.orchestra.run.smithi029.stdout:Architecture: x86_64 2023-12-01T21:54:59.240 INFO:teuthology.orchestra.run.smithi029.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2023-12-01T21:54:59.240 INFO:teuthology.orchestra.run.smithi029.stdout:Group : System Environment/Base 2023-12-01T21:54:59.240 INFO:teuthology.orchestra.run.smithi029.stdout:Size : 9265586 2023-12-01T21:54:59.240 INFO:teuthology.orchestra.run.smithi029.stdout:License : GPLv3+ 2023-12-01T21:54:59.240 INFO:teuthology.orchestra.run.smithi029.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-12-01T21:54:59.240 INFO:teuthology.orchestra.run.smithi029.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-12-01T21:54:59.240 INFO:teuthology.orchestra.run.smithi029.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-12-01T21:54:59.240 INFO:teuthology.orchestra.run.smithi029.stdout:Build Host : kbuilder.bsys.centos.org 2023-12-01T21:54:59.240 INFO:teuthology.orchestra.run.smithi029.stdout:Relocations : (not relocatable) 2023-12-01T21:54:59.240 INFO:teuthology.orchestra.run.smithi029.stdout:Packager : CentOS BuildSystem 2023-12-01T21:54:59.241 INFO:teuthology.orchestra.run.smithi029.stdout:Vendor : CentOS 2023-12-01T21:54:59.241 INFO:teuthology.orchestra.run.smithi029.stdout:URL : http://www.gnu.org/software/grub/ 2023-12-01T21:54:59.241 INFO:teuthology.orchestra.run.smithi029.stdout:Summary : Support tools for GRUB. 2023-12-01T21:54:59.241 INFO:teuthology.orchestra.run.smithi029.stdout:Description : 2023-12-01T21:54:59.241 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-01T21:54:59.241 INFO:teuthology.orchestra.run.smithi029.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-12-01T21:54:59.241 INFO:teuthology.orchestra.run.smithi029.stdout:customizable bootloader with modular architecture. It supports a rich 2023-12-01T21:54:59.241 INFO:teuthology.orchestra.run.smithi029.stdout:variety of kernel formats, file systems, computer architectures and 2023-12-01T21:54:59.241 INFO:teuthology.orchestra.run.smithi029.stdout:hardware devices. 2023-12-01T21:54:59.241 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-01T21:54:59.241 INFO:teuthology.orchestra.run.smithi029.stdout:This subpackage provides tools for support of all platforms. 2023-12-01T21:54:59.242 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-12-01T21:54:59.242 INFO:teuthology.task.kernel:Updating grub on smithi029 to boot 4.18.0-526.el8.x86_64 2023-12-01T21:54:59.242 DEBUG:teuthology.orchestra.run.smithi029:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-12-01T21:55:01.040 INFO:teuthology.orchestra.run.smithi073.stderr:done 2023-12-01T21:55:01.041 DEBUG:teuthology.orchestra.run.smithi073:> mktemp 2023-12-01T21:55:01.075 INFO:teuthology.orchestra.run.smithi073.stdout:/tmp/tmp.HxOgK9XhxS 2023-12-01T21:55:01.075 DEBUG:teuthology.orchestra.run.smithi073:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.HxOgK9XhxS 2023-12-01T21:55:01.148 DEBUG:teuthology.orchestra.run.smithi073:> sudo chmod 0666 /tmp/tmp.HxOgK9XhxS 2023-12-01T21:55:01.176 INFO:teuthology.orchestra.run.smithi029.stderr:Generating grub configuration file ... 2023-12-01T21:55:01.288 DEBUG:teuthology.orchestra.remote:smithi073:/tmp/tmp.HxOgK9XhxS is 6KB 2023-12-01T21:55:01.337 DEBUG:teuthology.orchestra.run.smithi073:> rm -fr /tmp/tmp.HxOgK9XhxS 2023-12-01T21:55:01.354 DEBUG:teuthology.orchestra.run.smithi073:> sudo /bin/ls /boot/loader/entries || true 2023-12-01T21:55:01.419 INFO:teuthology.orchestra.run.smithi073.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2023-12-01T21:55:01.420 INFO:teuthology.orchestra.run.smithi073.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2023-12-01T21:55:01.420 INFO:teuthology.orchestra.run.smithi073.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2023-12-01T21:55:01.420 INFO:teuthology.orchestra.run.smithi073.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-526.el8.x86_64.conf 2023-12-01T21:55:01.421 DEBUG:teuthology.orchestra.run.smithi073:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-526.el8.x86_64 2023-12-01T21:55:01.574 DEBUG:teuthology.orchestra.run.smithi073:> sudo shutdown -r now 2023-12-01T21:55:03.125 INFO:teuthology.orchestra.run.smithi029.stderr:done 2023-12-01T21:55:03.127 DEBUG:teuthology.orchestra.run.smithi029:> mktemp 2023-12-01T21:55:03.186 INFO:teuthology.orchestra.run.smithi029.stdout:/tmp/tmp.h7m8cFHz4o 2023-12-01T21:55:03.186 DEBUG:teuthology.orchestra.run.smithi029:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.h7m8cFHz4o 2023-12-01T21:55:03.225 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 0666 /tmp/tmp.h7m8cFHz4o 2023-12-01T21:55:03.370 DEBUG:teuthology.orchestra.remote:smithi029:/tmp/tmp.h7m8cFHz4o is 6KB 2023-12-01T21:55:03.420 DEBUG:teuthology.orchestra.run.smithi029:> rm -fr /tmp/tmp.h7m8cFHz4o 2023-12-01T21:55:03.438 DEBUG:teuthology.orchestra.run.smithi029:> sudo /bin/ls /boot/loader/entries || true 2023-12-01T21:55:03.511 INFO:teuthology.orchestra.run.smithi029.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2023-12-01T21:55:03.511 INFO:teuthology.orchestra.run.smithi029.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2023-12-01T21:55:03.511 INFO:teuthology.orchestra.run.smithi029.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2023-12-01T21:55:03.511 INFO:teuthology.orchestra.run.smithi029.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-526.el8.x86_64.conf 2023-12-01T21:55:03.512 DEBUG:teuthology.orchestra.run.smithi029:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-526.el8.x86_64 2023-12-01T21:55:03.682 DEBUG:teuthology.orchestra.run.smithi029:> sudo shutdown -r now 2023-12-01T21:55:31.603 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2023-12-01T21:55:31.604 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi073.front.sepia.ceph.com' 2023-12-01T21:55:31.604 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi073.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-01T21:55:33.686 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2023-12-01T21:55:33.686 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi029.front.sepia.ceph.com' 2023-12-01T21:55:33.687 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-01T21:56:31.662 DEBUG:teuthology.orchestra.remote:timed out 2023-12-01T21:56:33.690 DEBUG:teuthology.orchestra.remote:timed out 2023-12-01T21:56:40.670 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi073.front.sepia.ceph.com' 2023-12-01T21:56:40.671 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi073.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-01T21:56:41.099 DEBUG:teuthology.orchestra.run.smithi073:> true 2023-12-01T21:56:41.324 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi073.front.sepia.ceph.com' 2023-12-01T21:56:41.325 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "4.18.0-526.el8.x86_64"... 2023-12-01T21:56:41.325 DEBUG:teuthology.orchestra.run.smithi073:> uname -r 2023-12-01T21:56:41.342 INFO:teuthology.orchestra.run.smithi073.stdout:4.18.0-526.el8.x86_64 2023-12-01T21:56:41.342 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-526.el8.x86_64 vs 4.18.0-526.el8.x86_64 2023-12-01T21:56:41.342 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-12-01T21:56:41.342 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-12-01T21:56:42.343 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2023-12-01T21:56:42.344 DEBUG:teuthology.orchestra.run.smithi073:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-12-01T21:56:42.404 INFO:teuthology.orchestra.run.smithi073.stdout:ttyS1 2023-12-01T21:56:42.415 DEBUG:teuthology.parallel:result is None 2023-12-01T21:56:42.691 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi029.front.sepia.ceph.com' 2023-12-01T21:56:42.692 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-01T21:56:44.097 DEBUG:teuthology.orchestra.run.smithi029:> true 2023-12-01T21:56:44.869 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi029.front.sepia.ceph.com' 2023-12-01T21:56:44.870 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "4.18.0-526.el8.x86_64"... 2023-12-01T21:56:44.870 DEBUG:teuthology.orchestra.run.smithi029:> uname -r 2023-12-01T21:56:44.891 INFO:teuthology.orchestra.run.smithi029.stdout:4.18.0-526.el8.x86_64 2023-12-01T21:56:44.891 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-526.el8.x86_64 vs 4.18.0-526.el8.x86_64 2023-12-01T21:56:44.891 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-12-01T21:56:44.891 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-12-01T21:56:45.892 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2023-12-01T21:56:45.893 DEBUG:teuthology.orchestra.run.smithi029:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-12-01T21:56:46.017 INFO:teuthology.orchestra.run.smithi029.stdout:ttyS1 2023-12-01T21:56:46.029 DEBUG:teuthology.parallel:result is None 2023-12-01T21:56:46.030 INFO:teuthology.run_tasks:Running task internal.base... 2023-12-01T21:56:46.037 INFO:teuthology.task.internal:Creating test directory... 2023-12-01T21:56:46.037 DEBUG:teuthology.orchestra.run.smithi029:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-12-01T21:56:46.039 DEBUG:teuthology.orchestra.run.smithi073:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-12-01T21:56:46.264 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-12-01T21:56:46.271 INFO:teuthology.run_tasks:Running task internal.archive... 2023-12-01T21:56:46.277 INFO:teuthology.task.internal:Creating archive directory... 2023-12-01T21:56:46.277 DEBUG:teuthology.orchestra.run.smithi029:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-12-01T21:56:46.280 DEBUG:teuthology.orchestra.run.smithi073:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-12-01T21:56:46.784 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-12-01T21:56:46.791 INFO:teuthology.task.internal:Enabling coredump saving... 2023-12-01T21:56:46.791 DEBUG:teuthology.orchestra.run.smithi029:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-12-01T21:56:46.794 DEBUG:teuthology.orchestra.run.smithi073:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-12-01T21:56:46.837 INFO:teuthology.orchestra.run.smithi029.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-01T21:56:46.849 INFO:teuthology.orchestra.run.smithi029.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-01T21:56:47.222 INFO:teuthology.orchestra.run.smithi073.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-01T21:56:47.244 INFO:teuthology.orchestra.run.smithi073.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-01T21:56:47.256 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-12-01T21:56:47.263 INFO:teuthology.task.internal:Configuring sudo... 2023-12-01T21:56:47.263 DEBUG:teuthology.orchestra.run.smithi029:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-12-01T21:56:47.266 DEBUG:teuthology.orchestra.run.smithi073:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-12-01T21:56:47.531 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-12-01T21:56:47.541 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-12-01T21:56:47.541 DEBUG:teuthology.orchestra.run.smithi029:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-12-01T21:56:47.543 DEBUG:teuthology.orchestra.run.smithi073:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-12-01T21:56:47.562 DEBUG:teuthology.orchestra.run.smithi029:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-01T21:56:47.652 DEBUG:teuthology.orchestra.run.smithi029:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-01T21:56:47.711 DEBUG:teuthology.orchestra.run.smithi029:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-01T21:56:47.759 DEBUG:teuthology.orchestra.run.smithi029:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-01T21:56:47.805 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-12-01T21:56:47.806 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-12-01T21:56:47.894 DEBUG:teuthology.orchestra.run.smithi073:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-01T21:56:47.951 DEBUG:teuthology.orchestra.run.smithi073:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-01T21:56:48.016 DEBUG:teuthology.orchestra.run.smithi073:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-01T21:56:48.059 DEBUG:teuthology.orchestra.run.smithi073:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-01T21:56:48.101 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2023-12-01T21:56:48.101 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-12-01T21:56:48.168 DEBUG:teuthology.orchestra.run.smithi029:> sudo service rsyslog restart 2023-12-01T21:56:48.171 DEBUG:teuthology.orchestra.run.smithi073:> sudo service rsyslog restart 2023-12-01T21:56:48.247 INFO:teuthology.orchestra.run.smithi029.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-01T21:56:48.272 INFO:teuthology.orchestra.run.smithi073.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-01T21:56:48.757 INFO:teuthology.run_tasks:Running task internal.timer... 2023-12-01T21:56:48.765 INFO:teuthology.task.internal:Starting timer... 2023-12-01T21:56:48.765 INFO:teuthology.run_tasks:Running task pcp... 2023-12-01T21:56:48.776 INFO:teuthology.run_tasks:Running task selinux... 2023-12-01T21:56:48.785 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-12-01T21:56:48.785 DEBUG:teuthology.orchestra.run.smithi029:> sudo service auditd rotate 2023-12-01T21:56:48.923 INFO:teuthology.orchestra.run.smithi029.stdout:Rotating logs: 2023-12-01T21:56:48.925 DEBUG:teuthology.orchestra.run.smithi073:> sudo service auditd rotate 2023-12-01T21:56:48.993 INFO:teuthology.orchestra.run.smithi073.stdout:Rotating logs: 2023-12-01T21:56:48.995 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-12-01T21:56:48.995 DEBUG:teuthology.orchestra.run.smithi029:> /usr/sbin/getenforce 2023-12-01T21:56:49.031 INFO:teuthology.orchestra.run.smithi029.stdout:Permissive 2023-12-01T21:56:49.031 DEBUG:teuthology.orchestra.run.smithi073:> /usr/sbin/getenforce 2023-12-01T21:56:49.066 INFO:teuthology.orchestra.run.smithi073.stdout:Permissive 2023-12-01T21:56:49.067 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi029.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi073.front.sepia.ceph.com': 'permissive'} 2023-12-01T21:56:49.067 DEBUG:teuthology.orchestra.run.smithi029:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-01T21:56:49.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-01T21:56:49.109 DEBUG:teuthology.orchestra.run.smithi073:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-01T21:56:49.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-01T21:56:49.137 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-12-01T21:56:49.137 DEBUG:teuthology.orchestra.run.smithi029:> sudo /usr/sbin/setenforce permissive 2023-12-01T21:56:49.189 DEBUG:teuthology.orchestra.run.smithi073:> sudo /usr/sbin/setenforce permissive 2023-12-01T21:56:49.225 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-12-01T21:56:49.235 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-12-01T21:56:49.235 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-12-01T21:56:49.252 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-12-01T21:56:49.253 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi029.front.sepia.ceph.com,smithi073.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-12-01T22:05:14.131 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi029.front.sepia.ceph.com'), Remote(name='ubuntu@smithi073.front.sepia.ceph.com')] 2023-12-01T22:05:14.132 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi029.front.sepia.ceph.com' 2023-12-01T22:05:14.133 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-01T22:05:14.202 DEBUG:teuthology.orchestra.run.smithi029:> true 2023-12-01T22:05:14.290 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi029.front.sepia.ceph.com' 2023-12-01T22:05:14.291 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi073.front.sepia.ceph.com' 2023-12-01T22:05:14.292 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi073.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-01T22:05:14.377 DEBUG:teuthology.orchestra.run.smithi073:> true 2023-12-01T22:05:14.669 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi073.front.sepia.ceph.com' 2023-12-01T22:05:14.669 INFO:teuthology.run_tasks:Running task clock... 2023-12-01T22:05:14.683 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-12-01T22:05:14.683 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-12-01T22:05:14.683 DEBUG:teuthology.orchestra.run.smithi029:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-01T22:05:14.686 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-12-01T22:05:14.686 DEBUG:teuthology.orchestra.run.smithi073:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-01T22:05:14.722 INFO:teuthology.orchestra.run.smithi029.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-12-01T22:05:14.739 INFO:teuthology.orchestra.run.smithi029.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-12-01T22:05:14.745 INFO:teuthology.orchestra.run.smithi073.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-12-01T22:05:14.760 INFO:teuthology.orchestra.run.smithi073.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-12-01T22:05:14.807 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: ntpd: command not found 2023-12-01T22:05:14.821 INFO:teuthology.orchestra.run.smithi029.stdout:506 Cannot talk to daemon 2023-12-01T22:05:14.830 INFO:teuthology.orchestra.run.smithi073.stderr:sudo: ntpd: command not found 2023-12-01T22:05:14.838 INFO:teuthology.orchestra.run.smithi029.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-12-01T22:05:14.841 INFO:teuthology.orchestra.run.smithi073.stdout:506 Cannot talk to daemon 2023-12-01T22:05:14.853 INFO:teuthology.orchestra.run.smithi029.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-12-01T22:05:14.855 INFO:teuthology.orchestra.run.smithi073.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-12-01T22:05:14.869 INFO:teuthology.orchestra.run.smithi073.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-12-01T22:05:14.930 INFO:teuthology.orchestra.run.smithi029.stderr:bash: ntpq: command not found 2023-12-01T22:05:14.933 INFO:teuthology.orchestra.run.smithi029.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-01T22:05:14.933 INFO:teuthology.orchestra.run.smithi029.stdout:=============================================================================== 2023-12-01T22:05:14.934 INFO:teuthology.orchestra.run.smithi029.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-01T22:05:14.934 INFO:teuthology.orchestra.run.smithi029.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-01T22:05:14.934 INFO:teuthology.orchestra.run.smithi029.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-01T22:05:14.934 INFO:teuthology.orchestra.run.smithi029.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-01T22:05:14.942 INFO:teuthology.orchestra.run.smithi073.stderr:bash: ntpq: command not found 2023-12-01T22:05:14.946 INFO:teuthology.orchestra.run.smithi073.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-01T22:05:14.947 INFO:teuthology.orchestra.run.smithi073.stdout:=============================================================================== 2023-12-01T22:05:14.947 INFO:teuthology.orchestra.run.smithi073.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-01T22:05:14.947 INFO:teuthology.orchestra.run.smithi073.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-01T22:05:14.947 INFO:teuthology.orchestra.run.smithi073.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-01T22:05:14.948 INFO:teuthology.orchestra.run.smithi073.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-01T22:05:14.948 INFO:teuthology.run_tasks:Running task pexec... 2023-12-01T22:05:14.958 INFO:teuthology.task.pexec:Executing custom commands... 2023-12-01T22:05:14.958 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi029.front.sepia.ceph.com 2023-12-01T22:05:14.958 DEBUG:teuthology.orchestra.run.smithi029:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-12-01T22:05:14.959 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi073.front.sepia.ceph.com 2023-12-01T22:05:14.959 DEBUG:teuthology.orchestra.run.smithi073:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-12-01T22:05:15.864 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:02:31 ago on Fri 01 Dec 2023 10:02:44 PM UTC. 2023-12-01T22:05:15.995 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:01:45 ago on Fri 01 Dec 2023 10:03:30 PM UTC. 2023-12-01T22:05:17.994 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2023-12-01T22:05:17.995 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2023-12-01T22:05:17.995 INFO:teuthology.orchestra.run.smithi073.stdout: Package Architecture Version Repository Size 2023-12-01T22:05:17.995 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2023-12-01T22:05:17.995 INFO:teuthology.orchestra.run.smithi073.stdout:Resetting modules: 2023-12-01T22:05:17.995 INFO:teuthology.orchestra.run.smithi073.stdout: container-tools 2023-12-01T22:05:17.996 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:05:17.996 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction Summary 2023-12-01T22:05:17.996 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2023-12-01T22:05:17.996 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:05:18.248 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2023-12-01T22:05:18.411 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-12-01T22:05:18.412 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-12-01T22:05:18.412 INFO:teuthology.orchestra.run.smithi029.stdout: Package Architecture Version Repository Size 2023-12-01T22:05:18.413 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-12-01T22:05:18.413 INFO:teuthology.orchestra.run.smithi029.stdout:Resetting modules: 2023-12-01T22:05:18.413 INFO:teuthology.orchestra.run.smithi029.stdout: container-tools 2023-12-01T22:05:18.413 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-01T22:05:18.413 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2023-12-01T22:05:18.413 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-12-01T22:05:18.413 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-01T22:05:18.692 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-12-01T22:05:19.179 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:02:35 ago on Fri 01 Dec 2023 10:02:44 PM UTC. 2023-12-01T22:05:19.797 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:01:49 ago on Fri 01 Dec 2023 10:03:30 PM UTC. 2023-12-01T22:05:21.254 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2023-12-01T22:05:21.438 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2023-12-01T22:05:21.438 INFO:teuthology.orchestra.run.smithi073.stdout: Package Arch Version Repository Size 2023-12-01T22:05:21.438 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2023-12-01T22:05:21.438 INFO:teuthology.orchestra.run.smithi073.stdout:Installing group/module packages: 2023-12-01T22:05:21.438 INFO:teuthology.orchestra.run.smithi073.stdout: aardvark-dns x86_64 2:1.8.0-1.module_el8+712+4cd1bd69 appstream 1.0 M 2023-12-01T22:05:21.438 INFO:teuthology.orchestra.run.smithi073.stdout: buildah x86_64 1:1.32.0-1.module_el8+712+4cd1bd69 appstream 9.6 M 2023-12-01T22:05:21.438 INFO:teuthology.orchestra.run.smithi073.stdout: cockpit-podman noarch 78-1.module_el8+712+4cd1bd69 appstream 750 k 2023-12-01T22:05:21.438 INFO:teuthology.orchestra.run.smithi073.stdout: crun x86_64 1.9.2-1.module_el8+712+4cd1bd69 appstream 241 k 2023-12-01T22:05:21.438 INFO:teuthology.orchestra.run.smithi073.stdout: netavark x86_64 2:1.8.0-2.module_el8+712+4cd1bd69 appstream 3.9 M 2023-12-01T22:05:21.438 INFO:teuthology.orchestra.run.smithi073.stdout: python3-podman noarch 4.7.0-1.module_el8+712+4cd1bd69 appstream 178 k 2023-12-01T22:05:21.438 INFO:teuthology.orchestra.run.smithi073.stdout: skopeo x86_64 2:1.13.3-1.module_el8+712+4cd1bd69 appstream 8.1 M 2023-12-01T22:05:21.438 INFO:teuthology.orchestra.run.smithi073.stdout: toolbox x86_64 0.0.99.4-6.module_el8+712+4cd1bd69 appstream 2.5 M 2023-12-01T22:05:21.439 INFO:teuthology.orchestra.run.smithi073.stdout: udica noarch 0.2.6-20.module_el8+712+4cd1bd69 appstream 53 k 2023-12-01T22:05:21.439 INFO:teuthology.orchestra.run.smithi073.stdout:Installing dependencies: 2023-12-01T22:05:21.439 INFO:teuthology.orchestra.run.smithi073.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2023-12-01T22:05:21.439 INFO:teuthology.orchestra.run.smithi073.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2023-12-01T22:05:21.439 INFO:teuthology.orchestra.run.smithi073.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2023-12-01T22:05:21.439 INFO:teuthology.orchestra.run.smithi073.stdout:Installing weak dependencies: 2023-12-01T22:05:21.439 INFO:teuthology.orchestra.run.smithi073.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2023-12-01T22:05:21.439 INFO:teuthology.orchestra.run.smithi073.stdout:Installing module profiles: 2023-12-01T22:05:21.439 INFO:teuthology.orchestra.run.smithi073.stdout: container-tools/common 2023-12-01T22:05:21.439 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:05:21.439 INFO:teuthology.orchestra.run.smithi073.stdout:Enabling module streams: 2023-12-01T22:05:21.439 INFO:teuthology.orchestra.run.smithi073.stdout: container-tools rhel8 2023-12-01T22:05:21.439 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:05:21.439 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction Summary 2023-12-01T22:05:21.439 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2023-12-01T22:05:21.439 INFO:teuthology.orchestra.run.smithi073.stdout:Install 13 Packages 2023-12-01T22:05:21.440 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:05:21.440 INFO:teuthology.orchestra.run.smithi073.stdout:Total download size: 27 M 2023-12-01T22:05:21.440 INFO:teuthology.orchestra.run.smithi073.stdout:Installed size: 89 M 2023-12-01T22:05:21.440 INFO:teuthology.orchestra.run.smithi073.stdout:Downloading Packages: 2023-12-01T22:05:21.953 INFO:teuthology.orchestra.run.smithi073.stdout:(1/13): aardvark-dns-1.8.0-1.module_el8+712+4cd 3.1 MB/s | 1.0 MB 00:00 2023-12-01T22:05:21.987 INFO:teuthology.orchestra.run.smithi073.stdout:(2/13): cockpit-podman-78-1.module_el8+712+4cd1 2.1 MB/s | 750 kB 00:00 2023-12-01T22:05:22.013 INFO:teuthology.orchestra.run.smithi073.stdout:(3/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 645 kB/s | 38 kB 00:00 2023-12-01T22:05:22.054 INFO:teuthology.orchestra.run.smithi073.stdout:(4/13): crun-1.9.2-1.module_el8+712+4cd1bd69.x8 3.5 MB/s | 241 kB 00:00 2023-12-01T22:05:22.121 INFO:teuthology.orchestra.run.smithi073.stdout:(5/13): python3-podman-4.7.0-1.module_el8+712+4 2.6 MB/s | 178 kB 00:00 2023-12-01T22:05:22.136 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-12-01T22:05:22.180 INFO:teuthology.orchestra.run.smithi073.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 424 kB/s | 25 kB 00:00 2023-12-01T22:05:22.247 INFO:teuthology.orchestra.run.smithi073.stdout:(7/13): python3-pyxdg-0.25-16.el8.noarch.rpm 1.4 MB/s | 94 kB 00:00 2023-12-01T22:05:22.341 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-12-01T22:05:22.341 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2023-12-01T22:05:22.341 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-12-01T22:05:22.341 INFO:teuthology.orchestra.run.smithi029.stdout:Installing group/module packages: 2023-12-01T22:05:22.341 INFO:teuthology.orchestra.run.smithi029.stdout: aardvark-dns x86_64 2:1.8.0-1.module_el8+712+4cd1bd69 appstream 1.0 M 2023-12-01T22:05:22.341 INFO:teuthology.orchestra.run.smithi029.stdout: buildah x86_64 1:1.32.0-1.module_el8+712+4cd1bd69 appstream 9.6 M 2023-12-01T22:05:22.341 INFO:teuthology.orchestra.run.smithi029.stdout: cockpit-podman noarch 78-1.module_el8+712+4cd1bd69 appstream 750 k 2023-12-01T22:05:22.341 INFO:teuthology.orchestra.run.smithi029.stdout: crun x86_64 1.9.2-1.module_el8+712+4cd1bd69 appstream 241 k 2023-12-01T22:05:22.341 INFO:teuthology.orchestra.run.smithi029.stdout: netavark x86_64 2:1.8.0-2.module_el8+712+4cd1bd69 appstream 3.9 M 2023-12-01T22:05:22.341 INFO:teuthology.orchestra.run.smithi029.stdout: python3-podman noarch 4.7.0-1.module_el8+712+4cd1bd69 appstream 178 k 2023-12-01T22:05:22.342 INFO:teuthology.orchestra.run.smithi029.stdout: skopeo x86_64 2:1.13.3-1.module_el8+712+4cd1bd69 appstream 8.1 M 2023-12-01T22:05:22.342 INFO:teuthology.orchestra.run.smithi029.stdout: toolbox x86_64 0.0.99.4-6.module_el8+712+4cd1bd69 appstream 2.5 M 2023-12-01T22:05:22.342 INFO:teuthology.orchestra.run.smithi029.stdout: udica noarch 0.2.6-20.module_el8+712+4cd1bd69 appstream 53 k 2023-12-01T22:05:22.342 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2023-12-01T22:05:22.342 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2023-12-01T22:05:22.342 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2023-12-01T22:05:22.342 INFO:teuthology.orchestra.run.smithi029.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2023-12-01T22:05:22.342 INFO:teuthology.orchestra.run.smithi029.stdout:Installing weak dependencies: 2023-12-01T22:05:22.342 INFO:teuthology.orchestra.run.smithi029.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2023-12-01T22:05:22.342 INFO:teuthology.orchestra.run.smithi029.stdout:Installing module profiles: 2023-12-01T22:05:22.342 INFO:teuthology.orchestra.run.smithi029.stdout: container-tools/common 2023-12-01T22:05:22.342 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-01T22:05:22.342 INFO:teuthology.orchestra.run.smithi029.stdout:Enabling module streams: 2023-12-01T22:05:22.342 INFO:teuthology.orchestra.run.smithi029.stdout: container-tools rhel8 2023-12-01T22:05:22.342 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-01T22:05:22.343 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2023-12-01T22:05:22.343 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-12-01T22:05:22.343 INFO:teuthology.orchestra.run.smithi029.stdout:Install 13 Packages 2023-12-01T22:05:22.343 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-01T22:05:22.343 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 27 M 2023-12-01T22:05:22.343 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 89 M 2023-12-01T22:05:22.343 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2023-12-01T22:05:22.422 INFO:teuthology.orchestra.run.smithi073.stdout:(8/13): netavark-1.8.0-2.module_el8+712+4cd1bd6 9.6 MB/s | 3.9 MB 00:00 2023-12-01T22:05:22.648 INFO:teuthology.orchestra.run.smithi073.stdout:(9/13): toolbox-0.0.99.4-6.module_el8+712+4cd1b 11 MB/s | 2.5 MB 00:00 2023-12-01T22:05:22.732 INFO:teuthology.orchestra.run.smithi073.stdout:(10/13): udica-0.2.6-20.module_el8+712+4cd1bd69 632 kB/s | 53 kB 00:00 2023-12-01T22:05:22.790 INFO:teuthology.orchestra.run.smithi029.stdout:(1/13): cockpit-podman-78-1.module_el8+712+4cd1 3.0 MB/s | 750 kB 00:00 2023-12-01T22:05:22.798 INFO:teuthology.orchestra.run.smithi073.stdout:(11/13): yajl-2.1.0-12.el8.x86_64.rpm 615 kB/s | 41 kB 00:00 2023-12-01T22:05:22.849 INFO:teuthology.orchestra.run.smithi029.stdout:(2/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 647 kB/s | 38 kB 00:00 2023-12-01T22:05:22.924 INFO:teuthology.orchestra.run.smithi029.stdout:(3/13): crun-1.9.2-1.module_el8+712+4cd1bd69.x8 3.1 MB/s | 241 kB 00:00 2023-12-01T22:05:22.968 INFO:teuthology.orchestra.run.smithi029.stdout:(4/13): aardvark-dns-1.8.0-1.module_el8+712+4cd 2.4 MB/s | 1.0 MB 00:00 2023-12-01T22:05:23.045 INFO:teuthology.orchestra.run.smithi029.stdout:(5/13): python3-podman-4.7.0-1.module_el8+712+4 2.3 MB/s | 178 kB 00:00 2023-12-01T22:05:23.103 INFO:teuthology.orchestra.run.smithi029.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 423 kB/s | 25 kB 00:00 2023-12-01T22:05:23.149 INFO:teuthology.orchestra.run.smithi073.stdout:(12/13): skopeo-1.13.3-1.module_el8+712+4cd1bd6 9.0 MB/s | 8.1 MB 00:00 2023-12-01T22:05:23.161 INFO:teuthology.orchestra.run.smithi029.stdout:(7/13): python3-pyxdg-0.25-16.el8.noarch.rpm 1.6 MB/s | 94 kB 00:00 2023-12-01T22:05:23.346 INFO:teuthology.orchestra.run.smithi029.stdout:(8/13): netavark-1.8.0-2.module_el8+712+4cd1bd6 9.3 MB/s | 3.9 MB 00:00 2023-12-01T22:05:23.366 INFO:teuthology.orchestra.run.smithi073.stdout:(13/13): buildah-1.32.0-1.module_el8+712+4cd1bd 5.5 MB/s | 9.6 MB 00:01 2023-12-01T22:05:23.366 INFO:teuthology.orchestra.run.smithi073.stdout:-------------------------------------------------------------------------------- 2023-12-01T22:05:23.366 INFO:teuthology.orchestra.run.smithi073.stdout:Total 14 MB/s | 27 MB 00:01 2023-12-01T22:05:23.554 INFO:teuthology.orchestra.run.smithi029.stdout:(9/13): toolbox-0.0.99.4-6.module_el8+712+4cd1b 12 MB/s | 2.5 MB 00:00 2023-12-01T22:05:23.605 INFO:teuthology.orchestra.run.smithi029.stdout:(10/13): udica-0.2.6-20.module_el8+712+4cd1bd69 1.0 MB/s | 53 kB 00:00 2023-12-01T22:05:23.636 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction check 2023-12-01T22:05:23.657 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction check succeeded. 2023-12-01T22:05:23.657 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction test 2023-12-01T22:05:23.664 INFO:teuthology.orchestra.run.smithi029.stdout:(11/13): yajl-2.1.0-12.el8.x86_64.rpm 701 kB/s | 41 kB 00:00 2023-12-01T22:05:23.915 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction test succeeded. 2023-12-01T22:05:23.919 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction 2023-12-01T22:05:24.248 INFO:teuthology.orchestra.run.smithi029.stdout:(12/13): skopeo-1.13.3-1.module_el8+712+4cd1bd6 7.5 MB/s | 8.1 MB 00:01 2023-12-01T22:05:24.506 INFO:teuthology.orchestra.run.smithi029.stdout:(13/13): buildah-1.32.0-1.module_el8+712+4cd1bd 4.9 MB/s | 9.6 MB 00:01 2023-12-01T22:05:24.507 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2023-12-01T22:05:24.507 INFO:teuthology.orchestra.run.smithi029.stdout:Total 12 MB/s | 27 MB 00:02 2023-12-01T22:05:24.654 INFO:teuthology.orchestra.run.smithi073.stdout: Preparing : 1/1 2023-12-01T22:05:24.802 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2023-12-01T22:05:24.826 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2023-12-01T22:05:24.826 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2023-12-01T22:05:24.913 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2023-12-01T22:05:25.083 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 2/13 2023-12-01T22:05:25.107 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2023-12-01T22:05:25.111 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2023-12-01T22:05:25.292 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/13 2023-12-01T22:05:25.543 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2023-12-01T22:05:25.822 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2023-12-01T22:05:26.061 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2023-12-01T22:05:26.088 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 5/13 2023-12-01T22:05:26.245 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 2/13 2023-12-01T22:05:26.474 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/13 2023-12-01T22:05:26.737 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2023-12-01T22:05:27.061 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2023-12-01T22:05:27.300 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 7/13 2023-12-01T22:05:27.353 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 5/13 2023-12-01T22:05:27.513 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 8/13 2023-12-01T22:05:27.787 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 9/13 2023-12-01T22:05:28.179 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 10/13 2023-12-01T22:05:28.795 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2023-12-01T22:05:29.052 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 7/13 2023-12-01T22:05:29.091 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2023-12-01T22:05:29.283 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 8/13 2023-12-01T22:05:29.313 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 12/13 2023-12-01T22:05:29.558 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 9/13 2023-12-01T22:05:29.765 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2023-12-01T22:05:30.013 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 10/13 2023-12-01T22:05:30.190 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2023-12-01T22:05:30.190 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 1/13 2023-12-01T22:05:30.190 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2/13 2023-12-01T22:05:30.190 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 3/13 2023-12-01T22:05:30.190 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2023-12-01T22:05:30.190 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 5/13 2023-12-01T22:05:30.190 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2023-12-01T22:05:30.190 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 7/13 2023-12-01T22:05:30.190 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2023-12-01T22:05:30.190 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2023-12-01T22:05:30.191 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 10/13 2023-12-01T22:05:30.191 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2023-12-01T22:05:30.191 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 12/13 2023-12-01T22:05:30.975 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2023-12-01T22:05:31.002 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2023-12-01T22:05:31.002 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:05:31.002 INFO:teuthology.orchestra.run.smithi073.stdout:Installed: 2023-12-01T22:05:31.002 INFO:teuthology.orchestra.run.smithi073.stdout: aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86_64 2023-12-01T22:05:31.002 INFO:teuthology.orchestra.run.smithi073.stdout: buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2023-12-01T22:05:31.002 INFO:teuthology.orchestra.run.smithi073.stdout: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 2023-12-01T22:05:31.002 INFO:teuthology.orchestra.run.smithi073.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2023-12-01T22:05:31.002 INFO:teuthology.orchestra.run.smithi073.stdout: crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 2023-12-01T22:05:31.002 INFO:teuthology.orchestra.run.smithi073.stdout: netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 2023-12-01T22:05:31.002 INFO:teuthology.orchestra.run.smithi073.stdout: python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noarch 2023-12-01T22:05:31.002 INFO:teuthology.orchestra.run.smithi073.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-12-01T22:05:31.002 INFO:teuthology.orchestra.run.smithi073.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-12-01T22:05:31.002 INFO:teuthology.orchestra.run.smithi073.stdout: skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 2023-12-01T22:05:31.003 INFO:teuthology.orchestra.run.smithi073.stdout: toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 2023-12-01T22:05:31.003 INFO:teuthology.orchestra.run.smithi073.stdout: udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 2023-12-01T22:05:31.003 INFO:teuthology.orchestra.run.smithi073.stdout: yajl-2.1.0-12.el8.x86_64 2023-12-01T22:05:31.003 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:05:31.003 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2023-12-01T22:05:31.226 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 12/13 2023-12-01T22:05:31.266 DEBUG:teuthology.parallel:result is None 2023-12-01T22:05:31.451 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2023-12-01T22:05:31.931 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2023-12-01T22:05:31.931 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 1/13 2023-12-01T22:05:31.931 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2/13 2023-12-01T22:05:31.931 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 3/13 2023-12-01T22:05:31.931 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2023-12-01T22:05:31.932 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 5/13 2023-12-01T22:05:31.932 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2023-12-01T22:05:31.932 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 7/13 2023-12-01T22:05:31.932 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2023-12-01T22:05:31.932 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2023-12-01T22:05:31.932 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 10/13 2023-12-01T22:05:31.932 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2023-12-01T22:05:31.932 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 12/13 2023-12-01T22:05:32.789 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2023-12-01T22:05:32.789 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-01T22:05:32.789 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2023-12-01T22:05:32.789 INFO:teuthology.orchestra.run.smithi029.stdout: aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86_64 2023-12-01T22:05:32.789 INFO:teuthology.orchestra.run.smithi029.stdout: buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2023-12-01T22:05:32.789 INFO:teuthology.orchestra.run.smithi029.stdout: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 2023-12-01T22:05:32.789 INFO:teuthology.orchestra.run.smithi029.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2023-12-01T22:05:32.789 INFO:teuthology.orchestra.run.smithi029.stdout: crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 2023-12-01T22:05:32.789 INFO:teuthology.orchestra.run.smithi029.stdout: netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 2023-12-01T22:05:32.789 INFO:teuthology.orchestra.run.smithi029.stdout: python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noarch 2023-12-01T22:05:32.789 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-12-01T22:05:32.789 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-12-01T22:05:32.790 INFO:teuthology.orchestra.run.smithi029.stdout: skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 2023-12-01T22:05:32.790 INFO:teuthology.orchestra.run.smithi029.stdout: toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 2023-12-01T22:05:32.790 INFO:teuthology.orchestra.run.smithi029.stdout: udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 2023-12-01T22:05:32.790 INFO:teuthology.orchestra.run.smithi029.stdout: yajl-2.1.0-12.el8.x86_64 2023-12-01T22:05:32.790 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-01T22:05:32.790 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-12-01T22:05:33.052 DEBUG:teuthology.parallel:result is None 2023-12-01T22:05:33.052 INFO:teuthology.run_tasks:Running task pexec... 2023-12-01T22:05:33.061 INFO:teuthology.task.pexec:Executing custom commands... 2023-12-01T22:05:33.061 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi029.front.sepia.ceph.com 2023-12-01T22:05:33.061 DEBUG:teuthology.orchestra.run.smithi029:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-12-01T22:05:33.062 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi073.front.sepia.ceph.com 2023-12-01T22:05:33.062 DEBUG:teuthology.orchestra.run.smithi073:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-12-01T22:05:33.275 DEBUG:teuthology.parallel:result is None 2023-12-01T22:05:33.276 DEBUG:teuthology.parallel:result is None 2023-12-01T22:05:33.276 INFO:teuthology.run_tasks:Running task install... 2023-12-01T22:05:33.284 DEBUG:teuthology.task.install:project ceph 2023-12-01T22:05:33.285 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '0dc074ce0b0b3b4f393f56dd32ef36884ebf45a9'}} 2023-12-01T22:05:33.285 DEBUG:teuthology.task.install:config {'exclude_packages': ['ceph-volume'], 'tag': 'v16.2.4', 'flavor': 'default', 'sha1': '0dc074ce0b0b3b4f393f56dd32ef36884ebf45a9'} 2023-12-01T22:05:33.285 INFO:teuthology.task.install:Using flavor: default 2023-12-01T22:05:33.294 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-12-01T22:05:33.294 INFO:teuthology.task.install:extra packages: [] 2023-12-01T22:05:33.294 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '0dc074ce0b0b3b4f393f56dd32ef36884ebf45a9', 'tag': 'v16.2.4', 'wait_for_package': False} 2023-12-01T22:05:33.295 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2023-12-01T22:05:33.295 INFO:teuthology.packaging:ref: None 2023-12-01T22:05:33.295 INFO:teuthology.packaging:tag: v16.2.4 2023-12-01T22:05:33.295 INFO:teuthology.packaging:branch: None 2023-12-01T22:05:33.295 INFO:teuthology.packaging:sha1: 0dc074ce0b0b3b4f393f56dd32ef36884ebf45a9 2023-12-01T22:05:33.506 DEBUG:teuthology.repo_utils:git ls-remote https://github.com/ceph/ceph-ci v16.2.4^{} -> 3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-12-01T22:05:33.506 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-12-01T22:05:33.509 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '0dc074ce0b0b3b4f393f56dd32ef36884ebf45a9', 'tag': 'v16.2.4', 'wait_for_package': False} 2023-12-01T22:05:33.509 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2023-12-01T22:05:33.509 INFO:teuthology.packaging:ref: None 2023-12-01T22:05:33.509 INFO:teuthology.packaging:tag: v16.2.4 2023-12-01T22:05:33.509 INFO:teuthology.packaging:branch: None 2023-12-01T22:05:33.509 INFO:teuthology.packaging:sha1: 0dc074ce0b0b3b4f393f56dd32ef36884ebf45a9 2023-12-01T22:05:33.722 DEBUG:teuthology.repo_utils:git ls-remote https://github.com/ceph/ceph-ci v16.2.4^{} -> 3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-12-01T22:05:33.723 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-12-01T22:05:33.963 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/ 2023-12-01T22:05:33.963 INFO:teuthology.task.install.rpm:Package version is 16.2.4-0 2023-12-01T22:05:33.968 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/ 2023-12-01T22:05:33.968 INFO:teuthology.task.install.rpm:Package version is 16.2.4-0 2023-12-01T22:05:34.000 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-12-01T22:05:34.001 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-12-01T22:05:34.001 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-12-01T22:05:34.012 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-12-01T22:05:34.012 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2023-12-01T22:05:34.013 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-12-01T22:05:34.037 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-12-01T22:05:34.037 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2023-12-01T22:05:34.037 INFO:teuthology.packaging:ref: None 2023-12-01T22:05:34.037 INFO:teuthology.packaging:tag: v16.2.4 2023-12-01T22:05:34.037 INFO:teuthology.packaging:branch: None 2023-12-01T22:05:34.037 INFO:teuthology.packaging:sha1: 0dc074ce0b0b3b4f393f56dd32ef36884ebf45a9 2023-12-01T22:05:34.038 DEBUG:teuthology.orchestra.run.smithi029:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v16.2.4/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-12-01T22:05:34.046 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-12-01T22:05:34.046 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2023-12-01T22:05:34.046 INFO:teuthology.packaging:ref: None 2023-12-01T22:05:34.046 INFO:teuthology.packaging:tag: v16.2.4 2023-12-01T22:05:34.046 INFO:teuthology.packaging:branch: None 2023-12-01T22:05:34.046 INFO:teuthology.packaging:sha1: 0dc074ce0b0b3b4f393f56dd32ef36884ebf45a9 2023-12-01T22:05:34.047 DEBUG:teuthology.orchestra.run.smithi073:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v16.2.4/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-12-01T22:05:34.114 DEBUG:teuthology.orchestra.run.smithi029:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-12-01T22:05:34.118 DEBUG:teuthology.orchestra.run.smithi073:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-12-01T22:05:34.199 DEBUG:teuthology.orchestra.run.smithi073:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-12-01T22:05:34.202 DEBUG:teuthology.orchestra.run.smithi029:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-12-01T22:05:34.233 INFO:teuthology.orchestra.run.smithi029.stdout:check_obsoletes = 1 2023-12-01T22:05:34.235 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum clean all 2023-12-01T22:05:34.268 INFO:teuthology.orchestra.run.smithi073.stdout:check_obsoletes = 1 2023-12-01T22:05:34.269 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum clean all 2023-12-01T22:05:34.536 INFO:teuthology.orchestra.run.smithi073.stdout:76 files removed 2023-12-01T22:05:34.540 INFO:teuthology.orchestra.run.smithi029.stdout:76 files removed 2023-12-01T22:05:34.565 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install ceph-radosgw 2023-12-01T22:05:34.572 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-radosgw 2023-12-01T22:05:35.063 INFO:teuthology.orchestra.run.smithi073.stdout:ceph packages for x86_64 774 kB/s | 74 kB 00:00 2023-12-01T22:05:35.123 INFO:teuthology.orchestra.run.smithi029.stdout:ceph packages for x86_64 994 kB/s | 74 kB 00:00 2023-12-01T22:05:35.178 INFO:teuthology.orchestra.run.smithi073.stdout:ceph noarch packages 197 kB/s | 15 kB 00:00 2023-12-01T22:05:35.241 INFO:teuthology.orchestra.run.smithi029.stdout:ceph noarch packages 201 kB/s | 15 kB 00:00 2023-12-01T22:05:35.279 INFO:teuthology.orchestra.run.smithi073.stdout:ceph source packages 24 kB/s | 1.7 kB 00:00 2023-12-01T22:05:35.350 INFO:teuthology.orchestra.run.smithi029.stdout:ceph source packages 23 kB/s | 1.7 kB 00:00 2023-12-01T22:05:45.370 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 8 - AppStream 3.4 MB/s | 34 MB 00:09 2023-12-01T22:05:50.169 INFO:teuthology.orchestra.run.smithi073.stdout:CentOS Stream 8 - AppStream 2.3 MB/s | 34 MB 00:14 2023-12-01T22:06:05.214 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 8 - BaseOS 4.3 MB/s | 55 MB 00:12 2023-12-01T22:06:09.370 INFO:teuthology.orchestra.run.smithi073.stdout:CentOS Stream 8 - BaseOS 4.2 MB/s | 55 MB 00:12 2023-12-01T22:06:14.317 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 8 - Extras 47 kB/s | 18 kB 00:00 2023-12-01T22:06:14.927 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 8 - Extras common packages 17 kB/s | 6.9 kB 00:00 2023-12-01T22:06:15.579 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 8 - PowerTools 14 MB/s | 6.6 MB 00:00 2023-12-01T22:06:17.156 INFO:teuthology.orchestra.run.smithi029.stdout:Copr repo for python3-asyncssh owned by ceph 25 kB/s | 3.5 kB 00:00 2023-12-01T22:06:17.678 INFO:teuthology.orchestra.run.smithi073.stdout:CentOS Stream 8 - Extras 48 kB/s | 18 kB 00:00 2023-12-01T22:06:17.993 INFO:teuthology.orchestra.run.smithi029.stdout:Extra Packages for Enterprise Linux 24 MB/s | 16 MB 00:00 2023-12-01T22:06:18.120 INFO:teuthology.orchestra.run.smithi073.stdout:CentOS Stream 8 - Extras common packages 26 kB/s | 6.9 kB 00:00 2023-12-01T22:06:18.838 INFO:teuthology.orchestra.run.smithi073.stdout:CentOS Stream 8 - PowerTools 12 MB/s | 6.6 MB 00:00 2023-12-01T22:06:20.289 INFO:teuthology.orchestra.run.smithi073.stdout:Copr repo for python3-asyncssh owned by ceph 23 kB/s | 3.5 kB 00:00 2023-12-01T22:06:22.336 INFO:teuthology.orchestra.run.smithi029.stdout:lab-extras 400 kB/s | 24 kB 00:00 2023-12-01T22:06:23.531 INFO:teuthology.orchestra.run.smithi073.stdout:Extra Packages for Enterprise Linux 5.2 MB/s | 16 MB 00:03 2023-12-01T22:06:27.431 INFO:teuthology.orchestra.run.smithi073.stdout:lab-extras 417 kB/s | 24 kB 00:00 2023-12-01T22:06:27.519 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-12-01T22:06:27.522 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-12-01T22:06:27.522 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2023-12-01T22:06:27.523 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-12-01T22:06:27.523 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2023-12-01T22:06:27.523 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-radosgw x86_64 2:16.2.4-0.el8 ceph 13 M 2023-12-01T22:06:27.523 INFO:teuthology.orchestra.run.smithi029.stdout:Upgrading: 2023-12-01T22:06:27.523 INFO:teuthology.orchestra.run.smithi029.stdout: librados2 x86_64 2:16.2.4-0.el8 ceph 3.7 M 2023-12-01T22:06:27.523 INFO:teuthology.orchestra.run.smithi029.stdout: librbd1 x86_64 2:16.2.4-0.el8 ceph 4.0 M 2023-12-01T22:06:27.523 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2023-12-01T22:06:27.523 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-base x86_64 2:16.2.4-0.el8 ceph 6.0 M 2023-12-01T22:06:27.523 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-common x86_64 2:16.2.4-0.el8 ceph 24 M 2023-12-01T22:06:27.523 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-selinux x86_64 2:16.2.4-0.el8 ceph 25 k 2023-12-01T22:06:27.523 INFO:teuthology.orchestra.run.smithi029.stdout: daxctl-libs x86_64 71.1-7.el8 baseos 42 k 2023-12-01T22:06:27.523 INFO:teuthology.orchestra.run.smithi029.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2023-12-01T22:06:27.524 INFO:teuthology.orchestra.run.smithi029.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-12-01T22:06:27.524 INFO:teuthology.orchestra.run.smithi029.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2023-12-01T22:06:27.524 INFO:teuthology.orchestra.run.smithi029.stdout: libcephfs2 x86_64 2:16.2.4-0.el8 ceph 801 k 2023-12-01T22:06:27.524 INFO:teuthology.orchestra.run.smithi029.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-12-01T22:06:27.524 INFO:teuthology.orchestra.run.smithi029.stdout: libpmemobj x86_64 1.6.1-1.el8 appstream 145 k 2023-12-01T22:06:27.524 INFO:teuthology.orchestra.run.smithi029.stdout: librabbitmq x86_64 0.9.0-4.el8 baseos 47 k 2023-12-01T22:06:27.524 INFO:teuthology.orchestra.run.smithi029.stdout: libradosstriper1 x86_64 2:16.2.4-0.el8 ceph 486 k 2023-12-01T22:06:27.524 INFO:teuthology.orchestra.run.smithi029.stdout: librdkafka x86_64 0.11.4-3.el8 appstream 354 k 2023-12-01T22:06:27.524 INFO:teuthology.orchestra.run.smithi029.stdout: librgw2 x86_64 2:16.2.4-0.el8 ceph 4.1 M 2023-12-01T22:06:27.524 INFO:teuthology.orchestra.run.smithi029.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-12-01T22:06:27.524 INFO:teuthology.orchestra.run.smithi029.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2023-12-01T22:06:27.524 INFO:teuthology.orchestra.run.smithi029.stdout: ndctl-libs x86_64 71.1-7.el8 baseos 80 k 2023-12-01T22:06:27.524 INFO:teuthology.orchestra.run.smithi029.stdout: python3-ceph-argparse x86_64 2:16.2.4-0.el8 ceph 44 k 2023-12-01T22:06:27.525 INFO:teuthology.orchestra.run.smithi029.stdout: python3-ceph-common x86_64 2:16.2.4-0.el8 ceph 80 k 2023-12-01T22:06:27.525 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cephfs x86_64 2:16.2.4-0.el8 ceph 213 k 2023-12-01T22:06:27.525 INFO:teuthology.orchestra.run.smithi029.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2023-12-01T22:06:27.525 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rados x86_64 2:16.2.4-0.el8 ceph 387 k 2023-12-01T22:06:27.525 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rbd x86_64 2:16.2.4-0.el8 ceph 368 k 2023-12-01T22:06:27.525 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rgw x86_64 2:16.2.4-0.el8 ceph 114 k 2023-12-01T22:06:27.525 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-01T22:06:27.525 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2023-12-01T22:06:27.525 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-12-01T22:06:27.525 INFO:teuthology.orchestra.run.smithi029.stdout:Install 25 Packages 2023-12-01T22:06:27.525 INFO:teuthology.orchestra.run.smithi029.stdout:Upgrade 2 Packages 2023-12-01T22:06:27.525 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-01T22:06:27.526 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 59 M 2023-12-01T22:06:27.526 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2023-12-01T22:06:28.106 INFO:teuthology.orchestra.run.smithi029.stdout:(1/27): ceph-base-16.2.4-0.el8.x86_64.rpm 28 MB/s | 6.0 MB 00:00 2023-12-01T22:06:28.131 INFO:teuthology.orchestra.run.smithi029.stdout:(2/27): ceph-selinux-16.2.4-0.el8.x86_64.rpm 987 kB/s | 25 kB 00:00 2023-12-01T22:06:28.181 INFO:teuthology.orchestra.run.smithi029.stdout:(3/27): libcephfs2-16.2.4-0.el8.x86_64.rpm 16 MB/s | 801 kB 00:00 2023-12-01T22:06:28.224 INFO:teuthology.orchestra.run.smithi029.stdout:(4/27): libradosstriper1-16.2.4-0.el8.x86_64.rp 11 MB/s | 486 kB 00:00 2023-12-01T22:06:28.359 INFO:teuthology.orchestra.run.smithi029.stdout:(5/27): librgw2-16.2.4-0.el8.x86_64.rpm 30 MB/s | 4.1 MB 00:00 2023-12-01T22:06:28.385 INFO:teuthology.orchestra.run.smithi029.stdout:(6/27): python3-ceph-argparse-16.2.4-0.el8.x86_ 1.7 MB/s | 44 kB 00:00 2023-12-01T22:06:28.410 INFO:teuthology.orchestra.run.smithi029.stdout:(7/27): python3-ceph-common-16.2.4-0.el8.x86_64 3.1 MB/s | 80 kB 00:00 2023-12-01T22:06:28.444 INFO:teuthology.orchestra.run.smithi029.stdout:(8/27): python3-cephfs-16.2.4-0.el8.x86_64.rpm 6.2 MB/s | 213 kB 00:00 2023-12-01T22:06:28.478 INFO:teuthology.orchestra.run.smithi029.stdout:(9/27): python3-rados-16.2.4-0.el8.x86_64.rpm 11 MB/s | 387 kB 00:00 2023-12-01T22:06:28.520 INFO:teuthology.orchestra.run.smithi029.stdout:(10/27): python3-rbd-16.2.4-0.el8.x86_64.rpm 8.6 MB/s | 368 kB 00:00 2023-12-01T22:06:28.546 INFO:teuthology.orchestra.run.smithi029.stdout:(11/27): python3-rgw-16.2.4-0.el8.x86_64.rpm 4.5 MB/s | 114 kB 00:00 2023-12-01T22:06:28.780 INFO:teuthology.orchestra.run.smithi029.stdout:(12/27): ceph-radosgw-16.2.4-0.el8.x86_64.rpm 14 MB/s | 13 MB 00:00 2023-12-01T22:06:29.239 INFO:teuthology.orchestra.run.smithi029.stdout:(13/27): ceph-common-16.2.4-0.el8.x86_64.rpm 18 MB/s | 24 MB 00:01 2023-12-01T22:06:29.273 INFO:teuthology.orchestra.run.smithi029.stdout:(14/27): libpmemobj-1.6.1-1.el8.x86_64.rpm 200 kB/s | 145 kB 00:00 2023-12-01T22:06:29.332 INFO:teuthology.orchestra.run.smithi029.stdout:(15/27): librdkafka-0.11.4-3.el8.x86_64.rpm 642 kB/s | 354 kB 00:00 2023-12-01T22:06:29.357 INFO:teuthology.orchestra.run.smithi029.stdout:(16/27): python3-prettytable-0.7.2-14.el8.noarc 521 kB/s | 44 kB 00:00 2023-12-01T22:06:29.624 INFO:teuthology.orchestra.run.smithi029.stdout:(17/27): lttng-ust-2.8.1-11.el8.x86_64.rpm 674 kB/s | 259 kB 00:00 2023-12-01T22:06:29.650 INFO:teuthology.orchestra.run.smithi029.stdout:(18/27): daxctl-libs-71.1-7.el8.x86_64.rpm 133 kB/s | 42 kB 00:00 2023-12-01T22:06:29.675 INFO:teuthology.orchestra.run.smithi029.stdout:(19/27): librabbitmq-0.9.0-4.el8.x86_64.rpm 148 kB/s | 47 kB 00:00 2023-12-01T22:06:29.786 INFO:teuthology.orchestra.run.smithi029.stdout:(20/27): ndctl-libs-71.1-7.el8.x86_64.rpm 498 kB/s | 80 kB 00:00 2023-12-01T22:06:29.953 INFO:teuthology.orchestra.run.smithi029.stdout:(21/27): fmt-6.2.1-1.el8.x86_64.rpm 420 kB/s | 116 kB 00:00 2023-12-01T22:06:30.012 INFO:teuthology.orchestra.run.smithi029.stdout:(22/27): liboath-2.6.2-3.el8.x86_64.rpm 1.0 MB/s | 59 kB 00:00 2023-12-01T22:06:30.037 INFO:teuthology.orchestra.run.smithi029.stdout:(23/27): gperftools-libs-2.7-9.el8.x86_64.rpm 847 kB/s | 306 kB 00:00 2023-12-01T22:06:30.063 INFO:teuthology.orchestra.run.smithi029.stdout:(24/27): leveldb-1.22-1.el8.x86_64.rpm 657 kB/s | 181 kB 00:00 2023-12-01T22:06:30.097 INFO:teuthology.orchestra.run.smithi029.stdout:(25/27): libunwind-1.3.1-3.el8.x86_64.rpm 885 kB/s | 75 kB 00:00 2023-12-01T22:06:30.214 INFO:teuthology.orchestra.run.smithi029.stdout:(26/27): librados2-16.2.4-0.el8.x86_64.rpm 21 MB/s | 3.7 MB 00:00 2023-12-01T22:06:30.298 INFO:teuthology.orchestra.run.smithi029.stdout:(27/27): librbd1-16.2.4-0.el8.x86_64.rpm 17 MB/s | 4.0 MB 00:00 2023-12-01T22:06:30.303 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2023-12-01T22:06:30.303 INFO:teuthology.orchestra.run.smithi029.stdout:Total 21 MB/s | 59 MB 00:02 2023-12-01T22:06:30.373 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2023-12-01T22:06:30.427 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2023-12-01T22:06:30.427 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2023-12-01T22:06:30.792 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2023-12-01T22:06:30.796 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2023-12-01T22:06:31.917 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2023-12-01T22:06:32.006 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2023-12-01T22:06:32.009 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2023-12-01T22:06:32.009 INFO:teuthology.orchestra.run.smithi073.stdout: Package Arch Version Repository Size 2023-12-01T22:06:32.009 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2023-12-01T22:06:32.009 INFO:teuthology.orchestra.run.smithi073.stdout:Installing: 2023-12-01T22:06:32.009 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-radosgw x86_64 2:16.2.4-0.el8 ceph 13 M 2023-12-01T22:06:32.010 INFO:teuthology.orchestra.run.smithi073.stdout:Upgrading: 2023-12-01T22:06:32.010 INFO:teuthology.orchestra.run.smithi073.stdout: librados2 x86_64 2:16.2.4-0.el8 ceph 3.7 M 2023-12-01T22:06:32.010 INFO:teuthology.orchestra.run.smithi073.stdout: librbd1 x86_64 2:16.2.4-0.el8 ceph 4.0 M 2023-12-01T22:06:32.010 INFO:teuthology.orchestra.run.smithi073.stdout:Installing dependencies: 2023-12-01T22:06:32.010 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-base x86_64 2:16.2.4-0.el8 ceph 6.0 M 2023-12-01T22:06:32.010 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-common x86_64 2:16.2.4-0.el8 ceph 24 M 2023-12-01T22:06:32.010 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-selinux x86_64 2:16.2.4-0.el8 ceph 25 k 2023-12-01T22:06:32.010 INFO:teuthology.orchestra.run.smithi073.stdout: daxctl-libs x86_64 71.1-7.el8 baseos 42 k 2023-12-01T22:06:32.010 INFO:teuthology.orchestra.run.smithi073.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2023-12-01T22:06:32.010 INFO:teuthology.orchestra.run.smithi073.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-12-01T22:06:32.010 INFO:teuthology.orchestra.run.smithi073.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2023-12-01T22:06:32.010 INFO:teuthology.orchestra.run.smithi073.stdout: libcephfs2 x86_64 2:16.2.4-0.el8 ceph 801 k 2023-12-01T22:06:32.011 INFO:teuthology.orchestra.run.smithi073.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-12-01T22:06:32.011 INFO:teuthology.orchestra.run.smithi073.stdout: libpmemobj x86_64 1.6.1-1.el8 appstream 145 k 2023-12-01T22:06:32.011 INFO:teuthology.orchestra.run.smithi073.stdout: librabbitmq x86_64 0.9.0-4.el8 baseos 47 k 2023-12-01T22:06:32.011 INFO:teuthology.orchestra.run.smithi073.stdout: libradosstriper1 x86_64 2:16.2.4-0.el8 ceph 486 k 2023-12-01T22:06:32.011 INFO:teuthology.orchestra.run.smithi073.stdout: librdkafka x86_64 0.11.4-3.el8 appstream 354 k 2023-12-01T22:06:32.011 INFO:teuthology.orchestra.run.smithi073.stdout: librgw2 x86_64 2:16.2.4-0.el8 ceph 4.1 M 2023-12-01T22:06:32.011 INFO:teuthology.orchestra.run.smithi073.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-12-01T22:06:32.011 INFO:teuthology.orchestra.run.smithi073.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2023-12-01T22:06:32.011 INFO:teuthology.orchestra.run.smithi073.stdout: ndctl-libs x86_64 71.1-7.el8 baseos 80 k 2023-12-01T22:06:32.011 INFO:teuthology.orchestra.run.smithi073.stdout: python3-ceph-argparse x86_64 2:16.2.4-0.el8 ceph 44 k 2023-12-01T22:06:32.011 INFO:teuthology.orchestra.run.smithi073.stdout: python3-ceph-common x86_64 2:16.2.4-0.el8 ceph 80 k 2023-12-01T22:06:32.011 INFO:teuthology.orchestra.run.smithi073.stdout: python3-cephfs x86_64 2:16.2.4-0.el8 ceph 213 k 2023-12-01T22:06:32.011 INFO:teuthology.orchestra.run.smithi073.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2023-12-01T22:06:32.011 INFO:teuthology.orchestra.run.smithi073.stdout: python3-rados x86_64 2:16.2.4-0.el8 ceph 387 k 2023-12-01T22:06:32.011 INFO:teuthology.orchestra.run.smithi073.stdout: python3-rbd x86_64 2:16.2.4-0.el8 ceph 368 k 2023-12-01T22:06:32.012 INFO:teuthology.orchestra.run.smithi073.stdout: python3-rgw x86_64 2:16.2.4-0.el8 ceph 114 k 2023-12-01T22:06:32.012 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:06:32.012 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction Summary 2023-12-01T22:06:32.012 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2023-12-01T22:06:32.012 INFO:teuthology.orchestra.run.smithi073.stdout:Install 25 Packages 2023-12-01T22:06:32.012 INFO:teuthology.orchestra.run.smithi073.stdout:Upgrade 2 Packages 2023-12-01T22:06:32.012 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:06:32.012 INFO:teuthology.orchestra.run.smithi073.stdout:Total download size: 59 M 2023-12-01T22:06:32.013 INFO:teuthology.orchestra.run.smithi073.stdout:Downloading Packages: 2023-12-01T22:06:32.285 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/29 2023-12-01T22:06:32.523 INFO:teuthology.orchestra.run.smithi073.stdout:(1/27): ceph-base-16.2.4-0.el8.x86_64.rpm 25 MB/s | 6.0 MB 00:00 2023-12-01T22:06:32.539 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/29 2023-12-01T22:06:32.542 INFO:teuthology.orchestra.run.smithi073.stdout:(2/27): ceph-selinux-16.2.4-0.el8.x86_64.rpm 1.3 MB/s | 25 kB 00:00 2023-12-01T22:06:32.575 INFO:teuthology.orchestra.run.smithi073.stdout:(3/27): libcephfs2-16.2.4-0.el8.x86_64.rpm 24 MB/s | 801 kB 00:00 2023-12-01T22:06:32.609 INFO:teuthology.orchestra.run.smithi073.stdout:(4/27): libradosstriper1-16.2.4-0.el8.x86_64.rp 14 MB/s | 486 kB 00:00 2023-12-01T22:06:32.709 INFO:teuthology.orchestra.run.smithi073.stdout:(5/27): librgw2-16.2.4-0.el8.x86_64.rpm 41 MB/s | 4.1 MB 00:00 2023-12-01T22:06:32.726 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/29 2023-12-01T22:06:32.734 INFO:teuthology.orchestra.run.smithi073.stdout:(6/27): python3-ceph-argparse-16.2.4-0.el8.x86_ 1.8 MB/s | 44 kB 00:00 2023-12-01T22:06:32.760 INFO:teuthology.orchestra.run.smithi073.stdout:(7/27): python3-ceph-common-16.2.4-0.el8.x86_64 3.1 MB/s | 80 kB 00:00 2023-12-01T22:06:32.785 INFO:teuthology.orchestra.run.smithi073.stdout:(8/27): python3-cephfs-16.2.4-0.el8.x86_64.rpm 8.3 MB/s | 213 kB 00:00 2023-12-01T22:06:32.819 INFO:teuthology.orchestra.run.smithi073.stdout:(9/27): python3-rados-16.2.4-0.el8.x86_64.rpm 11 MB/s | 387 kB 00:00 2023-12-01T22:06:32.844 INFO:teuthology.orchestra.run.smithi073.stdout:(10/27): python3-rbd-16.2.4-0.el8.x86_64.rpm 14 MB/s | 368 kB 00:00 2023-12-01T22:06:32.869 INFO:teuthology.orchestra.run.smithi073.stdout:(11/27): python3-rgw-16.2.4-0.el8.x86_64.rpm 4.5 MB/s | 114 kB 00:00 2023-12-01T22:06:33.011 INFO:teuthology.orchestra.run.smithi073.stdout:(12/27): ceph-radosgw-16.2.4-0.el8.x86_64.rpm 17 MB/s | 13 MB 00:00 2023-12-01T22:06:33.177 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/29 2023-12-01T22:06:33.303 INFO:teuthology.orchestra.run.smithi073.stdout:(13/27): ceph-common-16.2.4-0.el8.x86_64.rpm 23 MB/s | 24 MB 00:01 2023-12-01T22:06:33.328 INFO:teuthology.orchestra.run.smithi073.stdout:(14/27): libpmemobj-1.6.1-1.el8.x86_64.rpm 316 kB/s | 145 kB 00:00 2023-12-01T22:06:33.387 INFO:teuthology.orchestra.run.smithi073.stdout:(15/27): python3-prettytable-0.7.2-14.el8.noarc 751 kB/s | 44 kB 00:00 2023-12-01T22:06:33.402 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : librados2-2:16.2.4-0.el8.x86_64 4/29 2023-12-01T22:06:33.420 INFO:teuthology.orchestra.run.smithi073.stdout:(16/27): librdkafka-0.11.4-3.el8.x86_64.rpm 865 kB/s | 354 kB 00:00 2023-12-01T22:06:33.488 INFO:teuthology.orchestra.run.smithi073.stdout:(17/27): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.4 MB/s | 259 kB 00:00 2023-12-01T22:06:33.509 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librados2-2:16.2.4-0.el8.x86_64 4/29 2023-12-01T22:06:33.696 INFO:teuthology.orchestra.run.smithi073.stdout:(18/27): daxctl-libs-71.1-7.el8.x86_64.rpm 137 kB/s | 42 kB 00:00 2023-12-01T22:06:33.721 INFO:teuthology.orchestra.run.smithi073.stdout:(19/27): librabbitmq-0.9.0-4.el8.x86_64.rpm 156 kB/s | 47 kB 00:00 2023-12-01T22:06:33.763 INFO:teuthology.orchestra.run.smithi073.stdout:(20/27): ndctl-libs-71.1-7.el8.x86_64.rpm 291 kB/s | 80 kB 00:00 2023-12-01T22:06:33.808 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-rados-2:16.2.4-0.el8.x86_64 5/29 2023-12-01T22:06:34.001 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2023-12-01T22:06:34.020 INFO:teuthology.orchestra.run.smithi073.stdout:(21/27): fmt-6.2.1-1.el8.x86_64.rpm 360 kB/s | 116 kB 00:00 2023-12-01T22:06:34.098 INFO:teuthology.orchestra.run.smithi073.stdout:(22/27): leveldb-1.22-1.el8.x86_64.rpm 558 kB/s | 181 kB 00:00 2023-12-01T22:06:34.104 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2023-12-01T22:06:34.123 INFO:teuthology.orchestra.run.smithi073.stdout:(23/27): liboath-2.6.2-3.el8.x86_64.rpm 576 kB/s | 59 kB 00:00 2023-12-01T22:06:34.148 INFO:teuthology.orchestra.run.smithi073.stdout:(24/27): gperftools-libs-2.7-9.el8.x86_64.rpm 719 kB/s | 306 kB 00:00 2023-12-01T22:06:34.232 INFO:teuthology.orchestra.run.smithi073.stdout:(25/27): librados2-16.2.4-0.el8.x86_64.rpm 34 MB/s | 3.7 MB 00:00 2023-12-01T22:06:34.248 INFO:teuthology.orchestra.run.smithi073.stdout:(26/27): libunwind-1.3.1-3.el8.x86_64.rpm 498 kB/s | 75 kB 00:00 2023-12-01T22:06:34.307 INFO:teuthology.orchestra.run.smithi073.stdout:(27/27): librbd1-16.2.4-0.el8.x86_64.rpm 25 MB/s | 4.0 MB 00:00 2023-12-01T22:06:34.312 INFO:teuthology.orchestra.run.smithi073.stdout:-------------------------------------------------------------------------------- 2023-12-01T22:06:34.313 INFO:teuthology.orchestra.run.smithi073.stdout:Total 25 MB/s | 59 MB 00:02 2023-12-01T22:06:34.320 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : librabbitmq-0.9.0-4.el8.x86_64 7/29 2023-12-01T22:06:34.381 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction check 2023-12-01T22:06:34.429 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction check succeeded. 2023-12-01T22:06:34.429 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction test 2023-12-01T22:06:34.510 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 8/29 2023-12-01T22:06:34.778 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction test succeeded. 2023-12-01T22:06:34.781 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction 2023-12-01T22:06:34.970 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 8/29 2023-12-01T22:06:35.159 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : librgw2-2:16.2.4-0.el8.x86_64 9/29 2023-12-01T22:06:35.240 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librgw2-2:16.2.4-0.el8.x86_64 9/29 2023-12-01T22:06:35.457 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/29 2023-12-01T22:06:35.673 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : daxctl-libs-71.1-7.el8.x86_64 11/29 2023-12-01T22:06:35.741 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: daxctl-libs-71.1-7.el8.x86_64 11/29 2023-12-01T22:06:35.893 INFO:teuthology.orchestra.run.smithi073.stdout: Preparing : 1/1 2023-12-01T22:06:35.928 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 12/29 2023-12-01T22:06:36.145 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cephfs-2:16.2.4-0.el8.x86_64 13/29 2023-12-01T22:06:36.150 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/29 2023-12-01T22:06:36.351 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ndctl-libs-71.1-7.el8.x86_64 14/29 2023-12-01T22:06:36.380 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/29 2023-12-01T22:06:36.437 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ndctl-libs-71.1-7.el8.x86_64 14/29 2023-12-01T22:06:36.556 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/29 2023-12-01T22:06:36.609 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libpmemobj-1.6.1-1.el8.x86_64 15/29 2023-12-01T22:06:36.976 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/29 2023-12-01T22:06:37.097 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: libpmemobj-1.6.1-1.el8.x86_64 15/29 2023-12-01T22:06:37.181 INFO:teuthology.orchestra.run.smithi073.stdout: Upgrading : librados2-2:16.2.4-0.el8.x86_64 4/29 2023-12-01T22:06:37.275 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: librados2-2:16.2.4-0.el8.x86_64 4/29 2023-12-01T22:06:37.301 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : librbd1-2:16.2.4-0.el8.x86_64 16/29 2023-12-01T22:06:37.404 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librbd1-2:16.2.4-0.el8.x86_64 16/29 2023-12-01T22:06:37.518 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-rados-2:16.2.4-0.el8.x86_64 5/29 2023-12-01T22:06:37.642 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-rbd-2:16.2.4-0.el8.x86_64 17/29 2023-12-01T22:06:37.681 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2023-12-01T22:06:37.728 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2023-12-01T22:06:37.908 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-rgw-2:16.2.4-0.el8.x86_64 18/29 2023-12-01T22:06:37.921 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : librabbitmq-0.9.0-4.el8.x86_64 7/29 2023-12-01T22:06:38.076 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2023-12-01T22:06:38.080 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 8/29 2023-12-01T22:06:38.151 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2023-12-01T22:06:38.380 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 20/29 2023-12-01T22:06:38.485 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 8/29 2023-12-01T22:06:38.586 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 21/29 2023-12-01T22:06:38.631 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : librgw2-2:16.2.4-0.el8.x86_64 9/29 2023-12-01T22:06:38.720 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: librgw2-2:16.2.4-0.el8.x86_64 9/29 2023-12-01T22:06:38.799 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 22/29 2023-12-01T22:06:38.877 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/29 2023-12-01T22:06:39.010 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-ceph-common-2:16.2.4-0.el8.x86_64 23/29 2023-12-01T22:06:39.085 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : daxctl-libs-71.1-7.el8.x86_64 11/29 2023-12-01T22:06:39.151 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: daxctl-libs-71.1-7.el8.x86_64 11/29 2023-12-01T22:06:39.336 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 12/29 2023-12-01T22:06:39.523 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-cephfs-2:16.2.4-0.el8.x86_64 13/29 2023-12-01T22:06:39.739 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ndctl-libs-71.1-7.el8.x86_64 14/29 2023-12-01T22:06:39.811 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: ndctl-libs-71.1-7.el8.x86_64 14/29 2023-12-01T22:06:39.955 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : libpmemobj-1.6.1-1.el8.x86_64 15/29 2023-12-01T22:06:40.391 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: libpmemobj-1.6.1-1.el8.x86_64 15/29 2023-12-01T22:06:40.605 INFO:teuthology.orchestra.run.smithi073.stdout: Upgrading : librbd1-2:16.2.4-0.el8.x86_64 16/29 2023-12-01T22:06:40.708 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: librbd1-2:16.2.4-0.el8.x86_64 16/29 2023-12-01T22:06:40.884 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-rbd-2:16.2.4-0.el8.x86_64 17/29 2023-12-01T22:06:41.124 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-rgw-2:16.2.4-0.el8.x86_64 18/29 2023-12-01T22:06:41.280 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2023-12-01T22:06:41.361 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2023-12-01T22:06:41.564 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 20/29 2023-12-01T22:06:41.805 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 21/29 2023-12-01T22:06:41.853 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2023-12-01T22:06:41.987 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 22/29 2023-12-01T22:06:42.093 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-common-2:16.2.4-0.el8.x86_64 24/29 2023-12-01T22:06:42.222 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-ceph-common-2:16.2.4-0.el8.x86_64 23/29 2023-12-01T22:06:42.885 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2023-12-01T22:06:43.181 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-base-2:16.2.4-0.el8.x86_64 25/29 2023-12-01T22:06:43.301 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-base-2:16.2.4-0.el8.x86_64 25/29 2023-12-01T22:06:43.450 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2023-12-01T22:06:44.814 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2023-12-01T22:06:45.047 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-common-2:16.2.4-0.el8.x86_64 24/29 2023-12-01T22:06:45.756 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2023-12-01T22:06:46.061 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-base-2:16.2.4-0.el8.x86_64 25/29 2023-12-01T22:06:46.146 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: ceph-base-2:16.2.4-0.el8.x86_64 25/29 2023-12-01T22:06:46.305 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2023-12-01T22:07:04.659 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2023-12-01T22:07:04.660 INFO:teuthology.orchestra.run.smithi073.stdout:skipping the directory /sys 2023-12-01T22:07:04.660 INFO:teuthology.orchestra.run.smithi073.stdout:skipping the directory /proc 2023-12-01T22:07:04.660 INFO:teuthology.orchestra.run.smithi073.stdout:skipping the directory /dev 2023-12-01T22:07:04.660 INFO:teuthology.orchestra.run.smithi073.stdout:skipping the directory /run 2023-12-01T22:07:04.660 INFO:teuthology.orchestra.run.smithi073.stdout:skipping the directory /mnt 2023-12-01T22:07:04.660 INFO:teuthology.orchestra.run.smithi073.stdout:skipping the directory /var/tmp 2023-12-01T22:07:04.661 INFO:teuthology.orchestra.run.smithi073.stdout:skipping the directory /home 2023-12-01T22:07:04.661 INFO:teuthology.orchestra.run.smithi073.stdout:skipping the directory /tmp 2023-12-01T22:07:04.661 INFO:teuthology.orchestra.run.smithi073.stdout:skipping the directory /dev 2023-12-01T22:07:04.661 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:07:06.016 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2023-12-01T22:07:06.077 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2023-12-01T22:07:06.077 INFO:teuthology.orchestra.run.smithi073.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 28/29 2023-12-01T22:07:06.271 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 28/29 2023-12-01T22:07:06.271 INFO:teuthology.orchestra.run.smithi073.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 29/29 2023-12-01T22:07:06.358 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2023-12-01T22:07:06.358 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /sys 2023-12-01T22:07:06.358 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /proc 2023-12-01T22:07:06.358 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /dev 2023-12-01T22:07:06.358 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /run 2023-12-01T22:07:06.358 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /mnt 2023-12-01T22:07:06.358 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /var/tmp 2023-12-01T22:07:06.358 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /home 2023-12-01T22:07:06.358 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /tmp 2023-12-01T22:07:06.358 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /dev 2023-12-01T22:07:06.358 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-01T22:07:07.021 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 29/29 2023-12-01T22:07:07.021 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-base-2:16.2.4-0.el8.x86_64 1/29 2023-12-01T22:07:07.021 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-common-2:16.2.4-0.el8.x86_64 2/29 2023-12-01T22:07:07.021 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-radosgw-2:16.2.4-0.el8.x86_64 3/29 2023-12-01T22:07:07.021 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-selinux-2:16.2.4-0.el8.x86_64 4/29 2023-12-01T22:07:07.021 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : libcephfs2-2:16.2.4-0.el8.x86_64 5/29 2023-12-01T22:07:07.021 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : libradosstriper1-2:16.2.4-0.el8.x86_64 6/29 2023-12-01T22:07:07.021 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : librgw2-2:16.2.4-0.el8.x86_64 7/29 2023-12-01T22:07:07.021 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 8/29 2023-12-01T22:07:07.021 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-ceph-common-2:16.2.4-0.el8.x86_64 9/29 2023-12-01T22:07:07.021 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-cephfs-2:16.2.4-0.el8.x86_64 10/29 2023-12-01T22:07:07.021 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-rados-2:16.2.4-0.el8.x86_64 11/29 2023-12-01T22:07:07.021 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-rbd-2:16.2.4-0.el8.x86_64 12/29 2023-12-01T22:07:07.022 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-rgw-2:16.2.4-0.el8.x86_64 13/29 2023-12-01T22:07:07.022 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : libpmemobj-1.6.1-1.el8.x86_64 14/29 2023-12-01T22:07:07.022 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 15/29 2023-12-01T22:07:07.022 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 16/29 2023-12-01T22:07:07.022 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 17/29 2023-12-01T22:07:07.022 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : daxctl-libs-71.1-7.el8.x86_64 18/29 2023-12-01T22:07:07.022 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : librabbitmq-0.9.0-4.el8.x86_64 19/29 2023-12-01T22:07:07.022 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ndctl-libs-71.1-7.el8.x86_64 20/29 2023-12-01T22:07:07.022 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 21/29 2023-12-01T22:07:07.022 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 22/29 2023-12-01T22:07:07.022 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : leveldb-1.22-1.el8.x86_64 23/29 2023-12-01T22:07:07.022 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 24/29 2023-12-01T22:07:07.022 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 25/29 2023-12-01T22:07:07.022 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : librados2-2:16.2.4-0.el8.x86_64 26/29 2023-12-01T22:07:07.023 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 27/29 2023-12-01T22:07:07.023 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : librbd1-2:16.2.4-0.el8.x86_64 28/29 2023-12-01T22:07:07.828 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2023-12-01T22:07:07.907 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2023-12-01T22:07:07.908 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 28/29 2023-12-01T22:07:08.150 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 28/29 2023-12-01T22:07:08.151 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 29/29 2023-12-01T22:07:08.265 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 29/29 2023-12-01T22:07:08.266 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:07:08.266 INFO:teuthology.orchestra.run.smithi073.stdout:Upgraded: 2023-12-01T22:07:08.266 INFO:teuthology.orchestra.run.smithi073.stdout: librados2-2:16.2.4-0.el8.x86_64 librbd1-2:16.2.4-0.el8.x86_64 2023-12-01T22:07:08.266 INFO:teuthology.orchestra.run.smithi073.stdout:Installed: 2023-12-01T22:07:08.266 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-base-2:16.2.4-0.el8.x86_64 2023-12-01T22:07:08.266 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-common-2:16.2.4-0.el8.x86_64 2023-12-01T22:07:08.266 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-radosgw-2:16.2.4-0.el8.x86_64 2023-12-01T22:07:08.266 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-selinux-2:16.2.4-0.el8.x86_64 2023-12-01T22:07:08.266 INFO:teuthology.orchestra.run.smithi073.stdout: daxctl-libs-71.1-7.el8.x86_64 2023-12-01T22:07:08.266 INFO:teuthology.orchestra.run.smithi073.stdout: fmt-6.2.1-1.el8.x86_64 2023-12-01T22:07:08.266 INFO:teuthology.orchestra.run.smithi073.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-12-01T22:07:08.266 INFO:teuthology.orchestra.run.smithi073.stdout: leveldb-1.22-1.el8.x86_64 2023-12-01T22:07:08.267 INFO:teuthology.orchestra.run.smithi073.stdout: libcephfs2-2:16.2.4-0.el8.x86_64 2023-12-01T22:07:08.267 INFO:teuthology.orchestra.run.smithi073.stdout: liboath-2.6.2-3.el8.x86_64 2023-12-01T22:07:08.267 INFO:teuthology.orchestra.run.smithi073.stdout: libpmemobj-1.6.1-1.el8.x86_64 2023-12-01T22:07:08.267 INFO:teuthology.orchestra.run.smithi073.stdout: librabbitmq-0.9.0-4.el8.x86_64 2023-12-01T22:07:08.267 INFO:teuthology.orchestra.run.smithi073.stdout: libradosstriper1-2:16.2.4-0.el8.x86_64 2023-12-01T22:07:08.267 INFO:teuthology.orchestra.run.smithi073.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-12-01T22:07:08.267 INFO:teuthology.orchestra.run.smithi073.stdout: librgw2-2:16.2.4-0.el8.x86_64 2023-12-01T22:07:08.267 INFO:teuthology.orchestra.run.smithi073.stdout: libunwind-1.3.1-3.el8.x86_64 2023-12-01T22:07:08.267 INFO:teuthology.orchestra.run.smithi073.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-12-01T22:07:08.267 INFO:teuthology.orchestra.run.smithi073.stdout: ndctl-libs-71.1-7.el8.x86_64 2023-12-01T22:07:08.267 INFO:teuthology.orchestra.run.smithi073.stdout: python3-ceph-argparse-2:16.2.4-0.el8.x86_64 2023-12-01T22:07:08.267 INFO:teuthology.orchestra.run.smithi073.stdout: python3-ceph-common-2:16.2.4-0.el8.x86_64 2023-12-01T22:07:08.267 INFO:teuthology.orchestra.run.smithi073.stdout: python3-cephfs-2:16.2.4-0.el8.x86_64 2023-12-01T22:07:08.267 INFO:teuthology.orchestra.run.smithi073.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-12-01T22:07:08.268 INFO:teuthology.orchestra.run.smithi073.stdout: python3-rados-2:16.2.4-0.el8.x86_64 2023-12-01T22:07:08.268 INFO:teuthology.orchestra.run.smithi073.stdout: python3-rbd-2:16.2.4-0.el8.x86_64 2023-12-01T22:07:08.268 INFO:teuthology.orchestra.run.smithi073.stdout: python3-rgw-2:16.2.4-0.el8.x86_64 2023-12-01T22:07:08.268 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:07:08.268 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2023-12-01T22:07:08.494 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install ceph-test 2023-12-01T22:07:08.966 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 29/29 2023-12-01T22:07:08.966 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-base-2:16.2.4-0.el8.x86_64 1/29 2023-12-01T22:07:08.966 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-common-2:16.2.4-0.el8.x86_64 2/29 2023-12-01T22:07:08.967 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-radosgw-2:16.2.4-0.el8.x86_64 3/29 2023-12-01T22:07:08.967 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-selinux-2:16.2.4-0.el8.x86_64 4/29 2023-12-01T22:07:08.967 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libcephfs2-2:16.2.4-0.el8.x86_64 5/29 2023-12-01T22:07:08.967 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libradosstriper1-2:16.2.4-0.el8.x86_64 6/29 2023-12-01T22:07:08.967 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librgw2-2:16.2.4-0.el8.x86_64 7/29 2023-12-01T22:07:08.967 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 8/29 2023-12-01T22:07:08.967 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-ceph-common-2:16.2.4-0.el8.x86_64 9/29 2023-12-01T22:07:08.967 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cephfs-2:16.2.4-0.el8.x86_64 10/29 2023-12-01T22:07:08.967 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-rados-2:16.2.4-0.el8.x86_64 11/29 2023-12-01T22:07:08.967 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-rbd-2:16.2.4-0.el8.x86_64 12/29 2023-12-01T22:07:08.967 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-rgw-2:16.2.4-0.el8.x86_64 13/29 2023-12-01T22:07:08.967 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libpmemobj-1.6.1-1.el8.x86_64 14/29 2023-12-01T22:07:08.967 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 15/29 2023-12-01T22:07:08.967 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 16/29 2023-12-01T22:07:08.967 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 17/29 2023-12-01T22:07:08.968 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : daxctl-libs-71.1-7.el8.x86_64 18/29 2023-12-01T22:07:08.968 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librabbitmq-0.9.0-4.el8.x86_64 19/29 2023-12-01T22:07:08.968 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ndctl-libs-71.1-7.el8.x86_64 20/29 2023-12-01T22:07:08.968 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 21/29 2023-12-01T22:07:08.968 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 22/29 2023-12-01T22:07:08.968 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : leveldb-1.22-1.el8.x86_64 23/29 2023-12-01T22:07:08.968 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 24/29 2023-12-01T22:07:08.968 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 25/29 2023-12-01T22:07:08.968 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librados2-2:16.2.4-0.el8.x86_64 26/29 2023-12-01T22:07:08.968 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 27/29 2023-12-01T22:07:08.968 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librbd1-2:16.2.4-0.el8.x86_64 28/29 2023-12-01T22:07:09.340 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:00:42 ago on Fri 01 Dec 2023 10:06:27 PM UTC. 2023-12-01T22:07:10.239 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 29/29 2023-12-01T22:07:10.239 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-01T22:07:10.240 INFO:teuthology.orchestra.run.smithi029.stdout:Upgraded: 2023-12-01T22:07:10.240 INFO:teuthology.orchestra.run.smithi029.stdout: librados2-2:16.2.4-0.el8.x86_64 librbd1-2:16.2.4-0.el8.x86_64 2023-12-01T22:07:10.240 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2023-12-01T22:07:10.240 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-base-2:16.2.4-0.el8.x86_64 2023-12-01T22:07:10.240 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-common-2:16.2.4-0.el8.x86_64 2023-12-01T22:07:10.240 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-radosgw-2:16.2.4-0.el8.x86_64 2023-12-01T22:07:10.240 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-selinux-2:16.2.4-0.el8.x86_64 2023-12-01T22:07:10.240 INFO:teuthology.orchestra.run.smithi029.stdout: daxctl-libs-71.1-7.el8.x86_64 2023-12-01T22:07:10.240 INFO:teuthology.orchestra.run.smithi029.stdout: fmt-6.2.1-1.el8.x86_64 2023-12-01T22:07:10.240 INFO:teuthology.orchestra.run.smithi029.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-12-01T22:07:10.240 INFO:teuthology.orchestra.run.smithi029.stdout: leveldb-1.22-1.el8.x86_64 2023-12-01T22:07:10.240 INFO:teuthology.orchestra.run.smithi029.stdout: libcephfs2-2:16.2.4-0.el8.x86_64 2023-12-01T22:07:10.240 INFO:teuthology.orchestra.run.smithi029.stdout: liboath-2.6.2-3.el8.x86_64 2023-12-01T22:07:10.240 INFO:teuthology.orchestra.run.smithi029.stdout: libpmemobj-1.6.1-1.el8.x86_64 2023-12-01T22:07:10.241 INFO:teuthology.orchestra.run.smithi029.stdout: librabbitmq-0.9.0-4.el8.x86_64 2023-12-01T22:07:10.241 INFO:teuthology.orchestra.run.smithi029.stdout: libradosstriper1-2:16.2.4-0.el8.x86_64 2023-12-01T22:07:10.241 INFO:teuthology.orchestra.run.smithi029.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-12-01T22:07:10.241 INFO:teuthology.orchestra.run.smithi029.stdout: librgw2-2:16.2.4-0.el8.x86_64 2023-12-01T22:07:10.241 INFO:teuthology.orchestra.run.smithi029.stdout: libunwind-1.3.1-3.el8.x86_64 2023-12-01T22:07:10.241 INFO:teuthology.orchestra.run.smithi029.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-12-01T22:07:10.241 INFO:teuthology.orchestra.run.smithi029.stdout: ndctl-libs-71.1-7.el8.x86_64 2023-12-01T22:07:10.241 INFO:teuthology.orchestra.run.smithi029.stdout: python3-ceph-argparse-2:16.2.4-0.el8.x86_64 2023-12-01T22:07:10.241 INFO:teuthology.orchestra.run.smithi029.stdout: python3-ceph-common-2:16.2.4-0.el8.x86_64 2023-12-01T22:07:10.241 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cephfs-2:16.2.4-0.el8.x86_64 2023-12-01T22:07:10.241 INFO:teuthology.orchestra.run.smithi029.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-12-01T22:07:10.241 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rados-2:16.2.4-0.el8.x86_64 2023-12-01T22:07:10.241 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rbd-2:16.2.4-0.el8.x86_64 2023-12-01T22:07:10.241 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rgw-2:16.2.4-0.el8.x86_64 2023-12-01T22:07:10.241 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-01T22:07:10.242 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-12-01T22:07:10.677 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-test 2023-12-01T22:07:11.029 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2023-12-01T22:07:11.030 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2023-12-01T22:07:11.030 INFO:teuthology.orchestra.run.smithi073.stdout: Package Architecture Version Repository Size 2023-12-01T22:07:11.030 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2023-12-01T22:07:11.030 INFO:teuthology.orchestra.run.smithi073.stdout:Installing: 2023-12-01T22:07:11.030 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-test x86_64 2:16.2.4-0.el8 ceph 52 M 2023-12-01T22:07:11.030 INFO:teuthology.orchestra.run.smithi073.stdout:Installing dependencies: 2023-12-01T22:07:11.030 INFO:teuthology.orchestra.run.smithi073.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2023-12-01T22:07:11.030 INFO:teuthology.orchestra.run.smithi073.stdout: libcephsqlite x86_64 2:16.2.4-0.el8 ceph 169 k 2023-12-01T22:07:11.030 INFO:teuthology.orchestra.run.smithi073.stdout: oniguruma x86_64 6.8.2-2.el8 appstream 187 k 2023-12-01T22:07:11.030 INFO:teuthology.orchestra.run.smithi073.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2023-12-01T22:07:11.031 INFO:teuthology.orchestra.run.smithi073.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2023-12-01T22:07:11.031 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:07:11.031 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction Summary 2023-12-01T22:07:11.031 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2023-12-01T22:07:11.031 INFO:teuthology.orchestra.run.smithi073.stdout:Install 6 Packages 2023-12-01T22:07:11.031 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:07:11.031 INFO:teuthology.orchestra.run.smithi073.stdout:Total download size: 53 M 2023-12-01T22:07:11.033 INFO:teuthology.orchestra.run.smithi073.stdout:Installed size: 229 M 2023-12-01T22:07:11.033 INFO:teuthology.orchestra.run.smithi073.stdout:Downloading Packages: 2023-12-01T22:07:11.127 INFO:teuthology.orchestra.run.smithi073.stdout:(1/6): libcephsqlite-16.2.4-0.el8.x86_64.rpm 3.1 MB/s | 169 kB 00:00 2023-12-01T22:07:11.668 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:00:49 ago on Fri 01 Dec 2023 10:06:22 PM UTC. 2023-12-01T22:07:11.777 INFO:teuthology.orchestra.run.smithi073.stdout:(2/6): ceph-test-16.2.4-0.el8.x86_64.rpm 74 MB/s | 52 MB 00:00 2023-12-01T22:07:11.802 INFO:teuthology.orchestra.run.smithi073.stdout:(3/6): oniguruma-6.8.2-2.el8.x86_64.rpm 277 kB/s | 187 kB 00:00 2023-12-01T22:07:11.836 INFO:teuthology.orchestra.run.smithi073.stdout:(4/6): jq-1.6-8.el8.x86_64.rpm 266 kB/s | 203 kB 00:00 2023-12-01T22:07:11.870 INFO:teuthology.orchestra.run.smithi073.stdout:(5/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.0 MB/s | 71 kB 00:00 2023-12-01T22:07:12.003 INFO:teuthology.orchestra.run.smithi073.stdout:(6/6): socat-1.7.4.1-1.el8.x86_64.rpm 1.4 MB/s | 323 kB 00:00 2023-12-01T22:07:12.004 INFO:teuthology.orchestra.run.smithi073.stdout:-------------------------------------------------------------------------------- 2023-12-01T22:07:12.004 INFO:teuthology.orchestra.run.smithi073.stdout:Total 54 MB/s | 53 MB 00:00 2023-12-01T22:07:12.047 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction check 2023-12-01T22:07:12.068 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction check succeeded. 2023-12-01T22:07:12.068 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction test 2023-12-01T22:07:12.379 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction test succeeded. 2023-12-01T22:07:12.382 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction 2023-12-01T22:07:12.855 INFO:teuthology.orchestra.run.smithi073.stdout: Preparing : 1/1 2023-12-01T22:07:13.120 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2023-12-01T22:07:13.303 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2023-12-01T22:07:13.484 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/6 2023-12-01T22:07:13.531 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-12-01T22:07:13.532 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-12-01T22:07:13.532 INFO:teuthology.orchestra.run.smithi029.stdout: Package Architecture Version Repository Size 2023-12-01T22:07:13.533 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-12-01T22:07:13.533 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2023-12-01T22:07:13.533 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-test x86_64 2:16.2.4-0.el8 ceph 52 M 2023-12-01T22:07:13.533 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2023-12-01T22:07:13.533 INFO:teuthology.orchestra.run.smithi029.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2023-12-01T22:07:13.533 INFO:teuthology.orchestra.run.smithi029.stdout: libcephsqlite x86_64 2:16.2.4-0.el8 ceph 169 k 2023-12-01T22:07:13.533 INFO:teuthology.orchestra.run.smithi029.stdout: oniguruma x86_64 6.8.2-2.el8 appstream 187 k 2023-12-01T22:07:13.533 INFO:teuthology.orchestra.run.smithi029.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2023-12-01T22:07:13.533 INFO:teuthology.orchestra.run.smithi029.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2023-12-01T22:07:13.533 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-01T22:07:13.533 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2023-12-01T22:07:13.533 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-12-01T22:07:13.533 INFO:teuthology.orchestra.run.smithi029.stdout:Install 6 Packages 2023-12-01T22:07:13.534 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-01T22:07:13.534 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 53 M 2023-12-01T22:07:13.534 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 229 M 2023-12-01T22:07:13.534 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2023-12-01T22:07:13.566 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/6 2023-12-01T22:07:13.791 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2023-12-01T22:07:13.795 INFO:teuthology.orchestra.run.smithi029.stdout:(1/6): libcephsqlite-16.2.4-0.el8.x86_64.rpm 2.4 MB/s | 169 kB 00:00 2023-12-01T22:07:13.937 INFO:teuthology.orchestra.run.smithi029.stdout:(2/6): jq-1.6-8.el8.x86_64.rpm 968 kB/s | 203 kB 00:00 2023-12-01T22:07:13.950 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2023-12-01T22:07:13.996 INFO:teuthology.orchestra.run.smithi029.stdout:(3/6): oniguruma-6.8.2-2.el8.x86_64.rpm 934 kB/s | 187 kB 00:00 2023-12-01T22:07:14.056 INFO:teuthology.orchestra.run.smithi029.stdout:(4/6): socat-1.7.4.1-1.el8.x86_64.rpm 2.7 MB/s | 323 kB 00:00 2023-12-01T22:07:14.082 INFO:teuthology.orchestra.run.smithi029.stdout:(5/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 825 kB/s | 71 kB 00:00 2023-12-01T22:07:15.007 INFO:teuthology.orchestra.run.smithi029.stdout:(6/6): ceph-test-16.2.4-0.el8.x86_64.rpm 41 MB/s | 52 MB 00:01 2023-12-01T22:07:15.009 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2023-12-01T22:07:15.010 INFO:teuthology.orchestra.run.smithi029.stdout:Total 36 MB/s | 53 MB 00:01 2023-12-01T22:07:15.057 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2023-12-01T22:07:15.080 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2023-12-01T22:07:15.081 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2023-12-01T22:07:15.417 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2023-12-01T22:07:15.420 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2023-12-01T22:07:15.931 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2023-12-01T22:07:16.180 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2023-12-01T22:07:16.385 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2023-12-01T22:07:16.596 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/6 2023-12-01T22:07:16.684 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/6 2023-12-01T22:07:16.906 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2023-12-01T22:07:17.079 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2023-12-01T22:07:18.809 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2023-12-01T22:07:19.056 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-test-2:16.2.4-0.el8.x86_64 6/6 2023-12-01T22:07:19.330 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: ceph-test-2:16.2.4-0.el8.x86_64 6/6 2023-12-01T22:07:19.330 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-test-2:16.2.4-0.el8.x86_64 1/6 2023-12-01T22:07:19.330 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : libcephsqlite-2:16.2.4-0.el8.x86_64 2/6 2023-12-01T22:07:19.330 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2023-12-01T22:07:19.330 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/6 2023-12-01T22:07:19.331 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2023-12-01T22:07:20.056 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2023-12-01T22:07:20.056 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:07:20.056 INFO:teuthology.orchestra.run.smithi073.stdout:Installed: 2023-12-01T22:07:20.057 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-test-2:16.2.4-0.el8.x86_64 jq-1.6-8.el8.x86_64 2023-12-01T22:07:20.057 INFO:teuthology.orchestra.run.smithi073.stdout: libcephsqlite-2:16.2.4-0.el8.x86_64 oniguruma-6.8.2-2.el8.x86_64 2023-12-01T22:07:20.057 INFO:teuthology.orchestra.run.smithi073.stdout: socat-1.7.4.1-1.el8.x86_64 xmlstarlet-1.6.1-20.el8.x86_64 2023-12-01T22:07:20.057 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:07:20.057 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2023-12-01T22:07:20.273 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install ceph 2023-12-01T22:07:21.131 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:00:54 ago on Fri 01 Dec 2023 10:06:27 PM UTC. 2023-12-01T22:07:22.452 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2023-12-01T22:07:22.712 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-test-2:16.2.4-0.el8.x86_64 6/6 2023-12-01T22:07:22.828 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2023-12-01T22:07:22.832 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2023-12-01T22:07:22.832 INFO:teuthology.orchestra.run.smithi073.stdout: Package Arch Version Repository Size 2023-12-01T22:07:22.832 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2023-12-01T22:07:22.832 INFO:teuthology.orchestra.run.smithi073.stdout:Installing: 2023-12-01T22:07:22.832 INFO:teuthology.orchestra.run.smithi073.stdout: ceph x86_64 2:16.2.4-0.el8 ceph 6.3 k 2023-12-01T22:07:22.832 INFO:teuthology.orchestra.run.smithi073.stdout:Installing dependencies: 2023-12-01T22:07:22.832 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-mds x86_64 2:16.2.4-0.el8 ceph 2.4 M 2023-12-01T22:07:22.832 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-mgr x86_64 2:16.2.4-0.el8 ceph 1.6 M 2023-12-01T22:07:22.832 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-mgr-modules-core noarch 2:16.2.4-0.el8 ceph-noarch 220 k 2023-12-01T22:07:22.832 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-mon x86_64 2:16.2.4-0.el8 ceph 5.0 M 2023-12-01T22:07:22.833 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-osd x86_64 2:16.2.4-0.el8 ceph 18 M 2023-12-01T22:07:22.833 INFO:teuthology.orchestra.run.smithi073.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-12-01T22:07:22.833 INFO:teuthology.orchestra.run.smithi073.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-12-01T22:07:22.833 INFO:teuthology.orchestra.run.smithi073.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2023-12-01T22:07:22.833 INFO:teuthology.orchestra.run.smithi073.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-12-01T22:07:22.833 INFO:teuthology.orchestra.run.smithi073.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-12-01T22:07:22.833 INFO:teuthology.orchestra.run.smithi073.stdout: python3-cryptography x86_64 3.2.1-6.el8 baseos 641 k 2023-12-01T22:07:22.833 INFO:teuthology.orchestra.run.smithi073.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-12-01T22:07:22.833 INFO:teuthology.orchestra.run.smithi073.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-12-01T22:07:22.833 INFO:teuthology.orchestra.run.smithi073.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-12-01T22:07:22.833 INFO:teuthology.orchestra.run.smithi073.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2023-12-01T22:07:22.834 INFO:teuthology.orchestra.run.smithi073.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2023-12-01T22:07:22.834 INFO:teuthology.orchestra.run.smithi073.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-12-01T22:07:22.834 INFO:teuthology.orchestra.run.smithi073.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-12-01T22:07:22.834 INFO:teuthology.orchestra.run.smithi073.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-12-01T22:07:22.834 INFO:teuthology.orchestra.run.smithi073.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2023-12-01T22:07:22.834 INFO:teuthology.orchestra.run.smithi073.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2023-12-01T22:07:22.834 INFO:teuthology.orchestra.run.smithi073.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2023-12-01T22:07:22.834 INFO:teuthology.orchestra.run.smithi073.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-12-01T22:07:22.834 INFO:teuthology.orchestra.run.smithi073.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-12-01T22:07:22.834 INFO:teuthology.orchestra.run.smithi073.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-12-01T22:07:22.834 INFO:teuthology.orchestra.run.smithi073.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-12-01T22:07:22.835 INFO:teuthology.orchestra.run.smithi073.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-12-01T22:07:22.835 INFO:teuthology.orchestra.run.smithi073.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-12-01T22:07:22.835 INFO:teuthology.orchestra.run.smithi073.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-12-01T22:07:22.835 INFO:teuthology.orchestra.run.smithi073.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2023-12-01T22:07:22.835 INFO:teuthology.orchestra.run.smithi073.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-12-01T22:07:22.835 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:07:22.835 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction Summary 2023-12-01T22:07:22.835 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2023-12-01T22:07:22.835 INFO:teuthology.orchestra.run.smithi073.stdout:Install 32 Packages 2023-12-01T22:07:22.835 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:07:22.836 INFO:teuthology.orchestra.run.smithi073.stdout:Total download size: 31 M 2023-12-01T22:07:22.836 INFO:teuthology.orchestra.run.smithi073.stdout:Installed size: 111 M 2023-12-01T22:07:22.836 INFO:teuthology.orchestra.run.smithi073.stdout:Downloading Packages: 2023-12-01T22:07:23.038 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-test-2:16.2.4-0.el8.x86_64 6/6 2023-12-01T22:07:23.038 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-test-2:16.2.4-0.el8.x86_64 1/6 2023-12-01T22:07:23.039 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libcephsqlite-2:16.2.4-0.el8.x86_64 2/6 2023-12-01T22:07:23.039 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2023-12-01T22:07:23.039 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/6 2023-12-01T22:07:23.039 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2023-12-01T22:07:23.480 INFO:teuthology.orchestra.run.smithi073.stdout:(1/32): ceph-16.2.4-0.el8.x86_64.rpm 99 kB/s | 6.3 kB 00:00 2023-12-01T22:07:23.558 INFO:teuthology.orchestra.run.smithi073.stdout:(2/32): ceph-mds-16.2.4-0.el8.x86_64.rpm 17 MB/s | 2.4 MB 00:00 2023-12-01T22:07:23.605 INFO:teuthology.orchestra.run.smithi073.stdout:(3/32): ceph-mgr-16.2.4-0.el8.x86_64.rpm 8.3 MB/s | 1.6 MB 00:00 2023-12-01T22:07:23.647 INFO:teuthology.orchestra.run.smithi073.stdout:(4/32): ceph-mgr-modules-core-16.2.4-0.el8.noar 5.2 MB/s | 220 kB 00:00 2023-12-01T22:07:23.731 INFO:teuthology.orchestra.run.smithi073.stdout:(5/32): ceph-mon-16.2.4-0.el8.x86_64.rpm 20 MB/s | 5.0 MB 00:00 2023-12-01T22:07:23.842 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2023-12-01T22:07:23.842 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-01T22:07:23.842 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2023-12-01T22:07:23.842 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-test-2:16.2.4-0.el8.x86_64 jq-1.6-8.el8.x86_64 2023-12-01T22:07:23.842 INFO:teuthology.orchestra.run.smithi029.stdout: libcephsqlite-2:16.2.4-0.el8.x86_64 oniguruma-6.8.2-2.el8.x86_64 2023-12-01T22:07:23.842 INFO:teuthology.orchestra.run.smithi029.stdout: socat-1.7.4.1-1.el8.x86_64 xmlstarlet-1.6.1-20.el8.x86_64 2023-12-01T22:07:23.842 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-01T22:07:23.842 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-12-01T22:07:23.948 INFO:teuthology.orchestra.run.smithi073.stdout:(6/32): ceph-osd-16.2.4-0.el8.x86_64.rpm 47 MB/s | 18 MB 00:00 2023-12-01T22:07:23.974 INFO:teuthology.orchestra.run.smithi073.stdout:(7/32): python3-markupsafe-0.23-19.el8.x86_64.r 160 kB/s | 39 kB 00:00 2023-12-01T22:07:23.999 INFO:teuthology.orchestra.run.smithi073.stdout:(8/32): python3-mako-1.0.6-14.el8.noarch.rpm 447 kB/s | 157 kB 00:00 2023-12-01T22:07:24.066 INFO:teuthology.orchestra.run.smithi073.stdout:(9/32): python3-pytz-2017.2-11.el8.noarch.rpm 620 kB/s | 57 kB 00:00 2023-12-01T22:07:24.077 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph 2023-12-01T22:07:24.141 INFO:teuthology.orchestra.run.smithi073.stdout:(10/32): python3-pyOpenSSL-19.0.0-1.el8.noarch. 538 kB/s | 103 kB 00:00 2023-12-01T22:07:24.191 INFO:teuthology.orchestra.run.smithi073.stdout:(11/32): python3-werkzeug-0.12.2-4.el8.noarch.r 2.3 MB/s | 457 kB 00:00 2023-12-01T22:07:24.592 INFO:teuthology.orchestra.run.smithi073.stdout:(12/32): python3-pycparser-2.14-14.el8.noarch.r 273 kB/s | 109 kB 00:00 2023-12-01T22:07:24.675 INFO:teuthology.orchestra.run.smithi073.stdout:(13/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 408 kB/s | 249 kB 00:00 2023-12-01T22:07:24.800 INFO:teuthology.orchestra.run.smithi073.stdout:(14/32): python3-cryptography-3.2.1-6.el8.x86_6 973 kB/s | 641 kB 00:00 2023-12-01T22:07:24.976 INFO:teuthology.orchestra.run.smithi073.stdout:(15/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 114 kB/s | 44 kB 00:00 2023-12-01T22:07:25.076 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:01:03 ago on Fri 01 Dec 2023 10:06:22 PM UTC. 2023-12-01T22:07:25.401 INFO:teuthology.orchestra.run.smithi073.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 255 kB/s | 185 kB 00:00 2023-12-01T22:07:25.526 INFO:teuthology.orchestra.run.smithi073.stdout:(17/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 238 kB/s | 173 kB 00:00 2023-12-01T22:07:25.545 INFO:teuthology.orchestra.run.smithi073.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 78 kB/s | 11 kB 00:00 2023-12-01T22:07:25.666 INFO:teuthology.orchestra.run.smithi073.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 126 kB/s | 18 kB 00:00 2023-12-01T22:07:25.694 INFO:teuthology.orchestra.run.smithi073.stdout:(20/32): python3-cherrypy-18.4.0-1.el8.noarch.r 536 kB/s | 384 kB 00:00 2023-12-01T22:07:25.711 INFO:teuthology.orchestra.run.smithi073.stdout:(21/32): python3-logutils-0.3.5-11.el8.noarch.r 291 kB/s | 49 kB 00:00 2023-12-01T22:07:25.770 INFO:teuthology.orchestra.run.smithi073.stdout:(22/32): python3-more-itertools-7.2.0-3.el8.noa 569 kB/s | 59 kB 00:00 2023-12-01T22:07:25.820 INFO:teuthology.orchestra.run.smithi073.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 143 kB/s | 16 kB 00:00 2023-12-01T22:07:25.879 INFO:teuthology.orchestra.run.smithi073.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 172 kB/s | 19 kB 00:00 2023-12-01T22:07:25.929 INFO:teuthology.orchestra.run.smithi073.stdout:(25/32): python3-singledispatch-3.4.0.3-18.el8. 220 kB/s | 24 kB 00:00 2023-12-01T22:07:25.987 INFO:teuthology.orchestra.run.smithi073.stdout:(26/32): python3-tempora-1.14.1-5.el8.noarch.rp 270 kB/s | 29 kB 00:00 2023-12-01T22:07:26.038 INFO:teuthology.orchestra.run.smithi073.stdout:(27/32): python3-trustme-0.6.0-4.el8.noarch.rpm 253 kB/s | 27 kB 00:00 2023-12-01T22:07:26.088 INFO:teuthology.orchestra.run.smithi073.stdout:(28/32): python3-pecan-1.3.2-9.el8.noarch.rpm 718 kB/s | 283 kB 00:00 2023-12-01T22:07:26.297 INFO:teuthology.orchestra.run.smithi073.stdout:(29/32): python3-webtest-2.0.33-1.el8.noarch.rp 410 kB/s | 85 kB 00:00 2023-12-01T22:07:26.413 INFO:teuthology.orchestra.run.smithi073.stdout:(30/32): python3-waitress-1.4.3-1.el8.noarch.rp 600 kB/s | 255 kB 00:00 2023-12-01T22:07:26.430 INFO:teuthology.orchestra.run.smithi073.stdout:(31/32): python3-zc-lockfile-2.0-2.el8.noarch.r 172 kB/s | 23 kB 00:00 2023-12-01T22:07:26.455 INFO:teuthology.orchestra.run.smithi073.stdout:(32/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 602 kB/s | 251 kB 00:00 2023-12-01T22:07:26.457 INFO:teuthology.orchestra.run.smithi073.stdout:-------------------------------------------------------------------------------- 2023-12-01T22:07:26.457 INFO:teuthology.orchestra.run.smithi073.stdout:Total 8.6 MB/s | 31 MB 00:03 2023-12-01T22:07:26.544 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction check 2023-12-01T22:07:26.586 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction check succeeded. 2023-12-01T22:07:26.586 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction test 2023-12-01T22:07:26.910 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction test succeeded. 2023-12-01T22:07:26.913 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction 2023-12-01T22:07:26.972 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-12-01T22:07:26.976 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-12-01T22:07:26.976 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2023-12-01T22:07:26.976 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-12-01T22:07:26.976 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2023-12-01T22:07:26.976 INFO:teuthology.orchestra.run.smithi029.stdout: ceph x86_64 2:16.2.4-0.el8 ceph 6.3 k 2023-12-01T22:07:26.976 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2023-12-01T22:07:26.976 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mds x86_64 2:16.2.4-0.el8 ceph 2.4 M 2023-12-01T22:07:26.976 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr x86_64 2:16.2.4-0.el8 ceph 1.6 M 2023-12-01T22:07:26.976 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-modules-core noarch 2:16.2.4-0.el8 ceph-noarch 220 k 2023-12-01T22:07:26.977 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mon x86_64 2:16.2.4-0.el8 ceph 5.0 M 2023-12-01T22:07:26.977 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-osd x86_64 2:16.2.4-0.el8 ceph 18 M 2023-12-01T22:07:26.977 INFO:teuthology.orchestra.run.smithi029.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-12-01T22:07:26.977 INFO:teuthology.orchestra.run.smithi029.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-12-01T22:07:26.977 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2023-12-01T22:07:26.977 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-12-01T22:07:26.977 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-12-01T22:07:26.977 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cryptography x86_64 3.2.1-6.el8 baseos 641 k 2023-12-01T22:07:26.977 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-12-01T22:07:26.977 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-12-01T22:07:26.977 INFO:teuthology.orchestra.run.smithi029.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-12-01T22:07:26.977 INFO:teuthology.orchestra.run.smithi029.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2023-12-01T22:07:26.977 INFO:teuthology.orchestra.run.smithi029.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2023-12-01T22:07:26.977 INFO:teuthology.orchestra.run.smithi029.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-12-01T22:07:26.978 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-12-01T22:07:26.978 INFO:teuthology.orchestra.run.smithi029.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-12-01T22:07:26.978 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2023-12-01T22:07:26.978 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2023-12-01T22:07:26.978 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2023-12-01T22:07:26.978 INFO:teuthology.orchestra.run.smithi029.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-12-01T22:07:26.978 INFO:teuthology.orchestra.run.smithi029.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-12-01T22:07:26.978 INFO:teuthology.orchestra.run.smithi029.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-12-01T22:07:26.978 INFO:teuthology.orchestra.run.smithi029.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-12-01T22:07:26.981 INFO:teuthology.orchestra.run.smithi029.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-12-01T22:07:26.981 INFO:teuthology.orchestra.run.smithi029.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-12-01T22:07:26.981 INFO:teuthology.orchestra.run.smithi029.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-12-01T22:07:26.981 INFO:teuthology.orchestra.run.smithi029.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2023-12-01T22:07:26.982 INFO:teuthology.orchestra.run.smithi029.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-12-01T22:07:26.982 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-01T22:07:26.982 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2023-12-01T22:07:26.982 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-12-01T22:07:26.982 INFO:teuthology.orchestra.run.smithi029.stdout:Install 32 Packages 2023-12-01T22:07:26.982 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-01T22:07:26.983 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 31 M 2023-12-01T22:07:26.983 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 111 M 2023-12-01T22:07:26.983 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2023-12-01T22:07:27.658 INFO:teuthology.orchestra.run.smithi029.stdout:(1/32): ceph-16.2.4-0.el8.x86_64.rpm 106 kB/s | 6.3 kB 00:00 2023-12-01T22:07:27.726 INFO:teuthology.orchestra.run.smithi029.stdout:(2/32): ceph-mgr-16.2.4-0.el8.x86_64.rpm 12 MB/s | 1.6 MB 00:00 2023-12-01T22:07:27.802 INFO:teuthology.orchestra.run.smithi029.stdout:(3/32): ceph-mds-16.2.4-0.el8.x86_64.rpm 12 MB/s | 2.4 MB 00:00 2023-12-01T22:07:27.835 INFO:teuthology.orchestra.run.smithi029.stdout:(4/32): ceph-mgr-modules-core-16.2.4-0.el8.noar 6.4 MB/s | 220 kB 00:00 2023-12-01T22:07:27.961 INFO:teuthology.orchestra.run.smithi029.stdout:(5/32): ceph-mon-16.2.4-0.el8.x86_64.rpm 16 MB/s | 5.0 MB 00:00 2023-12-01T22:07:28.055 INFO:teuthology.orchestra.run.smithi073.stdout: Preparing : 1/1 2023-12-01T22:07:28.278 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2023-12-01T22:07:28.338 INFO:teuthology.orchestra.run.smithi029.stdout:(6/32): ceph-osd-16.2.4-0.el8.x86_64.rpm 30 MB/s | 18 MB 00:00 2023-12-01T22:07:28.364 INFO:teuthology.orchestra.run.smithi029.stdout:(7/32): python3-markupsafe-0.23-19.el8.x86_64.r 96 kB/s | 39 kB 00:00 2023-12-01T22:07:28.389 INFO:teuthology.orchestra.run.smithi029.stdout:(8/32): python3-mako-1.0.6-14.el8.noarch.rpm 284 kB/s | 157 kB 00:00 2023-12-01T22:07:28.448 INFO:teuthology.orchestra.run.smithi029.stdout:(9/32): python3-pytz-2017.2-11.el8.noarch.rpm 679 kB/s | 57 kB 00:00 2023-12-01T22:07:28.507 INFO:teuthology.orchestra.run.smithi029.stdout:(10/32): python3-werkzeug-0.12.2-4.el8.noarch.r 3.8 MB/s | 457 kB 00:00 2023-12-01T22:07:28.531 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2023-12-01T22:07:28.541 INFO:teuthology.orchestra.run.smithi029.stdout:(11/32): python3-pyOpenSSL-19.0.0-1.el8.noarch. 511 kB/s | 103 kB 00:00 2023-12-01T22:07:28.788 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2023-12-01T22:07:28.996 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2023-12-01T22:07:29.202 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2023-12-01T22:07:29.258 INFO:teuthology.orchestra.run.smithi029.stdout:(12/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 307 kB/s | 249 kB 00:00 2023-12-01T22:07:29.350 INFO:teuthology.orchestra.run.smithi029.stdout:(13/32): python3-pycparser-2.14-14.el8.noarch.r 135 kB/s | 109 kB 00:00 2023-12-01T22:07:29.376 INFO:teuthology.orchestra.run.smithi029.stdout:(14/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 371 kB/s | 44 kB 00:00 2023-12-01T22:07:29.382 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2023-12-01T22:07:29.576 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2023-12-01T22:07:29.643 INFO:teuthology.orchestra.run.smithi029.stdout:(15/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 647 kB/s | 173 kB 00:00 2023-12-01T22:07:29.753 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2023-12-01T22:07:29.760 INFO:teuthology.orchestra.run.smithi029.stdout:(16/32): python3-cherrypy-18.4.0-1.el8.noarch.r 3.2 MB/s | 384 kB 00:00 2023-12-01T22:07:29.960 INFO:teuthology.orchestra.run.smithi029.stdout:(17/32): python3-jaraco-6.2-6.el8.noarch.rpm 55 kB/s | 11 kB 00:00 2023-12-01T22:07:29.977 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2023-12-01T22:07:30.103 INFO:teuthology.orchestra.run.smithi029.stdout:(18/32): python3-jaraco-functools-2.0-4.el8.noa 124 kB/s | 18 kB 00:00 2023-12-01T22:07:30.153 INFO:teuthology.orchestra.run.smithi029.stdout:(19/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 231 kB/s | 185 kB 00:00 2023-12-01T22:07:30.181 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2023-12-01T22:07:30.246 INFO:teuthology.orchestra.run.smithi029.stdout:(20/32): python3-more-itertools-7.2.0-3.el8.noa 638 kB/s | 59 kB 00:00 2023-12-01T22:07:30.281 INFO:teuthology.orchestra.run.smithi029.stdout:(21/32): python3-logutils-0.3.5-11.el8.noarch.r 274 kB/s | 49 kB 00:00 2023-12-01T22:07:30.403 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2023-12-01T22:07:30.632 INFO:teuthology.orchestra.run.smithi029.stdout:(22/32): python3-portend-2.6-1.el8.noarch.rpm 44 kB/s | 16 kB 00:00 2023-12-01T22:07:30.660 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2023-12-01T22:07:30.807 INFO:teuthology.orchestra.run.smithi029.stdout:(23/32): python3-cryptography-3.2.1-6.el8.x86_6 279 kB/s | 641 kB 00:02 2023-12-01T22:07:30.958 INFO:teuthology.orchestra.run.smithi029.stdout:(24/32): python3-singledispatch-3.4.0.3-18.el8. 159 kB/s | 24 kB 00:00 2023-12-01T22:07:31.024 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2023-12-01T22:07:31.325 INFO:teuthology.orchestra.run.smithi029.stdout:(25/32): python3-simplegeneric-0.8.1-17.el8.noa 27 kB/s | 19 kB 00:00 2023-12-01T22:07:31.325 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-cryptography-3.2.1-6.el8.x86_64 14/32 2023-12-01T22:07:31.489 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2023-12-01T22:07:31.684 INFO:teuthology.orchestra.run.smithi029.stdout:(26/32): python3-trustme-0.6.0-4.el8.noarch.rpm 77 kB/s | 27 kB 00:00 2023-12-01T22:07:31.738 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2023-12-01T22:07:31.834 INFO:teuthology.orchestra.run.smithi029.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 33 kB/s | 29 kB 00:00 2023-12-01T22:07:31.901 INFO:teuthology.orchestra.run.smithi029.stdout:(28/32): python3-waitress-1.4.3-1.el8.noarch.rp 1.1 MB/s | 255 kB 00:00 2023-12-01T22:07:31.923 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-12-01T22:07:32.118 INFO:teuthology.orchestra.run.smithi029.stdout:(29/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 885 kB/s | 251 kB 00:00 2023-12-01T22:07:32.144 INFO:teuthology.orchestra.run.smithi029.stdout:(30/32): python3-webtest-2.0.33-1.el8.noarch.rp 352 kB/s | 85 kB 00:00 2023-12-01T22:07:32.177 INFO:teuthology.orchestra.run.smithi029.stdout:(31/32): python3-zc-lockfile-2.0-2.el8.noarch.r 390 kB/s | 23 kB 00:00 2023-12-01T22:07:32.249 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2023-12-01T22:07:32.461 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2023-12-01T22:07:32.642 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2023-12-01T22:07:32.644 INFO:teuthology.orchestra.run.smithi029.stdout:(32/32): python3-pecan-1.3.2-9.el8.noarch.rpm 118 kB/s | 283 kB 00:02 2023-12-01T22:07:32.646 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2023-12-01T22:07:32.646 INFO:teuthology.orchestra.run.smithi029.stdout:Total 5.5 MB/s | 31 MB 00:05 2023-12-01T22:07:32.737 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2023-12-01T22:07:32.780 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2023-12-01T22:07:32.781 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2023-12-01T22:07:32.868 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2023-12-01T22:07:33.115 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2023-12-01T22:07:33.120 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2023-12-01T22:07:33.123 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2023-12-01T22:07:33.377 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2023-12-01T22:07:33.609 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2023-12-01T22:07:33.936 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2023-12-01T22:07:34.266 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2023-12-01T22:07:34.277 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2023-12-01T22:07:34.519 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2023-12-01T22:07:34.674 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 27/32 2023-12-01T22:07:34.730 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2023-12-01T22:07:34.945 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2023-12-01T22:07:35.025 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2023-12-01T22:07:35.252 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2023-12-01T22:07:35.458 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2023-12-01T22:07:35.666 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2023-12-01T22:07:35.856 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2023-12-01T22:07:36.051 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2023-12-01T22:07:36.289 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2023-12-01T22:07:36.474 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2023-12-01T22:07:36.534 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2023-12-01T22:07:36.636 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2023-12-01T22:07:36.763 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2023-12-01T22:07:37.013 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2023-12-01T22:07:37.072 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2023-12-01T22:07:37.286 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2023-12-01T22:07:37.421 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2023-12-01T22:07:37.509 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2023-12-01T22:07:37.693 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cryptography-3.2.1-6.el8.x86_64 14/32 2023-12-01T22:07:37.730 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2023-12-01T22:07:37.748 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2023-12-01T22:07:37.879 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2023-12-01T22:07:37.888 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-2:16.2.4-0.el8.x86_64 32/32 2023-12-01T22:07:38.163 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2023-12-01T22:07:38.302 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: ceph-2:16.2.4-0.el8.x86_64 32/32 2023-12-01T22:07:38.303 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-2:16.2.4-0.el8.x86_64 1/32 2023-12-01T22:07:38.303 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-mds-2:16.2.4-0.el8.x86_64 2/32 2023-12-01T22:07:38.303 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-mgr-2:16.2.4-0.el8.x86_64 3/32 2023-12-01T22:07:38.303 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-mon-2:16.2.4-0.el8.x86_64 4/32 2023-12-01T22:07:38.303 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-osd-2:16.2.4-0.el8.x86_64 5/32 2023-12-01T22:07:38.303 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 6/32 2023-12-01T22:07:38.303 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2023-12-01T22:07:38.303 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2023-12-01T22:07:38.303 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2023-12-01T22:07:38.303 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2023-12-01T22:07:38.303 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2023-12-01T22:07:38.303 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2023-12-01T22:07:38.303 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-cryptography-3.2.1-6.el8.x86_64 13/32 2023-12-01T22:07:38.304 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2023-12-01T22:07:38.304 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2023-12-01T22:07:38.304 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2023-12-01T22:07:38.304 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-12-01T22:07:38.304 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2023-12-01T22:07:38.304 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2023-12-01T22:07:38.304 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2023-12-01T22:07:38.304 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2023-12-01T22:07:38.304 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2023-12-01T22:07:38.304 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2023-12-01T22:07:38.304 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2023-12-01T22:07:38.304 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2023-12-01T22:07:38.304 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2023-12-01T22:07:38.305 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2023-12-01T22:07:38.305 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2023-12-01T22:07:38.305 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2023-12-01T22:07:38.305 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2023-12-01T22:07:38.305 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2023-12-01T22:07:38.389 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-12-01T22:07:38.718 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2023-12-01T22:07:38.936 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2023-12-01T22:07:39.132 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2023-12-01T22:07:39.354 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2023-12-01T22:07:39.508 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2023-12-01T22:07:39.508 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:07:39.508 INFO:teuthology.orchestra.run.smithi073.stdout:Installed: 2023-12-01T22:07:39.509 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-2:16.2.4-0.el8.x86_64 2023-12-01T22:07:39.509 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-mds-2:16.2.4-0.el8.x86_64 2023-12-01T22:07:39.509 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-mgr-2:16.2.4-0.el8.x86_64 2023-12-01T22:07:39.509 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 2023-12-01T22:07:39.509 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-mon-2:16.2.4-0.el8.x86_64 2023-12-01T22:07:39.509 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-osd-2:16.2.4-0.el8.x86_64 2023-12-01T22:07:39.509 INFO:teuthology.orchestra.run.smithi073.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-12-01T22:07:39.509 INFO:teuthology.orchestra.run.smithi073.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-12-01T22:07:39.509 INFO:teuthology.orchestra.run.smithi073.stdout: python3-cffi-1.11.5-6.el8.x86_64 2023-12-01T22:07:39.509 INFO:teuthology.orchestra.run.smithi073.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-12-01T22:07:39.509 INFO:teuthology.orchestra.run.smithi073.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-12-01T22:07:39.509 INFO:teuthology.orchestra.run.smithi073.stdout: python3-cryptography-3.2.1-6.el8.x86_64 2023-12-01T22:07:39.509 INFO:teuthology.orchestra.run.smithi073.stdout: python3-jaraco-6.2-6.el8.noarch 2023-12-01T22:07:39.509 INFO:teuthology.orchestra.run.smithi073.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-12-01T22:07:39.510 INFO:teuthology.orchestra.run.smithi073.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-12-01T22:07:39.510 INFO:teuthology.orchestra.run.smithi073.stdout: python3-mako-1.0.6-14.el8.noarch 2023-12-01T22:07:39.510 INFO:teuthology.orchestra.run.smithi073.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-12-01T22:07:39.510 INFO:teuthology.orchestra.run.smithi073.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-12-01T22:07:39.510 INFO:teuthology.orchestra.run.smithi073.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-12-01T22:07:39.510 INFO:teuthology.orchestra.run.smithi073.stdout: python3-portend-2.6-1.el8.noarch 2023-12-01T22:07:39.510 INFO:teuthology.orchestra.run.smithi073.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-12-01T22:07:39.510 INFO:teuthology.orchestra.run.smithi073.stdout: python3-pycparser-2.14-14.el8.noarch 2023-12-01T22:07:39.510 INFO:teuthology.orchestra.run.smithi073.stdout: python3-pytz-2017.2-11.el8.noarch 2023-12-01T22:07:39.510 INFO:teuthology.orchestra.run.smithi073.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-12-01T22:07:39.510 INFO:teuthology.orchestra.run.smithi073.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-12-01T22:07:39.510 INFO:teuthology.orchestra.run.smithi073.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-12-01T22:07:39.510 INFO:teuthology.orchestra.run.smithi073.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-12-01T22:07:39.510 INFO:teuthology.orchestra.run.smithi073.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-12-01T22:07:39.511 INFO:teuthology.orchestra.run.smithi073.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-12-01T22:07:39.511 INFO:teuthology.orchestra.run.smithi073.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-12-01T22:07:39.511 INFO:teuthology.orchestra.run.smithi073.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-12-01T22:07:39.511 INFO:teuthology.orchestra.run.smithi073.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-12-01T22:07:39.511 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:07:39.511 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2023-12-01T22:07:39.641 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2023-12-01T22:07:39.908 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2023-12-01T22:07:40.165 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2023-12-01T22:07:40.530 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2023-12-01T22:07:40.580 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install ceph-base 2023-12-01T22:07:40.890 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2023-12-01T22:07:41.438 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:01:14 ago on Fri 01 Dec 2023 10:06:27 PM UTC. 2023-12-01T22:07:41.888 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 27/32 2023-12-01T22:07:42.607 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2023-12-01T22:07:42.879 INFO:teuthology.orchestra.run.smithi073.stdout:Package ceph-base-2:16.2.4-0.el8.x86_64 is already installed. 2023-12-01T22:07:43.107 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2023-12-01T22:07:43.108 INFO:teuthology.orchestra.run.smithi073.stdout:Nothing to do. 2023-12-01T22:07:43.109 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2023-12-01T22:07:43.200 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install cephadm 2023-12-01T22:07:44.061 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:01:17 ago on Fri 01 Dec 2023 10:06:27 PM UTC. 2023-12-01T22:07:44.300 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2023-12-01T22:07:44.532 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2023-12-01T22:07:45.013 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2023-12-01T22:07:45.265 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2023-12-01T22:07:45.515 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2023-12-01T22:07:45.731 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2023-12-01T22:07:45.731 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2023-12-01T22:07:45.732 INFO:teuthology.orchestra.run.smithi073.stdout: Package Architecture Version Repository Size 2023-12-01T22:07:45.732 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2023-12-01T22:07:45.732 INFO:teuthology.orchestra.run.smithi073.stdout:Installing: 2023-12-01T22:07:45.732 INFO:teuthology.orchestra.run.smithi073.stdout: cephadm noarch 2:16.2.4-0.el8 ceph-noarch 71 k 2023-12-01T22:07:45.732 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:07:45.732 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction Summary 2023-12-01T22:07:45.732 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2023-12-01T22:07:45.732 INFO:teuthology.orchestra.run.smithi073.stdout:Install 1 Package 2023-12-01T22:07:45.732 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:07:45.733 INFO:teuthology.orchestra.run.smithi073.stdout:Total download size: 71 k 2023-12-01T22:07:45.733 INFO:teuthology.orchestra.run.smithi073.stdout:Installed size: 290 k 2023-12-01T22:07:45.733 INFO:teuthology.orchestra.run.smithi073.stdout:Downloading Packages: 2023-12-01T22:07:45.735 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2023-12-01T22:07:45.759 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2023-12-01T22:07:45.911 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-2:16.2.4-0.el8.x86_64 32/32 2023-12-01T22:07:46.045 INFO:teuthology.orchestra.run.smithi073.stdout:cephadm-16.2.4-0.el8.noarch.rpm 228 kB/s | 71 kB 00:00 2023-12-01T22:07:46.046 INFO:teuthology.orchestra.run.smithi073.stdout:-------------------------------------------------------------------------------- 2023-12-01T22:07:46.046 INFO:teuthology.orchestra.run.smithi073.stdout:Total 227 kB/s | 71 kB 00:00 2023-12-01T22:07:46.046 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction check 2023-12-01T22:07:46.053 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction check succeeded. 2023-12-01T22:07:46.053 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction test 2023-12-01T22:07:46.055 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction test succeeded. 2023-12-01T22:07:46.058 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction 2023-12-01T22:07:46.376 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-2:16.2.4-0.el8.x86_64 32/32 2023-12-01T22:07:46.376 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-2:16.2.4-0.el8.x86_64 1/32 2023-12-01T22:07:46.376 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mds-2:16.2.4-0.el8.x86_64 2/32 2023-12-01T22:07:46.376 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-2:16.2.4-0.el8.x86_64 3/32 2023-12-01T22:07:46.376 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mon-2:16.2.4-0.el8.x86_64 4/32 2023-12-01T22:07:46.377 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-osd-2:16.2.4-0.el8.x86_64 5/32 2023-12-01T22:07:46.377 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 6/32 2023-12-01T22:07:46.377 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2023-12-01T22:07:46.377 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2023-12-01T22:07:46.377 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2023-12-01T22:07:46.377 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2023-12-01T22:07:46.377 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2023-12-01T22:07:46.377 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2023-12-01T22:07:46.377 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cryptography-3.2.1-6.el8.x86_64 13/32 2023-12-01T22:07:46.377 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2023-12-01T22:07:46.377 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2023-12-01T22:07:46.377 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2023-12-01T22:07:46.377 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-12-01T22:07:46.377 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2023-12-01T22:07:46.377 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2023-12-01T22:07:46.377 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2023-12-01T22:07:46.378 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2023-12-01T22:07:46.378 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2023-12-01T22:07:46.378 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2023-12-01T22:07:46.378 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2023-12-01T22:07:46.378 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2023-12-01T22:07:46.378 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2023-12-01T22:07:46.378 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2023-12-01T22:07:46.379 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2023-12-01T22:07:46.379 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2023-12-01T22:07:46.379 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2023-12-01T22:07:46.379 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2023-12-01T22:07:46.717 INFO:teuthology.orchestra.run.smithi073.stdout: Preparing : 1/1 2023-12-01T22:07:47.344 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2023-12-01T22:07:47.573 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : cephadm-2:16.2.4-0.el8.noarch 1/1 2023-12-01T22:07:47.601 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2023-12-01T22:07:47.601 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-01T22:07:47.601 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2023-12-01T22:07:47.601 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-2:16.2.4-0.el8.x86_64 2023-12-01T22:07:47.601 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mds-2:16.2.4-0.el8.x86_64 2023-12-01T22:07:47.601 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-2:16.2.4-0.el8.x86_64 2023-12-01T22:07:47.601 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 2023-12-01T22:07:47.601 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mon-2:16.2.4-0.el8.x86_64 2023-12-01T22:07:47.601 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-osd-2:16.2.4-0.el8.x86_64 2023-12-01T22:07:47.602 INFO:teuthology.orchestra.run.smithi029.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-12-01T22:07:47.602 INFO:teuthology.orchestra.run.smithi029.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-12-01T22:07:47.602 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cffi-1.11.5-6.el8.x86_64 2023-12-01T22:07:47.602 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-12-01T22:07:47.602 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-12-01T22:07:47.602 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cryptography-3.2.1-6.el8.x86_64 2023-12-01T22:07:47.602 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-6.2-6.el8.noarch 2023-12-01T22:07:47.602 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-12-01T22:07:47.602 INFO:teuthology.orchestra.run.smithi029.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-12-01T22:07:47.602 INFO:teuthology.orchestra.run.smithi029.stdout: python3-mako-1.0.6-14.el8.noarch 2023-12-01T22:07:47.602 INFO:teuthology.orchestra.run.smithi029.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-12-01T22:07:47.602 INFO:teuthology.orchestra.run.smithi029.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-12-01T22:07:47.602 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-12-01T22:07:47.602 INFO:teuthology.orchestra.run.smithi029.stdout: python3-portend-2.6-1.el8.noarch 2023-12-01T22:07:47.603 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-12-01T22:07:47.603 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pycparser-2.14-14.el8.noarch 2023-12-01T22:07:47.603 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pytz-2017.2-11.el8.noarch 2023-12-01T22:07:47.603 INFO:teuthology.orchestra.run.smithi029.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-12-01T22:07:47.603 INFO:teuthology.orchestra.run.smithi029.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-12-01T22:07:47.603 INFO:teuthology.orchestra.run.smithi029.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-12-01T22:07:47.603 INFO:teuthology.orchestra.run.smithi029.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-12-01T22:07:47.603 INFO:teuthology.orchestra.run.smithi029.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-12-01T22:07:47.603 INFO:teuthology.orchestra.run.smithi029.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-12-01T22:07:47.603 INFO:teuthology.orchestra.run.smithi029.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-12-01T22:07:47.603 INFO:teuthology.orchestra.run.smithi029.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-12-01T22:07:47.603 INFO:teuthology.orchestra.run.smithi029.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-12-01T22:07:47.603 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-01T22:07:47.603 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-12-01T22:07:47.773 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-base 2023-12-01T22:07:47.820 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2023-12-01T22:07:48.536 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : cephadm-2:16.2.4-0.el8.noarch 1/1 2023-12-01T22:07:48.537 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:07:48.537 INFO:teuthology.orchestra.run.smithi073.stdout:Installed: 2023-12-01T22:07:48.537 INFO:teuthology.orchestra.run.smithi073.stdout: cephadm-2:16.2.4-0.el8.noarch 2023-12-01T22:07:48.537 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:07:48.537 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2023-12-01T22:07:48.736 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install ceph-immutable-object-cache 2023-12-01T22:07:48.767 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:01:26 ago on Fri 01 Dec 2023 10:06:22 PM UTC. 2023-12-01T22:07:49.602 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:01:22 ago on Fri 01 Dec 2023 10:06:27 PM UTC. 2023-12-01T22:07:50.387 INFO:teuthology.orchestra.run.smithi029.stdout:Package ceph-base-2:16.2.4-0.el8.x86_64 is already installed. 2023-12-01T22:07:50.635 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-12-01T22:07:50.637 INFO:teuthology.orchestra.run.smithi029.stdout:Nothing to do. 2023-12-01T22:07:50.637 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-12-01T22:07:50.780 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install cephadm 2023-12-01T22:07:51.295 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2023-12-01T22:07:51.296 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2023-12-01T22:07:51.296 INFO:teuthology.orchestra.run.smithi073.stdout: Package Arch Version Repo Size 2023-12-01T22:07:51.296 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2023-12-01T22:07:51.296 INFO:teuthology.orchestra.run.smithi073.stdout:Installing: 2023-12-01T22:07:51.297 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-immutable-object-cache x86_64 2:16.2.4-0.el8 ceph 163 k 2023-12-01T22:07:51.297 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:07:51.297 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction Summary 2023-12-01T22:07:51.297 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2023-12-01T22:07:51.297 INFO:teuthology.orchestra.run.smithi073.stdout:Install 1 Package 2023-12-01T22:07:51.297 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:07:51.297 INFO:teuthology.orchestra.run.smithi073.stdout:Total download size: 163 k 2023-12-01T22:07:51.297 INFO:teuthology.orchestra.run.smithi073.stdout:Installed size: 587 k 2023-12-01T22:07:51.297 INFO:teuthology.orchestra.run.smithi073.stdout:Downloading Packages: 2023-12-01T22:07:51.343 INFO:teuthology.orchestra.run.smithi073.stdout:ceph-immutable-object-cache-16.2.4-0.el8.x86_64 3.4 MB/s | 163 kB 00:00 2023-12-01T22:07:51.344 INFO:teuthology.orchestra.run.smithi073.stdout:-------------------------------------------------------------------------------- 2023-12-01T22:07:51.344 INFO:teuthology.orchestra.run.smithi073.stdout:Total 3.3 MB/s | 163 kB 00:00 2023-12-01T22:07:51.344 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction check 2023-12-01T22:07:51.354 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction check succeeded. 2023-12-01T22:07:51.354 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction test 2023-12-01T22:07:51.401 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction test succeeded. 2023-12-01T22:07:51.404 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction 2023-12-01T22:07:51.517 INFO:teuthology.orchestra.run.smithi073.stdout: Preparing : 1/1 2023-12-01T22:07:51.695 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2023-12-01T22:07:51.778 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:01:29 ago on Fri 01 Dec 2023 10:06:22 PM UTC. 2023-12-01T22:07:52.105 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2023-12-01T22:07:53.286 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2023-12-01T22:07:53.286 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:07:53.286 INFO:teuthology.orchestra.run.smithi073.stdout:Installed: 2023-12-01T22:07:53.286 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 2023-12-01T22:07:53.287 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:07:53.287 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2023-12-01T22:07:53.509 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install ceph-mgr 2023-12-01T22:07:53.644 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-12-01T22:07:53.644 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-12-01T22:07:53.645 INFO:teuthology.orchestra.run.smithi029.stdout: Package Architecture Version Repository Size 2023-12-01T22:07:53.645 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-12-01T22:07:53.645 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2023-12-01T22:07:53.645 INFO:teuthology.orchestra.run.smithi029.stdout: cephadm noarch 2:16.2.4-0.el8 ceph-noarch 71 k 2023-12-01T22:07:53.645 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-01T22:07:53.645 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2023-12-01T22:07:53.645 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-12-01T22:07:53.645 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2023-12-01T22:07:53.645 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-01T22:07:53.646 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 71 k 2023-12-01T22:07:53.646 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 290 k 2023-12-01T22:07:53.646 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2023-12-01T22:07:54.195 INFO:teuthology.orchestra.run.smithi029.stdout:cephadm-16.2.4-0.el8.noarch.rpm 130 kB/s | 71 kB 00:00 2023-12-01T22:07:54.196 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2023-12-01T22:07:54.196 INFO:teuthology.orchestra.run.smithi029.stdout:Total 129 kB/s | 71 kB 00:00 2023-12-01T22:07:54.196 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2023-12-01T22:07:54.205 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2023-12-01T22:07:54.205 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2023-12-01T22:07:54.208 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2023-12-01T22:07:54.212 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2023-12-01T22:07:54.380 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:01:27 ago on Fri 01 Dec 2023 10:06:27 PM UTC. 2023-12-01T22:07:55.240 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2023-12-01T22:07:55.826 INFO:teuthology.orchestra.run.smithi073.stdout:Package ceph-mgr-2:16.2.4-0.el8.x86_64 is already installed. 2023-12-01T22:07:56.050 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2023-12-01T22:07:56.051 INFO:teuthology.orchestra.run.smithi073.stdout:Nothing to do. 2023-12-01T22:07:56.051 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2023-12-01T22:07:56.153 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install ceph-mgr-dashboard 2023-12-01T22:07:56.613 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2023-12-01T22:07:56.827 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : cephadm-2:16.2.4-0.el8.noarch 1/1 2023-12-01T22:07:57.010 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:01:30 ago on Fri 01 Dec 2023 10:06:27 PM UTC. 2023-12-01T22:07:57.090 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2023-12-01T22:07:57.813 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : cephadm-2:16.2.4-0.el8.noarch 1/1 2023-12-01T22:07:57.813 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-01T22:07:57.813 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2023-12-01T22:07:57.813 INFO:teuthology.orchestra.run.smithi029.stdout: cephadm-2:16.2.4-0.el8.noarch 2023-12-01T22:07:57.813 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-01T22:07:57.813 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-12-01T22:07:58.059 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-immutable-object-cache 2023-12-01T22:07:58.682 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2023-12-01T22:07:58.683 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2023-12-01T22:07:58.683 INFO:teuthology.orchestra.run.smithi073.stdout: Package Arch Version Repository Size 2023-12-01T22:07:58.683 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2023-12-01T22:07:58.683 INFO:teuthology.orchestra.run.smithi073.stdout:Installing: 2023-12-01T22:07:58.683 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-mgr-dashboard noarch 2:16.2.4-0.el8 ceph-noarch 2.7 M 2023-12-01T22:07:58.683 INFO:teuthology.orchestra.run.smithi073.stdout:Installing dependencies: 2023-12-01T22:07:58.683 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-grafana-dashboards noarch 2:16.2.4-0.el8 ceph-noarch 24 k 2023-12-01T22:07:58.683 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-prometheus-alerts noarch 2:16.2.4-0.el8 ceph-noarch 8.9 k 2023-12-01T22:07:58.683 INFO:teuthology.orchestra.run.smithi073.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2023-12-01T22:07:58.683 INFO:teuthology.orchestra.run.smithi073.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-12-01T22:07:58.683 INFO:teuthology.orchestra.run.smithi073.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-12-01T22:07:58.684 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:07:58.684 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction Summary 2023-12-01T22:07:58.684 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2023-12-01T22:07:58.684 INFO:teuthology.orchestra.run.smithi073.stdout:Install 6 Packages 2023-12-01T22:07:58.684 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:07:58.685 INFO:teuthology.orchestra.run.smithi073.stdout:Total download size: 3.0 M 2023-12-01T22:07:58.685 INFO:teuthology.orchestra.run.smithi073.stdout:Installed size: 15 M 2023-12-01T22:07:58.685 INFO:teuthology.orchestra.run.smithi073.stdout:Downloading Packages: 2023-12-01T22:07:59.057 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:01:37 ago on Fri 01 Dec 2023 10:06:22 PM UTC. 2023-12-01T22:07:59.113 INFO:teuthology.orchestra.run.smithi073.stdout:(1/6): ceph-grafana-dashboards-16.2.4-0.el8.noa 549 kB/s | 24 kB 00:00 2023-12-01T22:07:59.130 INFO:teuthology.orchestra.run.smithi073.stdout:(2/6): ceph-prometheus-alerts-16.2.4-0.el8.noar 147 kB/s | 8.9 kB 00:00 2023-12-01T22:07:59.180 INFO:teuthology.orchestra.run.smithi073.stdout:(3/6): ceph-mgr-dashboard-16.2.4-0.el8.noarch.r 24 MB/s | 2.7 MB 00:00 2023-12-01T22:07:59.456 INFO:teuthology.orchestra.run.smithi073.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 103 kB/s | 34 kB 00:00 2023-12-01T22:07:59.622 INFO:teuthology.orchestra.run.smithi073.stdout:(5/6): python3-jwt-1.6.1-2.el8.noarch.rpm 84 kB/s | 43 kB 00:00 2023-12-01T22:08:00.356 INFO:teuthology.orchestra.run.smithi073.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 168 kB/s | 196 kB 00:01 2023-12-01T22:08:00.359 INFO:teuthology.orchestra.run.smithi073.stdout:-------------------------------------------------------------------------------- 2023-12-01T22:08:00.359 INFO:teuthology.orchestra.run.smithi073.stdout:Total 1.8 MB/s | 3.0 MB 00:01 2023-12-01T22:08:00.371 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction check 2023-12-01T22:08:00.381 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction check succeeded. 2023-12-01T22:08:00.381 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction test 2023-12-01T22:08:00.544 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction test succeeded. 2023-12-01T22:08:00.547 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction 2023-12-01T22:08:00.822 INFO:teuthology.orchestra.run.smithi073.stdout: Preparing : 1/1 2023-12-01T22:08:00.926 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-12-01T22:08:00.927 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-12-01T22:08:00.927 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repo Size 2023-12-01T22:08:00.927 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-12-01T22:08:00.927 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2023-12-01T22:08:00.927 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-immutable-object-cache x86_64 2:16.2.4-0.el8 ceph 163 k 2023-12-01T22:08:00.927 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-01T22:08:00.927 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2023-12-01T22:08:00.928 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-12-01T22:08:00.928 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2023-12-01T22:08:00.928 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-01T22:08:00.928 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 163 k 2023-12-01T22:08:00.928 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 587 k 2023-12-01T22:08:00.928 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2023-12-01T22:08:00.975 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-immutable-object-cache-16.2.4-0.el8.x86_64 3.4 MB/s | 163 kB 00:00 2023-12-01T22:08:00.976 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2023-12-01T22:08:00.976 INFO:teuthology.orchestra.run.smithi029.stdout:Total 3.3 MB/s | 163 kB 00:00 2023-12-01T22:08:00.976 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2023-12-01T22:08:00.989 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2023-12-01T22:08:00.989 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2023-12-01T22:08:01.032 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2023-12-01T22:08:01.043 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2023-12-01T22:08:01.046 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2023-12-01T22:08:01.162 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2023-12-01T22:08:01.320 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2023-12-01T22:08:01.374 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2023-12-01T22:08:01.520 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2023-12-01T22:08:01.658 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 4/6 2023-12-01T22:08:01.845 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2023-12-01T22:08:02.609 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2023-12-01T22:08:02.609 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-01T22:08:02.609 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2023-12-01T22:08:02.609 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 2023-12-01T22:08:02.609 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-01T22:08:02.609 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-12-01T22:08:02.661 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 5/6 2023-12-01T22:08:02.853 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-mgr 2023-12-01T22:08:02.932 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2023-12-01T22:08:03.206 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2023-12-01T22:08:03.206 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 1/6 2023-12-01T22:08:03.206 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2/6 2023-12-01T22:08:03.206 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 3/6 2023-12-01T22:08:03.206 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2023-12-01T22:08:03.206 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2023-12-01T22:08:03.846 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:01:41 ago on Fri 01 Dec 2023 10:06:22 PM UTC. 2023-12-01T22:08:04.049 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2023-12-01T22:08:04.049 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:08:04.049 INFO:teuthology.orchestra.run.smithi073.stdout:Installed: 2023-12-01T22:08:04.049 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 2023-12-01T22:08:04.049 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2023-12-01T22:08:04.049 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 2023-12-01T22:08:04.049 INFO:teuthology.orchestra.run.smithi073.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-12-01T22:08:04.049 INFO:teuthology.orchestra.run.smithi073.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-12-01T22:08:04.049 INFO:teuthology.orchestra.run.smithi073.stdout: python3-routes-2.4.1-12.el8.noarch 2023-12-01T22:08:04.050 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:08:04.050 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2023-12-01T22:08:04.167 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install ceph-mgr-diskprediction-local 2023-12-01T22:08:05.016 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:01:38 ago on Fri 01 Dec 2023 10:06:27 PM UTC. 2023-12-01T22:08:05.459 INFO:teuthology.orchestra.run.smithi029.stdout:Package ceph-mgr-2:16.2.4-0.el8.x86_64 is already installed. 2023-12-01T22:08:05.715 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-12-01T22:08:05.717 INFO:teuthology.orchestra.run.smithi029.stdout:Nothing to do. 2023-12-01T22:08:05.717 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-12-01T22:08:05.866 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-mgr-dashboard 2023-12-01T22:08:06.703 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2023-12-01T22:08:06.704 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================================ 2023-12-01T22:08:06.704 INFO:teuthology.orchestra.run.smithi073.stdout: Package Arch Version Repository Size 2023-12-01T22:08:06.704 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================================ 2023-12-01T22:08:06.704 INFO:teuthology.orchestra.run.smithi073.stdout:Installing: 2023-12-01T22:08:06.704 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.4-0.el8 ceph-noarch 8.6 M 2023-12-01T22:08:06.704 INFO:teuthology.orchestra.run.smithi073.stdout:Installing dependencies: 2023-12-01T22:08:06.704 INFO:teuthology.orchestra.run.smithi073.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2023-12-01T22:08:06.704 INFO:teuthology.orchestra.run.smithi073.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2023-12-01T22:08:06.704 INFO:teuthology.orchestra.run.smithi073.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2023-12-01T22:08:06.704 INFO:teuthology.orchestra.run.smithi073.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2023-12-01T22:08:06.704 INFO:teuthology.orchestra.run.smithi073.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2023-12-01T22:08:06.705 INFO:teuthology.orchestra.run.smithi073.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2023-12-01T22:08:06.705 INFO:teuthology.orchestra.run.smithi073.stdout: python3-scipy x86_64 1.0.0-21.module_el8.5.0+771+e5d9a225 appstream 14 M 2023-12-01T22:08:06.705 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:08:06.705 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction Summary 2023-12-01T22:08:06.705 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================================ 2023-12-01T22:08:06.705 INFO:teuthology.orchestra.run.smithi073.stdout:Install 8 Packages 2023-12-01T22:08:06.705 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:08:06.705 INFO:teuthology.orchestra.run.smithi073.stdout:Total download size: 37 M 2023-12-01T22:08:06.706 INFO:teuthology.orchestra.run.smithi073.stdout:Installed size: 210 M 2023-12-01T22:08:06.706 INFO:teuthology.orchestra.run.smithi073.stdout:Downloading Packages: 2023-12-01T22:08:06.854 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:01:44 ago on Fri 01 Dec 2023 10:06:22 PM UTC. 2023-12-01T22:08:06.955 INFO:teuthology.orchestra.run.smithi073.stdout:(1/8): ceph-mgr-diskprediction-local-16.2.4-0.e 51 MB/s | 8.6 MB 00:00 2023-12-01T22:08:07.313 INFO:teuthology.orchestra.run.smithi073.stdout:(2/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 10 MB/s | 3.7 MB 00:00 2023-12-01T22:08:07.364 INFO:teuthology.orchestra.run.smithi073.stdout:(3/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 4.4 MB/s | 225 kB 00:00 2023-12-01T22:08:07.522 INFO:teuthology.orchestra.run.smithi073.stdout:(4/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 6.6 MB/s | 4.9 MB 00:00 2023-12-01T22:08:07.623 INFO:teuthology.orchestra.run.smithi073.stdout:(5/8): openblas-0.3.15-4.el8.x86_64.rpm 5.7 MB/s | 4.7 MB 00:00 2023-12-01T22:08:07.931 INFO:teuthology.orchestra.run.smithi073.stdout:(6/8): python3-scipy-1.0.0-21.module_el8.5.0+77 24 MB/s | 14 MB 00:00 2023-12-01T22:08:08.098 INFO:teuthology.orchestra.run.smithi073.stdout:(7/8): libquadmath-8.5.0-21.el8.x86_64.rpm 363 kB/s | 172 kB 00:00 2023-12-01T22:08:08.720 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-12-01T22:08:08.721 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-12-01T22:08:08.721 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2023-12-01T22:08:08.722 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-12-01T22:08:08.722 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2023-12-01T22:08:08.722 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-dashboard noarch 2:16.2.4-0.el8 ceph-noarch 2.7 M 2023-12-01T22:08:08.722 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2023-12-01T22:08:08.722 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-grafana-dashboards noarch 2:16.2.4-0.el8 ceph-noarch 24 k 2023-12-01T22:08:08.722 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-prometheus-alerts noarch 2:16.2.4-0.el8 ceph-noarch 8.9 k 2023-12-01T22:08:08.722 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2023-12-01T22:08:08.722 INFO:teuthology.orchestra.run.smithi029.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-12-01T22:08:08.722 INFO:teuthology.orchestra.run.smithi029.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-12-01T22:08:08.722 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-01T22:08:08.722 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2023-12-01T22:08:08.722 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-12-01T22:08:08.723 INFO:teuthology.orchestra.run.smithi029.stdout:Install 6 Packages 2023-12-01T22:08:08.723 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-01T22:08:08.723 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 3.0 M 2023-12-01T22:08:08.723 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 15 M 2023-12-01T22:08:08.723 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2023-12-01T22:08:08.915 INFO:teuthology.orchestra.run.smithi073.stdout:(8/8): libgfortran-8.5.0-21.el8.x86_64.rpm 463 kB/s | 645 kB 00:01 2023-12-01T22:08:08.915 INFO:teuthology.orchestra.run.smithi073.stdout:-------------------------------------------------------------------------------- 2023-12-01T22:08:08.916 INFO:teuthology.orchestra.run.smithi073.stdout:Total 17 MB/s | 37 MB 00:02 2023-12-01T22:08:09.152 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction check 2023-12-01T22:08:09.170 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction check succeeded. 2023-12-01T22:08:09.170 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction test 2023-12-01T22:08:09.369 INFO:teuthology.orchestra.run.smithi029.stdout:(1/6): ceph-grafana-dashboards-16.2.4-0.el8.noa 493 kB/s | 24 kB 00:00 2023-12-01T22:08:09.394 INFO:teuthology.orchestra.run.smithi029.stdout:(2/6): ceph-prometheus-alerts-16.2.4-0.el8.noar 120 kB/s | 8.9 kB 00:00 2023-12-01T22:08:09.478 INFO:teuthology.orchestra.run.smithi029.stdout:(3/6): ceph-mgr-dashboard-16.2.4-0.el8.noarch.r 17 MB/s | 2.7 MB 00:00 2023-12-01T22:08:09.510 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction test succeeded. 2023-12-01T22:08:09.513 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction 2023-12-01T22:08:09.612 INFO:teuthology.orchestra.run.smithi029.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 155 kB/s | 34 kB 00:00 2023-12-01T22:08:09.705 INFO:teuthology.orchestra.run.smithi029.stdout:(5/6): python3-routes-2.4.1-12.el8.noarch.rpm 872 kB/s | 196 kB 00:00 2023-12-01T22:08:09.788 INFO:teuthology.orchestra.run.smithi029.stdout:(6/6): python3-jwt-1.6.1-2.el8.noarch.rpm 102 kB/s | 43 kB 00:00 2023-12-01T22:08:09.791 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2023-12-01T22:08:09.791 INFO:teuthology.orchestra.run.smithi029.stdout:Total 2.8 MB/s | 3.0 MB 00:01 2023-12-01T22:08:09.804 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2023-12-01T22:08:09.816 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2023-12-01T22:08:09.816 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2023-12-01T22:08:10.014 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2023-12-01T22:08:10.017 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2023-12-01T22:08:10.302 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2023-12-01T22:08:10.577 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2023-12-01T22:08:10.609 INFO:teuthology.orchestra.run.smithi073.stdout: Preparing : 1/1 2023-12-01T22:08:10.750 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2023-12-01T22:08:10.854 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2023-12-01T22:08:10.862 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2023-12-01T22:08:11.008 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2023-12-01T22:08:11.101 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2023-12-01T22:08:11.231 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 4/6 2023-12-01T22:08:11.587 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2023-12-01T22:08:11.808 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2023-12-01T22:08:12.418 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2023-12-01T22:08:12.625 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-12-01T22:08:13.004 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 5/6 2023-12-01T22:08:13.278 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2023-12-01T22:08:13.307 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-12-01T22:08:13.616 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2023-12-01T22:08:13.616 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 1/6 2023-12-01T22:08:13.616 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2/6 2023-12-01T22:08:13.616 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 3/6 2023-12-01T22:08:13.616 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2023-12-01T22:08:13.616 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2023-12-01T22:08:13.627 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2023-12-01T22:08:13.883 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-12-01T22:08:14.375 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2023-12-01T22:08:14.375 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-01T22:08:14.375 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2023-12-01T22:08:14.375 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 2023-12-01T22:08:14.376 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2023-12-01T22:08:14.376 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 2023-12-01T22:08:14.376 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-12-01T22:08:14.376 INFO:teuthology.orchestra.run.smithi029.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-12-01T22:08:14.376 INFO:teuthology.orchestra.run.smithi029.stdout: python3-routes-2.4.1-12.el8.noarch 2023-12-01T22:08:14.376 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-01T22:08:14.376 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-12-01T22:08:14.534 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-mgr-diskprediction-local 2023-12-01T22:08:15.533 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:01:53 ago on Fri 01 Dec 2023 10:06:22 PM UTC. 2023-12-01T22:08:15.654 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-12-01T22:08:16.932 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 7/8 2023-12-01T22:08:17.145 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2023-12-01T22:08:17.401 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-12-01T22:08:17.403 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================================ 2023-12-01T22:08:17.403 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2023-12-01T22:08:17.403 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================================ 2023-12-01T22:08:17.403 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2023-12-01T22:08:17.403 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.4-0.el8 ceph-noarch 8.6 M 2023-12-01T22:08:17.403 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2023-12-01T22:08:17.403 INFO:teuthology.orchestra.run.smithi029.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2023-12-01T22:08:17.403 INFO:teuthology.orchestra.run.smithi029.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2023-12-01T22:08:17.403 INFO:teuthology.orchestra.run.smithi029.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2023-12-01T22:08:17.403 INFO:teuthology.orchestra.run.smithi029.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2023-12-01T22:08:17.403 INFO:teuthology.orchestra.run.smithi029.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2023-12-01T22:08:17.403 INFO:teuthology.orchestra.run.smithi029.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2023-12-01T22:08:17.403 INFO:teuthology.orchestra.run.smithi029.stdout: python3-scipy x86_64 1.0.0-21.module_el8.5.0+771+e5d9a225 appstream 14 M 2023-12-01T22:08:17.404 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-01T22:08:17.404 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2023-12-01T22:08:17.404 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================================ 2023-12-01T22:08:17.404 INFO:teuthology.orchestra.run.smithi029.stdout:Install 8 Packages 2023-12-01T22:08:17.404 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-01T22:08:17.404 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 37 M 2023-12-01T22:08:17.404 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 210 M 2023-12-01T22:08:17.405 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2023-12-01T22:08:17.457 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2023-12-01T22:08:17.457 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 1/8 2023-12-01T22:08:17.457 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2023-12-01T22:08:17.457 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2023-12-01T22:08:17.457 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2023-12-01T22:08:17.457 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2023-12-01T22:08:17.457 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 6/8 2023-12-01T22:08:17.458 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2023-12-01T22:08:18.162 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2023-12-01T22:08:18.162 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:08:18.162 INFO:teuthology.orchestra.run.smithi073.stdout:Installed: 2023-12-01T22:08:18.162 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 2023-12-01T22:08:18.162 INFO:teuthology.orchestra.run.smithi073.stdout: libgfortran-8.5.0-21.el8.x86_64 2023-12-01T22:08:18.163 INFO:teuthology.orchestra.run.smithi073.stdout: libquadmath-8.5.0-21.el8.x86_64 2023-12-01T22:08:18.163 INFO:teuthology.orchestra.run.smithi073.stdout: openblas-0.3.15-4.el8.x86_64 2023-12-01T22:08:18.163 INFO:teuthology.orchestra.run.smithi073.stdout: openblas-threads-0.3.15-4.el8.x86_64 2023-12-01T22:08:18.163 INFO:teuthology.orchestra.run.smithi073.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-12-01T22:08:18.163 INFO:teuthology.orchestra.run.smithi073.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-12-01T22:08:18.163 INFO:teuthology.orchestra.run.smithi073.stdout: python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64 2023-12-01T22:08:18.163 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:08:18.163 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2023-12-01T22:08:18.277 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install ceph-mgr-rook 2023-12-01T22:08:18.434 INFO:teuthology.orchestra.run.smithi029.stdout:(1/8): ceph-mgr-diskprediction-local-16.2.4-0.e 11 MB/s | 8.6 MB 00:00 2023-12-01T22:08:19.000 INFO:teuthology.orchestra.run.smithi029.stdout:(2/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 3.7 MB/s | 4.9 MB 00:01 2023-12-01T22:08:19.067 INFO:teuthology.orchestra.run.smithi029.stdout:(3/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 3.3 MB/s | 225 kB 00:00 2023-12-01T22:08:19.130 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:01:52 ago on Fri 01 Dec 2023 10:06:27 PM UTC. 2023-12-01T22:08:19.793 INFO:teuthology.orchestra.run.smithi029.stdout:(4/8): python3-scipy-1.0.0-21.module_el8.5.0+77 19 MB/s | 14 MB 00:00 2023-12-01T22:08:19.919 INFO:teuthology.orchestra.run.smithi029.stdout:(5/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 2.5 MB/s | 3.7 MB 00:01 2023-12-01T22:08:20.344 INFO:teuthology.orchestra.run.smithi029.stdout:(6/8): openblas-0.3.15-4.el8.x86_64.rpm 1.8 MB/s | 4.7 MB 00:02 2023-12-01T22:08:20.578 INFO:teuthology.orchestra.run.smithi029.stdout:(7/8): libquadmath-8.5.0-21.el8.x86_64.rpm 262 kB/s | 172 kB 00:00 2023-12-01T22:08:20.803 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2023-12-01T22:08:20.805 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2023-12-01T22:08:20.805 INFO:teuthology.orchestra.run.smithi073.stdout: Package Arch Version Repository Size 2023-12-01T22:08:20.805 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2023-12-01T22:08:20.805 INFO:teuthology.orchestra.run.smithi073.stdout:Installing: 2023-12-01T22:08:20.805 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-mgr-rook noarch 2:16.2.4-0.el8 ceph-noarch 27 k 2023-12-01T22:08:20.805 INFO:teuthology.orchestra.run.smithi073.stdout:Installing dependencies: 2023-12-01T22:08:20.805 INFO:teuthology.orchestra.run.smithi073.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-12-01T22:08:20.805 INFO:teuthology.orchestra.run.smithi073.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-12-01T22:08:20.806 INFO:teuthology.orchestra.run.smithi073.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-12-01T22:08:20.806 INFO:teuthology.orchestra.run.smithi073.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2023-12-01T22:08:20.806 INFO:teuthology.orchestra.run.smithi073.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2023-12-01T22:08:20.806 INFO:teuthology.orchestra.run.smithi073.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-12-01T22:08:20.806 INFO:teuthology.orchestra.run.smithi073.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2023-12-01T22:08:20.806 INFO:teuthology.orchestra.run.smithi073.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2023-12-01T22:08:20.806 INFO:teuthology.orchestra.run.smithi073.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2023-12-01T22:08:20.806 INFO:teuthology.orchestra.run.smithi073.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2023-12-01T22:08:20.806 INFO:teuthology.orchestra.run.smithi073.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-12-01T22:08:20.806 INFO:teuthology.orchestra.run.smithi073.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-12-01T22:08:20.806 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:08:20.806 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction Summary 2023-12-01T22:08:20.807 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2023-12-01T22:08:20.807 INFO:teuthology.orchestra.run.smithi073.stdout:Install 13 Packages 2023-12-01T22:08:20.807 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:08:20.807 INFO:teuthology.orchestra.run.smithi073.stdout:Total download size: 1.9 M 2023-12-01T22:08:20.807 INFO:teuthology.orchestra.run.smithi073.stdout:Installed size: 24 M 2023-12-01T22:08:20.808 INFO:teuthology.orchestra.run.smithi073.stdout:Downloading Packages: 2023-12-01T22:08:20.886 INFO:teuthology.orchestra.run.smithi029.stdout:(8/8): libgfortran-8.5.0-21.el8.x86_64.rpm 590 kB/s | 645 kB 00:01 2023-12-01T22:08:20.887 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2023-12-01T22:08:20.887 INFO:teuthology.orchestra.run.smithi029.stdout:Total 11 MB/s | 37 MB 00:03 2023-12-01T22:08:21.139 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2023-12-01T22:08:21.159 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2023-12-01T22:08:21.159 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2023-12-01T22:08:21.247 INFO:teuthology.orchestra.run.smithi073.stdout:(1/13): ceph-mgr-rook-16.2.4-0.el8.noarch.rpm 799 kB/s | 27 kB 00:00 2023-12-01T22:08:21.356 INFO:teuthology.orchestra.run.smithi073.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 140 kB/s | 20 kB 00:00 2023-12-01T22:08:21.381 INFO:teuthology.orchestra.run.smithi073.stdout:(3/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 158 kB/s | 27 kB 00:00 2023-12-01T22:08:21.465 INFO:teuthology.orchestra.run.smithi073.stdout:(4/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 581 kB/s | 126 kB 00:00 2023-12-01T22:08:21.515 INFO:teuthology.orchestra.run.smithi073.stdout:(5/13): python3-pyasn1-modules-0.3.7-6.el8.noar 692 kB/s | 110 kB 00:00 2023-12-01T22:08:21.520 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2023-12-01T22:08:21.523 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2023-12-01T22:08:21.674 INFO:teuthology.orchestra.run.smithi073.stdout:(6/13): python3-cachetools-3.1.1-4.el8.noarch.r 208 kB/s | 33 kB 00:00 2023-12-01T22:08:21.733 INFO:teuthology.orchestra.run.smithi073.stdout:(7/13): python3-certifi-2018.10.15-7.el8.noarch 279 kB/s | 16 kB 00:00 2023-12-01T22:08:21.841 INFO:teuthology.orchestra.run.smithi073.stdout:(8/13): python3-google-auth-1.1.1-10.el8.noarch 749 kB/s | 81 kB 00:00 2023-12-01T22:08:21.967 INFO:teuthology.orchestra.run.smithi073.stdout:(9/13): python3-requests-oauthlib-1.0.0-1.el8.n 85 kB/s | 43 kB 00:00 2023-12-01T22:08:22.067 INFO:teuthology.orchestra.run.smithi073.stdout:(10/13): python3-kubernetes-11.0.0-6.el8.noarch 5.2 MB/s | 1.2 MB 00:00 2023-12-01T22:08:22.125 INFO:teuthology.orchestra.run.smithi073.stdout:(11/13): python3-websocket-client-0.56.0-5.el8. 1.0 MB/s | 61 kB 00:00 2023-12-01T22:08:22.184 INFO:teuthology.orchestra.run.smithi073.stdout:(12/13): python3-rsa-4.9-2.el8.noarch.rpm 296 kB/s | 64 kB 00:00 2023-12-01T22:08:22.186 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2023-12-01T22:08:22.355 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2023-12-01T22:08:22.497 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2023-12-01T22:08:22.667 INFO:teuthology.orchestra.run.smithi073.stdout:(13/13): python3-oauthlib-2.1.0-1.el8.noarch.rp 120 kB/s | 155 kB 00:01 2023-12-01T22:08:22.668 INFO:teuthology.orchestra.run.smithi073.stdout:-------------------------------------------------------------------------------- 2023-12-01T22:08:22.668 INFO:teuthology.orchestra.run.smithi073.stdout:Total 1.0 MB/s | 1.9 MB 00:01 2023-12-01T22:08:22.680 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2023-12-01T22:08:22.730 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction check 2023-12-01T22:08:22.747 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction check succeeded. 2023-12-01T22:08:22.747 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction test 2023-12-01T22:08:22.842 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction test succeeded. 2023-12-01T22:08:22.845 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction 2023-12-01T22:08:23.343 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2023-12-01T22:08:23.349 INFO:teuthology.orchestra.run.smithi073.stdout: Preparing : 1/1 2023-12-01T22:08:23.563 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2023-12-01T22:08:23.583 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-12-01T22:08:23.814 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2023-12-01T22:08:24.066 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2023-12-01T22:08:24.257 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2023-12-01T22:08:24.305 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2023-12-01T22:08:24.447 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-12-01T22:08:24.553 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2023-12-01T22:08:24.795 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2023-12-01T22:08:25.079 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2023-12-01T22:08:25.245 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-12-01T22:08:25.320 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2023-12-01T22:08:25.612 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2023-12-01T22:08:25.880 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-12-01T22:08:26.341 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2023-12-01T22:08:26.621 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2023-12-01T22:08:26.879 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2023-12-01T22:08:27.071 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2023-12-01T22:08:27.239 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2023-12-01T22:08:27.537 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2023-12-01T22:08:27.538 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-mgr-rook-2:16.2.4-0.el8.noarch 1/13 2023-12-01T22:08:27.538 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2023-12-01T22:08:27.538 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2023-12-01T22:08:27.538 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2023-12-01T22:08:27.538 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2023-12-01T22:08:27.538 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2023-12-01T22:08:27.538 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2023-12-01T22:08:27.538 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2023-12-01T22:08:27.538 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-12-01T22:08:27.538 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2023-12-01T22:08:27.538 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2023-12-01T22:08:27.538 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2023-12-01T22:08:27.911 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-12-01T22:08:28.292 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2023-12-01T22:08:28.293 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:08:28.293 INFO:teuthology.orchestra.run.smithi073.stdout:Installed: 2023-12-01T22:08:28.293 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-mgr-rook-2:16.2.4-0.el8.noarch 2023-12-01T22:08:28.293 INFO:teuthology.orchestra.run.smithi073.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-12-01T22:08:28.293 INFO:teuthology.orchestra.run.smithi073.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-12-01T22:08:28.293 INFO:teuthology.orchestra.run.smithi073.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-12-01T22:08:28.293 INFO:teuthology.orchestra.run.smithi073.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-12-01T22:08:28.293 INFO:teuthology.orchestra.run.smithi073.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-12-01T22:08:28.293 INFO:teuthology.orchestra.run.smithi073.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-12-01T22:08:28.293 INFO:teuthology.orchestra.run.smithi073.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-12-01T22:08:28.293 INFO:teuthology.orchestra.run.smithi073.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-12-01T22:08:28.293 INFO:teuthology.orchestra.run.smithi073.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-12-01T22:08:28.293 INFO:teuthology.orchestra.run.smithi073.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-12-01T22:08:28.294 INFO:teuthology.orchestra.run.smithi073.stdout: python3-rsa-4.9-2.el8.noarch 2023-12-01T22:08:28.294 INFO:teuthology.orchestra.run.smithi073.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-12-01T22:08:28.294 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:08:28.294 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2023-12-01T22:08:28.411 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install ceph-mgr-cephadm 2023-12-01T22:08:29.278 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:02:02 ago on Fri 01 Dec 2023 10:06:27 PM UTC. 2023-12-01T22:08:29.358 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 7/8 2023-12-01T22:08:29.584 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2023-12-01T22:08:29.960 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2023-12-01T22:08:29.960 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 1/8 2023-12-01T22:08:29.961 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2023-12-01T22:08:29.961 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2023-12-01T22:08:29.961 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2023-12-01T22:08:29.961 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2023-12-01T22:08:29.961 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 6/8 2023-12-01T22:08:29.961 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2023-12-01T22:08:30.745 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2023-12-01T22:08:30.745 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-01T22:08:30.745 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2023-12-01T22:08:30.745 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 2023-12-01T22:08:30.746 INFO:teuthology.orchestra.run.smithi029.stdout: libgfortran-8.5.0-21.el8.x86_64 2023-12-01T22:08:30.746 INFO:teuthology.orchestra.run.smithi029.stdout: libquadmath-8.5.0-21.el8.x86_64 2023-12-01T22:08:30.746 INFO:teuthology.orchestra.run.smithi029.stdout: openblas-0.3.15-4.el8.x86_64 2023-12-01T22:08:30.746 INFO:teuthology.orchestra.run.smithi029.stdout: openblas-threads-0.3.15-4.el8.x86_64 2023-12-01T22:08:30.746 INFO:teuthology.orchestra.run.smithi029.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-12-01T22:08:30.746 INFO:teuthology.orchestra.run.smithi029.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-12-01T22:08:30.746 INFO:teuthology.orchestra.run.smithi029.stdout: python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64 2023-12-01T22:08:30.746 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-01T22:08:30.746 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-12-01T22:08:30.863 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-mgr-rook 2023-12-01T22:08:30.961 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2023-12-01T22:08:30.962 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2023-12-01T22:08:30.962 INFO:teuthology.orchestra.run.smithi073.stdout: Package Arch Version Repository Size 2023-12-01T22:08:30.962 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2023-12-01T22:08:30.963 INFO:teuthology.orchestra.run.smithi073.stdout:Installing: 2023-12-01T22:08:30.963 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-mgr-cephadm noarch 2:16.2.4-0.el8 ceph-noarch 89 k 2023-12-01T22:08:30.963 INFO:teuthology.orchestra.run.smithi073.stdout:Installing dependencies: 2023-12-01T22:08:30.963 INFO:teuthology.orchestra.run.smithi073.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2023-12-01T22:08:30.963 INFO:teuthology.orchestra.run.smithi073.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2023-12-01T22:08:30.963 INFO:teuthology.orchestra.run.smithi073.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2023-12-01T22:08:30.963 INFO:teuthology.orchestra.run.smithi073.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2023-12-01T22:08:30.963 INFO:teuthology.orchestra.run.smithi073.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2023-12-01T22:08:30.963 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:08:30.963 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction Summary 2023-12-01T22:08:30.963 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2023-12-01T22:08:30.963 INFO:teuthology.orchestra.run.smithi073.stdout:Install 6 Packages 2023-12-01T22:08:30.963 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:08:30.964 INFO:teuthology.orchestra.run.smithi073.stdout:Total download size: 5.8 M 2023-12-01T22:08:30.964 INFO:teuthology.orchestra.run.smithi073.stdout:Installed size: 24 M 2023-12-01T22:08:30.964 INFO:teuthology.orchestra.run.smithi073.stdout:Downloading Packages: 2023-12-01T22:08:31.261 INFO:teuthology.orchestra.run.smithi073.stdout:(1/6): ceph-mgr-cephadm-16.2.4-0.el8.noarch.rpm 2.2 MB/s | 89 kB 00:00 2023-12-01T22:08:31.486 INFO:teuthology.orchestra.run.smithi073.stdout:(2/6): python3-apipkg-1.5-6.el8.noarch.rpm 94 kB/s | 21 kB 00:00 2023-12-01T22:08:31.520 INFO:teuthology.orchestra.run.smithi073.stdout:(3/6): python3-jinja2-2.10.1-3.el8.noarch.rpm 1.8 MB/s | 538 kB 00:00 2023-12-01T22:08:31.687 INFO:teuthology.orchestra.run.smithi073.stdout:(4/6): python3-execnet-1.7.1-1.el8.noarch.rpm 1.7 MB/s | 355 kB 00:00 2023-12-01T22:08:31.712 INFO:teuthology.orchestra.run.smithi073.stdout:(5/6): python3-remoto-1.2.1-1.el8.noarch.rpm 254 kB/s | 49 kB 00:00 2023-12-01T22:08:31.855 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:02:09 ago on Fri 01 Dec 2023 10:06:22 PM UTC. 2023-12-01T22:08:31.921 INFO:teuthology.orchestra.run.smithi073.stdout:(6/6): python3-babel-2.5.1-7.el8.noarch.rpm 6.9 MB/s | 4.8 MB 00:00 2023-12-01T22:08:31.921 INFO:teuthology.orchestra.run.smithi073.stdout:-------------------------------------------------------------------------------- 2023-12-01T22:08:31.921 INFO:teuthology.orchestra.run.smithi073.stdout:Total 6.1 MB/s | 5.8 MB 00:00 2023-12-01T22:08:31.972 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction check 2023-12-01T22:08:31.981 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction check succeeded. 2023-12-01T22:08:31.981 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction test 2023-12-01T22:08:32.084 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction test succeeded. 2023-12-01T22:08:32.088 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction 2023-12-01T22:08:32.358 INFO:teuthology.orchestra.run.smithi073.stdout: Preparing : 1/1 2023-12-01T22:08:32.621 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2023-12-01T22:08:32.926 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2023-12-01T22:08:33.729 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-12-01T22:08:33.731 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-12-01T22:08:33.731 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2023-12-01T22:08:33.731 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-12-01T22:08:33.731 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2023-12-01T22:08:33.732 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-rook noarch 2:16.2.4-0.el8 ceph-noarch 27 k 2023-12-01T22:08:33.732 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2023-12-01T22:08:33.732 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-12-01T22:08:33.732 INFO:teuthology.orchestra.run.smithi029.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-12-01T22:08:33.732 INFO:teuthology.orchestra.run.smithi029.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-12-01T22:08:33.732 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2023-12-01T22:08:33.732 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2023-12-01T22:08:33.732 INFO:teuthology.orchestra.run.smithi029.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-12-01T22:08:33.732 INFO:teuthology.orchestra.run.smithi029.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2023-12-01T22:08:33.732 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2023-12-01T22:08:33.732 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2023-12-01T22:08:33.732 INFO:teuthology.orchestra.run.smithi029.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2023-12-01T22:08:33.732 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-12-01T22:08:33.732 INFO:teuthology.orchestra.run.smithi029.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-12-01T22:08:33.733 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-01T22:08:33.733 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2023-12-01T22:08:33.733 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-12-01T22:08:33.733 INFO:teuthology.orchestra.run.smithi029.stdout:Install 13 Packages 2023-12-01T22:08:33.733 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-01T22:08:33.733 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 1.9 M 2023-12-01T22:08:33.733 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 24 M 2023-12-01T22:08:33.733 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2023-12-01T22:08:33.880 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2023-12-01T22:08:34.048 INFO:teuthology.orchestra.run.smithi029.stdout:(1/13): ceph-mgr-rook-16.2.4-0.el8.noarch.rpm 588 kB/s | 27 kB 00:00 2023-12-01T22:08:34.140 INFO:teuthology.orchestra.run.smithi029.stdout:(2/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 193 kB/s | 27 kB 00:00 2023-12-01T22:08:34.174 INFO:teuthology.orchestra.run.smithi029.stdout:(3/13): python3-jsonpointer-1.10-11.el8.noarch. 116 kB/s | 20 kB 00:00 2023-12-01T22:08:34.208 INFO:teuthology.orchestra.run.smithi029.stdout:(4/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 791 kB/s | 126 kB 00:00 2023-12-01T22:08:34.234 INFO:teuthology.orchestra.run.smithi029.stdout:(5/13): python3-pyasn1-modules-0.3.7-6.el8.noar 1.2 MB/s | 110 kB 00:00 2023-12-01T22:08:34.241 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2023-12-01T22:08:34.410 INFO:teuthology.orchestra.run.smithi029.stdout:(6/13): python3-cachetools-3.1.1-4.el8.noarch.r 188 kB/s | 33 kB 00:00 2023-12-01T22:08:34.460 INFO:teuthology.orchestra.run.smithi029.stdout:(7/13): python3-certifi-2018.10.15-7.el8.noarch 324 kB/s | 16 kB 00:00 2023-12-01T22:08:34.477 INFO:teuthology.orchestra.run.smithi029.stdout:(8/13): python3-requests-oauthlib-1.0.0-1.el8.n 159 kB/s | 43 kB 00:00 2023-12-01T22:08:34.493 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 5/6 2023-12-01T22:08:34.561 INFO:teuthology.orchestra.run.smithi029.stdout:(9/13): python3-google-auth-1.1.1-10.el8.noarch 806 kB/s | 81 kB 00:00 2023-12-01T22:08:34.628 INFO:teuthology.orchestra.run.smithi029.stdout:(10/13): python3-oauthlib-2.1.0-1.el8.noarch.rp 341 kB/s | 155 kB 00:00 2023-12-01T22:08:34.662 INFO:teuthology.orchestra.run.smithi029.stdout:(11/13): python3-rsa-4.9-2.el8.noarch.rpm 642 kB/s | 64 kB 00:00 2023-12-01T22:08:34.721 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2023-12-01T22:08:34.846 INFO:teuthology.orchestra.run.smithi029.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 282 kB/s | 61 kB 00:00 2023-12-01T22:08:34.992 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2023-12-01T22:08:34.992 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 1/6 2023-12-01T22:08:34.992 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2023-12-01T22:08:34.992 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/6 2023-12-01T22:08:34.993 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2023-12-01T22:08:34.993 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2023-12-01T22:08:35.246 INFO:teuthology.orchestra.run.smithi029.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 1.5 MB/s | 1.2 MB 00:00 2023-12-01T22:08:35.249 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2023-12-01T22:08:35.249 INFO:teuthology.orchestra.run.smithi029.stdout:Total 1.3 MB/s | 1.9 MB 00:01 2023-12-01T22:08:35.310 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2023-12-01T22:08:35.327 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2023-12-01T22:08:35.328 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2023-12-01T22:08:35.426 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2023-12-01T22:08:35.430 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2023-12-01T22:08:35.682 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2023-12-01T22:08:35.682 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:08:35.682 INFO:teuthology.orchestra.run.smithi073.stdout:Installed: 2023-12-01T22:08:35.683 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch python3-apipkg-1.5-6.el8.noarch 2023-12-01T22:08:35.683 INFO:teuthology.orchestra.run.smithi073.stdout: python3-babel-2.5.1-7.el8.noarch python3-execnet-1.7.1-1.el8.noarch 2023-12-01T22:08:35.683 INFO:teuthology.orchestra.run.smithi073.stdout: python3-jinja2-2.10.1-3.el8.noarch python3-remoto-1.2.1-1.el8.noarch 2023-12-01T22:08:35.683 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:08:35.683 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2023-12-01T22:08:35.831 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install ceph-fuse 2023-12-01T22:08:35.945 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2023-12-01T22:08:36.197 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-12-01T22:08:36.421 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2023-12-01T22:08:36.677 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2023-12-01T22:08:36.687 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:02:09 ago on Fri 01 Dec 2023 10:06:27 PM UTC. 2023-12-01T22:08:36.937 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2023-12-01T22:08:37.207 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2023-12-01T22:08:37.465 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2023-12-01T22:08:37.794 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2023-12-01T22:08:38.054 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2023-12-01T22:08:38.362 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2023-12-01T22:08:38.362 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2023-12-01T22:08:38.363 INFO:teuthology.orchestra.run.smithi073.stdout: Package Architecture Version Repository Size 2023-12-01T22:08:38.363 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2023-12-01T22:08:38.363 INFO:teuthology.orchestra.run.smithi073.stdout:Installing: 2023-12-01T22:08:38.363 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-fuse x86_64 2:16.2.4-0.el8 ceph 885 k 2023-12-01T22:08:38.363 INFO:teuthology.orchestra.run.smithi073.stdout:Installing dependencies: 2023-12-01T22:08:38.363 INFO:teuthology.orchestra.run.smithi073.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2023-12-01T22:08:38.363 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:08:38.363 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction Summary 2023-12-01T22:08:38.363 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2023-12-01T22:08:38.363 INFO:teuthology.orchestra.run.smithi073.stdout:Install 2 Packages 2023-12-01T22:08:38.363 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:08:38.364 INFO:teuthology.orchestra.run.smithi073.stdout:Total download size: 968 k 2023-12-01T22:08:38.364 INFO:teuthology.orchestra.run.smithi073.stdout:Installed size: 3.2 M 2023-12-01T22:08:38.364 INFO:teuthology.orchestra.run.smithi073.stdout:Downloading Packages: 2023-12-01T22:08:38.465 INFO:teuthology.orchestra.run.smithi073.stdout:(1/2): ceph-fuse-16.2.4-0.el8.x86_64.rpm 15 MB/s | 885 kB 00:00 2023-12-01T22:08:38.691 INFO:teuthology.orchestra.run.smithi073.stdout:(2/2): fuse-2.9.7-17.el8.x86_64.rpm 293 kB/s | 83 kB 00:00 2023-12-01T22:08:38.691 INFO:teuthology.orchestra.run.smithi073.stdout:-------------------------------------------------------------------------------- 2023-12-01T22:08:38.691 INFO:teuthology.orchestra.run.smithi073.stdout:Total 2.9 MB/s | 968 kB 00:00 2023-12-01T22:08:38.701 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction check 2023-12-01T22:08:38.721 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction check succeeded. 2023-12-01T22:08:38.721 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction test 2023-12-01T22:08:38.809 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction test succeeded. 2023-12-01T22:08:38.812 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction 2023-12-01T22:08:38.980 INFO:teuthology.orchestra.run.smithi073.stdout: Preparing : 1/1 2023-12-01T22:08:39.218 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2023-12-01T22:08:39.263 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2023-12-01T22:08:39.496 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2023-12-01T22:08:39.520 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2023-12-01T22:08:39.771 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2023-12-01T22:08:39.887 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2023-12-01T22:08:39.887 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-fuse-2:16.2.4-0.el8.x86_64 1/2 2023-12-01T22:08:39.995 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2023-12-01T22:08:40.161 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2023-12-01T22:08:40.494 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2023-12-01T22:08:40.494 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-rook-2:16.2.4-0.el8.noarch 1/13 2023-12-01T22:08:40.494 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2023-12-01T22:08:40.494 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2023-12-01T22:08:40.495 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2023-12-01T22:08:40.495 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2023-12-01T22:08:40.495 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2023-12-01T22:08:40.495 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2023-12-01T22:08:40.495 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2023-12-01T22:08:40.495 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-12-01T22:08:40.495 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2023-12-01T22:08:40.495 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2023-12-01T22:08:40.495 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2023-12-01T22:08:40.598 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2023-12-01T22:08:40.598 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:08:40.598 INFO:teuthology.orchestra.run.smithi073.stdout:Installed: 2023-12-01T22:08:40.598 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-fuse-2:16.2.4-0.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2023-12-01T22:08:40.598 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:08:40.598 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2023-12-01T22:08:40.830 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install librados-devel 2023-12-01T22:08:41.323 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2023-12-01T22:08:41.323 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-01T22:08:41.323 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2023-12-01T22:08:41.323 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-rook-2:16.2.4-0.el8.noarch 2023-12-01T22:08:41.323 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-12-01T22:08:41.323 INFO:teuthology.orchestra.run.smithi029.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-12-01T22:08:41.323 INFO:teuthology.orchestra.run.smithi029.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-12-01T22:08:41.323 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-12-01T22:08:41.323 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-12-01T22:08:41.323 INFO:teuthology.orchestra.run.smithi029.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-12-01T22:08:41.323 INFO:teuthology.orchestra.run.smithi029.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-12-01T22:08:41.324 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-12-01T22:08:41.324 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-12-01T22:08:41.324 INFO:teuthology.orchestra.run.smithi029.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-12-01T22:08:41.324 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rsa-4.9-2.el8.noarch 2023-12-01T22:08:41.324 INFO:teuthology.orchestra.run.smithi029.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-12-01T22:08:41.324 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-01T22:08:41.324 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-12-01T22:08:41.467 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-mgr-cephadm 2023-12-01T22:08:41.694 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:02:14 ago on Fri 01 Dec 2023 10:06:27 PM UTC. 2023-12-01T22:08:42.461 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:02:20 ago on Fri 01 Dec 2023 10:06:22 PM UTC. 2023-12-01T22:08:43.362 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2023-12-01T22:08:43.363 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2023-12-01T22:08:43.363 INFO:teuthology.orchestra.run.smithi073.stdout: Package Architecture Version Repository Size 2023-12-01T22:08:43.363 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2023-12-01T22:08:43.363 INFO:teuthology.orchestra.run.smithi073.stdout:Installing: 2023-12-01T22:08:43.363 INFO:teuthology.orchestra.run.smithi073.stdout: librados-devel x86_64 2:16.2.4-0.el8 ceph 131 k 2023-12-01T22:08:43.363 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:08:43.363 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction Summary 2023-12-01T22:08:43.364 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2023-12-01T22:08:43.364 INFO:teuthology.orchestra.run.smithi073.stdout:Install 1 Package 2023-12-01T22:08:43.364 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:08:43.364 INFO:teuthology.orchestra.run.smithi073.stdout:Total download size: 131 k 2023-12-01T22:08:43.364 INFO:teuthology.orchestra.run.smithi073.stdout:Installed size: 439 k 2023-12-01T22:08:43.364 INFO:teuthology.orchestra.run.smithi073.stdout:Downloading Packages: 2023-12-01T22:08:43.428 INFO:teuthology.orchestra.run.smithi073.stdout:librados-devel-16.2.4-0.el8.x86_64.rpm 2.0 MB/s | 131 kB 00:00 2023-12-01T22:08:43.429 INFO:teuthology.orchestra.run.smithi073.stdout:-------------------------------------------------------------------------------- 2023-12-01T22:08:43.429 INFO:teuthology.orchestra.run.smithi073.stdout:Total 2.0 MB/s | 131 kB 00:00 2023-12-01T22:08:43.429 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction check 2023-12-01T22:08:43.434 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction check succeeded. 2023-12-01T22:08:43.434 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction test 2023-12-01T22:08:43.483 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction test succeeded. 2023-12-01T22:08:43.486 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction 2023-12-01T22:08:43.622 INFO:teuthology.orchestra.run.smithi073.stdout: Preparing : 1/1 2023-12-01T22:08:43.836 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2023-12-01T22:08:44.122 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: librados-devel-2:16.2.4-0.el8.x86_64 1/1 2023-12-01T22:08:44.356 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-12-01T22:08:44.357 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-12-01T22:08:44.357 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2023-12-01T22:08:44.357 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-12-01T22:08:44.357 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2023-12-01T22:08:44.357 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-cephadm noarch 2:16.2.4-0.el8 ceph-noarch 89 k 2023-12-01T22:08:44.357 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2023-12-01T22:08:44.357 INFO:teuthology.orchestra.run.smithi029.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2023-12-01T22:08:44.357 INFO:teuthology.orchestra.run.smithi029.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2023-12-01T22:08:44.357 INFO:teuthology.orchestra.run.smithi029.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2023-12-01T22:08:44.358 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2023-12-01T22:08:44.358 INFO:teuthology.orchestra.run.smithi029.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2023-12-01T22:08:44.358 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-01T22:08:44.358 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2023-12-01T22:08:44.358 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-12-01T22:08:44.358 INFO:teuthology.orchestra.run.smithi029.stdout:Install 6 Packages 2023-12-01T22:08:44.358 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-01T22:08:44.359 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 5.8 M 2023-12-01T22:08:44.359 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 24 M 2023-12-01T22:08:44.359 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2023-12-01T22:08:44.786 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2023-12-01T22:08:44.787 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:08:44.787 INFO:teuthology.orchestra.run.smithi073.stdout:Installed: 2023-12-01T22:08:44.787 INFO:teuthology.orchestra.run.smithi073.stdout: librados-devel-2:16.2.4-0.el8.x86_64 2023-12-01T22:08:44.787 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:08:44.787 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2023-12-01T22:08:44.812 INFO:teuthology.orchestra.run.smithi029.stdout:(1/6): ceph-mgr-cephadm-16.2.4-0.el8.noarch.rpm 484 kB/s | 89 kB 00:00 2023-12-01T22:08:44.976 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install libcephfs2 2023-12-01T22:08:45.049 INFO:teuthology.orchestra.run.smithi029.stdout:(2/6): python3-apipkg-1.5-6.el8.noarch.rpm 89 kB/s | 21 kB 00:00 2023-12-01T22:08:45.367 INFO:teuthology.orchestra.run.smithi029.stdout:(3/6): python3-execnet-1.7.1-1.el8.noarch.rpm 1.1 MB/s | 355 kB 00:00 2023-12-01T22:08:45.408 INFO:teuthology.orchestra.run.smithi029.stdout:(4/6): python3-remoto-1.2.1-1.el8.noarch.rpm 1.2 MB/s | 49 kB 00:00 2023-12-01T22:08:45.483 INFO:teuthology.orchestra.run.smithi029.stdout:(5/6): python3-jinja2-2.10.1-3.el8.noarch.rpm 628 kB/s | 538 kB 00:00 2023-12-01T22:08:45.725 INFO:teuthology.orchestra.run.smithi029.stdout:(6/6): python3-babel-2.5.1-7.el8.noarch.rpm 4.4 MB/s | 4.8 MB 00:01 2023-12-01T22:08:45.726 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2023-12-01T22:08:45.727 INFO:teuthology.orchestra.run.smithi029.stdout:Total 4.2 MB/s | 5.8 MB 00:01 2023-12-01T22:08:45.784 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2023-12-01T22:08:45.795 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2023-12-01T22:08:45.795 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2023-12-01T22:08:45.844 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:02:18 ago on Fri 01 Dec 2023 10:06:27 PM UTC. 2023-12-01T22:08:45.908 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2023-12-01T22:08:45.912 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2023-12-01T22:08:46.200 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2023-12-01T22:08:46.507 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2023-12-01T22:08:46.840 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2023-12-01T22:08:47.317 INFO:teuthology.orchestra.run.smithi073.stdout:Package libcephfs2-2:16.2.4-0.el8.x86_64 is already installed. 2023-12-01T22:08:47.542 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2023-12-01T22:08:47.543 INFO:teuthology.orchestra.run.smithi073.stdout:Nothing to do. 2023-12-01T22:08:47.543 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2023-12-01T22:08:47.886 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2023-12-01T22:08:48.243 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install libcephfs-devel 2023-12-01T22:08:48.265 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2023-12-01T22:08:48.519 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 5/6 2023-12-01T22:08:48.735 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2023-12-01T22:08:49.077 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2023-12-01T22:08:49.077 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 1/6 2023-12-01T22:08:49.077 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2023-12-01T22:08:49.077 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/6 2023-12-01T22:08:49.077 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2023-12-01T22:08:49.077 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2023-12-01T22:08:49.100 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:02:22 ago on Fri 01 Dec 2023 10:06:27 PM UTC. 2023-12-01T22:08:49.844 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2023-12-01T22:08:49.844 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-01T22:08:49.844 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2023-12-01T22:08:49.844 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch python3-apipkg-1.5-6.el8.noarch 2023-12-01T22:08:49.844 INFO:teuthology.orchestra.run.smithi029.stdout: python3-babel-2.5.1-7.el8.noarch python3-execnet-1.7.1-1.el8.noarch 2023-12-01T22:08:49.844 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jinja2-2.10.1-3.el8.noarch python3-remoto-1.2.1-1.el8.noarch 2023-12-01T22:08:49.844 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-01T22:08:49.844 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-12-01T22:08:49.993 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-fuse 2023-12-01T22:08:50.774 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2023-12-01T22:08:50.774 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2023-12-01T22:08:50.774 INFO:teuthology.orchestra.run.smithi073.stdout: Package Architecture Version Repository Size 2023-12-01T22:08:50.774 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2023-12-01T22:08:50.774 INFO:teuthology.orchestra.run.smithi073.stdout:Installing: 2023-12-01T22:08:50.774 INFO:teuthology.orchestra.run.smithi073.stdout: libcephfs-devel x86_64 2:16.2.4-0.el8 ceph 24 k 2023-12-01T22:08:50.774 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:08:50.774 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction Summary 2023-12-01T22:08:50.774 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2023-12-01T22:08:50.774 INFO:teuthology.orchestra.run.smithi073.stdout:Install 1 Package 2023-12-01T22:08:50.775 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:08:50.775 INFO:teuthology.orchestra.run.smithi073.stdout:Total download size: 24 k 2023-12-01T22:08:50.775 INFO:teuthology.orchestra.run.smithi073.stdout:Installed size: 94 k 2023-12-01T22:08:50.775 INFO:teuthology.orchestra.run.smithi073.stdout:Downloading Packages: 2023-12-01T22:08:50.845 INFO:teuthology.orchestra.run.smithi073.stdout:libcephfs-devel-16.2.4-0.el8.x86_64.rpm 342 kB/s | 24 kB 00:00 2023-12-01T22:08:50.846 INFO:teuthology.orchestra.run.smithi073.stdout:-------------------------------------------------------------------------------- 2023-12-01T22:08:50.846 INFO:teuthology.orchestra.run.smithi073.stdout:Total 336 kB/s | 24 kB 00:00 2023-12-01T22:08:50.846 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction check 2023-12-01T22:08:50.849 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction check succeeded. 2023-12-01T22:08:50.850 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction test 2023-12-01T22:08:50.852 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction test succeeded. 2023-12-01T22:08:50.855 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction 2023-12-01T22:08:50.906 INFO:teuthology.orchestra.run.smithi073.stdout: Preparing : 1/1 2023-12-01T22:08:50.990 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:02:28 ago on Fri 01 Dec 2023 10:06:22 PM UTC. 2023-12-01T22:08:51.118 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2023-12-01T22:08:51.379 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2023-12-01T22:08:52.034 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2023-12-01T22:08:52.034 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:08:52.034 INFO:teuthology.orchestra.run.smithi073.stdout:Installed: 2023-12-01T22:08:52.035 INFO:teuthology.orchestra.run.smithi073.stdout: libcephfs-devel-2:16.2.4-0.el8.x86_64 2023-12-01T22:08:52.035 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:08:52.035 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2023-12-01T22:08:52.147 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install librados2 2023-12-01T22:08:52.869 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-12-01T22:08:52.870 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-12-01T22:08:52.870 INFO:teuthology.orchestra.run.smithi029.stdout: Package Architecture Version Repository Size 2023-12-01T22:08:52.870 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-12-01T22:08:52.870 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2023-12-01T22:08:52.870 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-fuse x86_64 2:16.2.4-0.el8 ceph 885 k 2023-12-01T22:08:52.871 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2023-12-01T22:08:52.871 INFO:teuthology.orchestra.run.smithi029.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2023-12-01T22:08:52.871 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-01T22:08:52.871 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2023-12-01T22:08:52.871 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-12-01T22:08:52.871 INFO:teuthology.orchestra.run.smithi029.stdout:Install 2 Packages 2023-12-01T22:08:52.871 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-01T22:08:52.871 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 968 k 2023-12-01T22:08:52.872 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 3.2 M 2023-12-01T22:08:52.872 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2023-12-01T22:08:53.002 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:02:26 ago on Fri 01 Dec 2023 10:06:27 PM UTC. 2023-12-01T22:08:53.129 INFO:teuthology.orchestra.run.smithi029.stdout:(1/2): ceph-fuse-16.2.4-0.el8.x86_64.rpm 13 MB/s | 885 kB 00:00 2023-12-01T22:08:53.514 INFO:teuthology.orchestra.run.smithi029.stdout:(2/2): fuse-2.9.7-17.el8.x86_64.rpm 185 kB/s | 83 kB 00:00 2023-12-01T22:08:53.514 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2023-12-01T22:08:53.515 INFO:teuthology.orchestra.run.smithi029.stdout:Total 1.5 MB/s | 968 kB 00:00 2023-12-01T22:08:53.527 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2023-12-01T22:08:53.552 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2023-12-01T22:08:53.552 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2023-12-01T22:08:53.657 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2023-12-01T22:08:53.660 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2023-12-01T22:08:53.843 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2023-12-01T22:08:54.161 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2023-12-01T22:08:54.378 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2023-12-01T22:08:54.455 INFO:teuthology.orchestra.run.smithi073.stdout:Package librados2-2:16.2.4-0.el8.x86_64 is already installed. 2023-12-01T22:08:54.678 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2023-12-01T22:08:54.679 INFO:teuthology.orchestra.run.smithi073.stdout:Nothing to do. 2023-12-01T22:08:54.679 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2023-12-01T22:08:54.772 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install librbd1 2023-12-01T22:08:54.838 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2023-12-01T22:08:54.838 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-fuse-2:16.2.4-0.el8.x86_64 1/2 2023-12-01T22:08:55.623 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:02:28 ago on Fri 01 Dec 2023 10:06:27 PM UTC. 2023-12-01T22:08:55.670 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2023-12-01T22:08:55.670 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-01T22:08:55.670 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2023-12-01T22:08:55.671 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-fuse-2:16.2.4-0.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2023-12-01T22:08:55.671 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-01T22:08:55.671 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-12-01T22:08:57.084 INFO:teuthology.orchestra.run.smithi073.stdout:Package librbd1-2:16.2.4-0.el8.x86_64 is already installed. 2023-12-01T22:08:57.306 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2023-12-01T22:08:57.307 INFO:teuthology.orchestra.run.smithi073.stdout:Nothing to do. 2023-12-01T22:08:57.307 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2023-12-01T22:08:57.414 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install python3-rados 2023-12-01T22:08:58.037 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install librados-devel 2023-12-01T22:08:58.274 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:02:31 ago on Fri 01 Dec 2023 10:06:27 PM UTC. 2023-12-01T22:08:59.040 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:02:37 ago on Fri 01 Dec 2023 10:06:22 PM UTC. 2023-12-01T22:08:59.726 INFO:teuthology.orchestra.run.smithi073.stdout:Package python3-rados-2:16.2.4-0.el8.x86_64 is already installed. 2023-12-01T22:08:59.950 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2023-12-01T22:08:59.951 INFO:teuthology.orchestra.run.smithi073.stdout:Nothing to do. 2023-12-01T22:08:59.951 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2023-12-01T22:09:00.051 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install python3-rgw 2023-12-01T22:09:00.907 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:02:33 ago on Fri 01 Dec 2023 10:06:27 PM UTC. 2023-12-01T22:09:00.920 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-12-01T22:09:00.921 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-12-01T22:09:00.921 INFO:teuthology.orchestra.run.smithi029.stdout: Package Architecture Version Repository Size 2023-12-01T22:09:00.921 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-12-01T22:09:00.921 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2023-12-01T22:09:00.921 INFO:teuthology.orchestra.run.smithi029.stdout: librados-devel x86_64 2:16.2.4-0.el8 ceph 131 k 2023-12-01T22:09:00.921 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-01T22:09:00.921 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2023-12-01T22:09:00.921 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-12-01T22:09:00.921 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2023-12-01T22:09:00.921 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-01T22:09:00.922 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 131 k 2023-12-01T22:09:00.922 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 439 k 2023-12-01T22:09:00.922 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2023-12-01T22:09:00.967 INFO:teuthology.orchestra.run.smithi029.stdout:librados-devel-16.2.4-0.el8.x86_64.rpm 2.9 MB/s | 131 kB 00:00 2023-12-01T22:09:00.967 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2023-12-01T22:09:00.968 INFO:teuthology.orchestra.run.smithi029.stdout:Total 2.8 MB/s | 131 kB 00:00 2023-12-01T22:09:00.968 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2023-12-01T22:09:00.974 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2023-12-01T22:09:00.974 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2023-12-01T22:09:01.032 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2023-12-01T22:09:01.035 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2023-12-01T22:09:01.153 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2023-12-01T22:09:01.401 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2023-12-01T22:09:01.728 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librados-devel-2:16.2.4-0.el8.x86_64 1/1 2023-12-01T22:09:02.365 INFO:teuthology.orchestra.run.smithi073.stdout:Package python3-rgw-2:16.2.4-0.el8.x86_64 is already installed. 2023-12-01T22:09:02.508 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2023-12-01T22:09:02.509 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-01T22:09:02.509 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2023-12-01T22:09:02.509 INFO:teuthology.orchestra.run.smithi029.stdout: librados-devel-2:16.2.4-0.el8.x86_64 2023-12-01T22:09:02.509 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-01T22:09:02.509 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-12-01T22:09:02.589 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2023-12-01T22:09:02.590 INFO:teuthology.orchestra.run.smithi073.stdout:Nothing to do. 2023-12-01T22:09:02.590 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2023-12-01T22:09:02.683 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install python3-cephfs 2023-12-01T22:09:02.707 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install libcephfs2 2023-12-01T22:09:03.539 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:02:36 ago on Fri 01 Dec 2023 10:06:27 PM UTC. 2023-12-01T22:09:03.703 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:02:41 ago on Fri 01 Dec 2023 10:06:22 PM UTC. 2023-12-01T22:09:04.968 INFO:teuthology.orchestra.run.smithi073.stdout:Package python3-cephfs-2:16.2.4-0.el8.x86_64 is already installed. 2023-12-01T22:09:05.190 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2023-12-01T22:09:05.192 INFO:teuthology.orchestra.run.smithi073.stdout:Nothing to do. 2023-12-01T22:09:05.192 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2023-12-01T22:09:05.290 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install python3-rbd 2023-12-01T22:09:05.315 INFO:teuthology.orchestra.run.smithi029.stdout:Package libcephfs2-2:16.2.4-0.el8.x86_64 is already installed. 2023-12-01T22:09:05.563 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-12-01T22:09:05.565 INFO:teuthology.orchestra.run.smithi029.stdout:Nothing to do. 2023-12-01T22:09:05.565 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-12-01T22:09:05.699 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install libcephfs-devel 2023-12-01T22:09:06.146 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:02:39 ago on Fri 01 Dec 2023 10:06:27 PM UTC. 2023-12-01T22:09:06.694 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:02:44 ago on Fri 01 Dec 2023 10:06:22 PM UTC. 2023-12-01T22:09:07.614 INFO:teuthology.orchestra.run.smithi073.stdout:Package python3-rbd-2:16.2.4-0.el8.x86_64 is already installed. 2023-12-01T22:09:07.838 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2023-12-01T22:09:07.839 INFO:teuthology.orchestra.run.smithi073.stdout:Nothing to do. 2023-12-01T22:09:07.839 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2023-12-01T22:09:07.930 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install rbd-fuse 2023-12-01T22:09:08.548 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-12-01T22:09:08.549 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-12-01T22:09:08.549 INFO:teuthology.orchestra.run.smithi029.stdout: Package Architecture Version Repository Size 2023-12-01T22:09:08.549 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-12-01T22:09:08.549 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2023-12-01T22:09:08.549 INFO:teuthology.orchestra.run.smithi029.stdout: libcephfs-devel x86_64 2:16.2.4-0.el8 ceph 24 k 2023-12-01T22:09:08.549 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-01T22:09:08.549 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2023-12-01T22:09:08.549 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-12-01T22:09:08.549 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2023-12-01T22:09:08.549 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-01T22:09:08.550 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 24 k 2023-12-01T22:09:08.550 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 94 k 2023-12-01T22:09:08.550 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2023-12-01T22:09:08.594 INFO:teuthology.orchestra.run.smithi029.stdout:libcephfs-devel-16.2.4-0.el8.x86_64.rpm 558 kB/s | 24 kB 00:00 2023-12-01T22:09:08.594 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2023-12-01T22:09:08.597 INFO:teuthology.orchestra.run.smithi029.stdout:Total 540 kB/s | 24 kB 00:00 2023-12-01T22:09:08.597 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2023-12-01T22:09:08.599 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2023-12-01T22:09:08.599 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2023-12-01T22:09:08.601 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2023-12-01T22:09:08.605 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2023-12-01T22:09:08.653 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2023-12-01T22:09:08.778 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:02:41 ago on Fri 01 Dec 2023 10:06:27 PM UTC. 2023-12-01T22:09:08.882 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2023-12-01T22:09:09.189 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2023-12-01T22:09:10.116 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2023-12-01T22:09:10.116 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-01T22:09:10.116 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2023-12-01T22:09:10.116 INFO:teuthology.orchestra.run.smithi029.stdout: libcephfs-devel-2:16.2.4-0.el8.x86_64 2023-12-01T22:09:10.116 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-01T22:09:10.116 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-12-01T22:09:10.248 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install librados2 2023-12-01T22:09:10.458 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2023-12-01T22:09:10.459 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2023-12-01T22:09:10.459 INFO:teuthology.orchestra.run.smithi073.stdout: Package Architecture Version Repository Size 2023-12-01T22:09:10.459 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2023-12-01T22:09:10.460 INFO:teuthology.orchestra.run.smithi073.stdout:Installing: 2023-12-01T22:09:10.460 INFO:teuthology.orchestra.run.smithi073.stdout: rbd-fuse x86_64 2:16.2.4-0.el8 ceph 89 k 2023-12-01T22:09:10.460 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:09:10.460 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction Summary 2023-12-01T22:09:10.460 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2023-12-01T22:09:10.460 INFO:teuthology.orchestra.run.smithi073.stdout:Install 1 Package 2023-12-01T22:09:10.460 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:09:10.460 INFO:teuthology.orchestra.run.smithi073.stdout:Total download size: 89 k 2023-12-01T22:09:10.461 INFO:teuthology.orchestra.run.smithi073.stdout:Installed size: 298 k 2023-12-01T22:09:10.461 INFO:teuthology.orchestra.run.smithi073.stdout:Downloading Packages: 2023-12-01T22:09:10.508 INFO:teuthology.orchestra.run.smithi073.stdout:rbd-fuse-16.2.4-0.el8.x86_64.rpm 1.8 MB/s | 89 kB 00:00 2023-12-01T22:09:10.508 INFO:teuthology.orchestra.run.smithi073.stdout:-------------------------------------------------------------------------------- 2023-12-01T22:09:10.508 INFO:teuthology.orchestra.run.smithi073.stdout:Total 1.8 MB/s | 89 kB 00:00 2023-12-01T22:09:10.509 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction check 2023-12-01T22:09:10.518 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction check succeeded. 2023-12-01T22:09:10.518 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction test 2023-12-01T22:09:10.568 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction test succeeded. 2023-12-01T22:09:10.570 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction 2023-12-01T22:09:10.707 INFO:teuthology.orchestra.run.smithi073.stdout: Preparing : 1/1 2023-12-01T22:09:10.907 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2023-12-01T22:09:11.188 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2023-12-01T22:09:11.241 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:02:49 ago on Fri 01 Dec 2023 10:06:22 PM UTC. 2023-12-01T22:09:11.899 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2023-12-01T22:09:11.899 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:09:11.899 INFO:teuthology.orchestra.run.smithi073.stdout:Installed: 2023-12-01T22:09:11.900 INFO:teuthology.orchestra.run.smithi073.stdout: rbd-fuse-2:16.2.4-0.el8.x86_64 2023-12-01T22:09:11.900 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:09:11.900 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2023-12-01T22:09:12.134 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install rbd-mirror 2023-12-01T22:09:12.874 INFO:teuthology.orchestra.run.smithi029.stdout:Package librados2-2:16.2.4-0.el8.x86_64 is already installed. 2023-12-01T22:09:12.998 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:02:45 ago on Fri 01 Dec 2023 10:06:27 PM UTC. 2023-12-01T22:09:13.127 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-12-01T22:09:13.128 INFO:teuthology.orchestra.run.smithi029.stdout:Nothing to do. 2023-12-01T22:09:13.129 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-12-01T22:09:13.251 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install librbd1 2023-12-01T22:09:14.242 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:02:52 ago on Fri 01 Dec 2023 10:06:22 PM UTC. 2023-12-01T22:09:14.687 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2023-12-01T22:09:14.688 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2023-12-01T22:09:14.688 INFO:teuthology.orchestra.run.smithi073.stdout: Package Architecture Version Repository Size 2023-12-01T22:09:14.688 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2023-12-01T22:09:14.688 INFO:teuthology.orchestra.run.smithi073.stdout:Installing: 2023-12-01T22:09:14.688 INFO:teuthology.orchestra.run.smithi073.stdout: rbd-mirror x86_64 2:16.2.4-0.el8 ceph 3.7 M 2023-12-01T22:09:14.688 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:09:14.688 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction Summary 2023-12-01T22:09:14.688 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2023-12-01T22:09:14.688 INFO:teuthology.orchestra.run.smithi073.stdout:Install 1 Package 2023-12-01T22:09:14.689 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:09:14.689 INFO:teuthology.orchestra.run.smithi073.stdout:Total download size: 3.7 M 2023-12-01T22:09:14.689 INFO:teuthology.orchestra.run.smithi073.stdout:Installed size: 18 M 2023-12-01T22:09:14.689 INFO:teuthology.orchestra.run.smithi073.stdout:Downloading Packages: 2023-12-01T22:09:14.777 INFO:teuthology.orchestra.run.smithi073.stdout:rbd-mirror-16.2.4-0.el8.x86_64.rpm 42 MB/s | 3.7 MB 00:00 2023-12-01T22:09:14.778 INFO:teuthology.orchestra.run.smithi073.stdout:-------------------------------------------------------------------------------- 2023-12-01T22:09:14.778 INFO:teuthology.orchestra.run.smithi073.stdout:Total 41 MB/s | 3.7 MB 00:00 2023-12-01T22:09:14.779 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction check 2023-12-01T22:09:14.790 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction check succeeded. 2023-12-01T22:09:14.790 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction test 2023-12-01T22:09:14.854 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction test succeeded. 2023-12-01T22:09:14.857 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction 2023-12-01T22:09:15.322 INFO:teuthology.orchestra.run.smithi073.stdout: Preparing : 1/1 2023-12-01T22:09:15.591 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2023-12-01T22:09:15.851 INFO:teuthology.orchestra.run.smithi029.stdout:Package librbd1-2:16.2.4-0.el8.x86_64 is already installed. 2023-12-01T22:09:16.025 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2023-12-01T22:09:16.099 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-12-01T22:09:16.101 INFO:teuthology.orchestra.run.smithi029.stdout:Nothing to do. 2023-12-01T22:09:16.101 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-12-01T22:09:16.232 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install python3-rados 2023-12-01T22:09:16.704 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2023-12-01T22:09:16.704 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:09:16.704 INFO:teuthology.orchestra.run.smithi073.stdout:Installed: 2023-12-01T22:09:16.704 INFO:teuthology.orchestra.run.smithi073.stdout: rbd-mirror-2:16.2.4-0.el8.x86_64 2023-12-01T22:09:16.705 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:09:16.705 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2023-12-01T22:09:16.916 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install rbd-nbd 2023-12-01T22:09:17.228 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:02:55 ago on Fri 01 Dec 2023 10:06:22 PM UTC. 2023-12-01T22:09:17.774 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:02:50 ago on Fri 01 Dec 2023 10:06:27 PM UTC. 2023-12-01T22:09:18.848 INFO:teuthology.orchestra.run.smithi029.stdout:Package python3-rados-2:16.2.4-0.el8.x86_64 is already installed. 2023-12-01T22:09:19.100 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-12-01T22:09:19.101 INFO:teuthology.orchestra.run.smithi029.stdout:Nothing to do. 2023-12-01T22:09:19.102 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-12-01T22:09:19.225 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install python3-rgw 2023-12-01T22:09:19.449 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2023-12-01T22:09:19.449 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2023-12-01T22:09:19.449 INFO:teuthology.orchestra.run.smithi073.stdout: Package Architecture Version Repository Size 2023-12-01T22:09:19.449 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2023-12-01T22:09:19.450 INFO:teuthology.orchestra.run.smithi073.stdout:Installing: 2023-12-01T22:09:19.450 INFO:teuthology.orchestra.run.smithi073.stdout: rbd-nbd x86_64 2:16.2.4-0.el8 ceph 173 k 2023-12-01T22:09:19.450 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:09:19.450 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction Summary 2023-12-01T22:09:19.450 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2023-12-01T22:09:19.450 INFO:teuthology.orchestra.run.smithi073.stdout:Install 1 Package 2023-12-01T22:09:19.450 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:09:19.451 INFO:teuthology.orchestra.run.smithi073.stdout:Total download size: 173 k 2023-12-01T22:09:19.451 INFO:teuthology.orchestra.run.smithi073.stdout:Installed size: 649 k 2023-12-01T22:09:19.451 INFO:teuthology.orchestra.run.smithi073.stdout:Downloading Packages: 2023-12-01T22:09:19.580 INFO:teuthology.orchestra.run.smithi073.stdout:rbd-nbd-16.2.4-0.el8.x86_64.rpm 1.3 MB/s | 173 kB 00:00 2023-12-01T22:09:19.581 INFO:teuthology.orchestra.run.smithi073.stdout:-------------------------------------------------------------------------------- 2023-12-01T22:09:19.581 INFO:teuthology.orchestra.run.smithi073.stdout:Total 1.3 MB/s | 173 kB 00:00 2023-12-01T22:09:19.581 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction check 2023-12-01T22:09:19.592 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction check succeeded. 2023-12-01T22:09:19.592 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction test 2023-12-01T22:09:19.641 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction test succeeded. 2023-12-01T22:09:19.644 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction 2023-12-01T22:09:19.764 INFO:teuthology.orchestra.run.smithi073.stdout: Preparing : 1/1 2023-12-01T22:09:19.989 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2023-12-01T22:09:20.224 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:02:58 ago on Fri 01 Dec 2023 10:06:22 PM UTC. 2023-12-01T22:09:20.276 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2023-12-01T22:09:20.958 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2023-12-01T22:09:20.958 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:09:20.958 INFO:teuthology.orchestra.run.smithi073.stdout:Installed: 2023-12-01T22:09:20.958 INFO:teuthology.orchestra.run.smithi073.stdout: rbd-nbd-2:16.2.4-0.el8.x86_64 2023-12-01T22:09:20.958 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:09:20.958 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2023-12-01T22:09:21.153 DEBUG:teuthology.parallel:result is None 2023-12-01T22:09:21.841 INFO:teuthology.orchestra.run.smithi029.stdout:Package python3-rgw-2:16.2.4-0.el8.x86_64 is already installed. 2023-12-01T22:09:22.096 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-12-01T22:09:22.097 INFO:teuthology.orchestra.run.smithi029.stdout:Nothing to do. 2023-12-01T22:09:22.097 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-12-01T22:09:22.232 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install python3-cephfs 2023-12-01T22:09:23.228 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:03:01 ago on Fri 01 Dec 2023 10:06:22 PM UTC. 2023-12-01T22:09:24.865 INFO:teuthology.orchestra.run.smithi029.stdout:Package python3-cephfs-2:16.2.4-0.el8.x86_64 is already installed. 2023-12-01T22:09:25.116 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-12-01T22:09:25.117 INFO:teuthology.orchestra.run.smithi029.stdout:Nothing to do. 2023-12-01T22:09:25.117 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-12-01T22:09:25.232 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install python3-rbd 2023-12-01T22:09:26.230 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:03:04 ago on Fri 01 Dec 2023 10:06:22 PM UTC. 2023-12-01T22:09:27.838 INFO:teuthology.orchestra.run.smithi029.stdout:Package python3-rbd-2:16.2.4-0.el8.x86_64 is already installed. 2023-12-01T22:09:28.090 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-12-01T22:09:28.092 INFO:teuthology.orchestra.run.smithi029.stdout:Nothing to do. 2023-12-01T22:09:28.092 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-12-01T22:09:28.217 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install rbd-fuse 2023-12-01T22:09:29.209 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:03:07 ago on Fri 01 Dec 2023 10:06:22 PM UTC. 2023-12-01T22:09:31.094 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-12-01T22:09:31.095 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-12-01T22:09:31.095 INFO:teuthology.orchestra.run.smithi029.stdout: Package Architecture Version Repository Size 2023-12-01T22:09:31.095 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-12-01T22:09:31.095 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2023-12-01T22:09:31.095 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-fuse x86_64 2:16.2.4-0.el8 ceph 89 k 2023-12-01T22:09:31.095 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-01T22:09:31.095 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2023-12-01T22:09:31.095 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-12-01T22:09:31.095 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2023-12-01T22:09:31.095 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-01T22:09:31.096 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 89 k 2023-12-01T22:09:31.096 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 298 k 2023-12-01T22:09:31.096 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2023-12-01T22:09:31.137 INFO:teuthology.orchestra.run.smithi029.stdout:rbd-fuse-16.2.4-0.el8.x86_64.rpm 2.1 MB/s | 89 kB 00:00 2023-12-01T22:09:31.138 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2023-12-01T22:09:31.138 INFO:teuthology.orchestra.run.smithi029.stdout:Total 2.1 MB/s | 89 kB 00:00 2023-12-01T22:09:31.138 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2023-12-01T22:09:31.149 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2023-12-01T22:09:31.149 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2023-12-01T22:09:31.206 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2023-12-01T22:09:31.209 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2023-12-01T22:09:31.326 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2023-12-01T22:09:31.538 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2023-12-01T22:09:31.859 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2023-12-01T22:09:32.644 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2023-12-01T22:09:32.645 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-01T22:09:32.645 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2023-12-01T22:09:32.645 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-fuse-2:16.2.4-0.el8.x86_64 2023-12-01T22:09:32.645 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-01T22:09:32.645 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-12-01T22:09:32.876 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install rbd-mirror 2023-12-01T22:09:33.874 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:03:11 ago on Fri 01 Dec 2023 10:06:22 PM UTC. 2023-12-01T22:09:35.749 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-12-01T22:09:35.750 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-12-01T22:09:35.750 INFO:teuthology.orchestra.run.smithi029.stdout: Package Architecture Version Repository Size 2023-12-01T22:09:35.750 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-12-01T22:09:35.750 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2023-12-01T22:09:35.750 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-mirror x86_64 2:16.2.4-0.el8 ceph 3.7 M 2023-12-01T22:09:35.750 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-01T22:09:35.750 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2023-12-01T22:09:35.750 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-12-01T22:09:35.750 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2023-12-01T22:09:35.750 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-01T22:09:35.751 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 3.7 M 2023-12-01T22:09:35.751 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 18 M 2023-12-01T22:09:35.751 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2023-12-01T22:09:35.878 INFO:teuthology.orchestra.run.smithi029.stdout:rbd-mirror-16.2.4-0.el8.x86_64.rpm 29 MB/s | 3.7 MB 00:00 2023-12-01T22:09:35.879 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2023-12-01T22:09:35.879 INFO:teuthology.orchestra.run.smithi029.stdout:Total 29 MB/s | 3.7 MB 00:00 2023-12-01T22:09:35.879 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2023-12-01T22:09:35.893 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2023-12-01T22:09:35.894 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2023-12-01T22:09:35.966 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2023-12-01T22:09:35.970 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2023-12-01T22:09:36.475 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2023-12-01T22:09:36.688 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2023-12-01T22:09:37.158 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2023-12-01T22:09:37.909 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2023-12-01T22:09:37.909 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-01T22:09:37.909 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2023-12-01T22:09:37.909 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-mirror-2:16.2.4-0.el8.x86_64 2023-12-01T22:09:37.910 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-01T22:09:37.910 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-12-01T22:09:38.140 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install rbd-nbd 2023-12-01T22:09:39.131 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:03:17 ago on Fri 01 Dec 2023 10:06:22 PM UTC. 2023-12-01T22:09:41.008 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-12-01T22:09:41.009 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-12-01T22:09:41.009 INFO:teuthology.orchestra.run.smithi029.stdout: Package Architecture Version Repository Size 2023-12-01T22:09:41.009 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-12-01T22:09:41.009 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2023-12-01T22:09:41.009 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-nbd x86_64 2:16.2.4-0.el8 ceph 173 k 2023-12-01T22:09:41.009 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-01T22:09:41.009 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2023-12-01T22:09:41.009 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-12-01T22:09:41.009 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2023-12-01T22:09:41.009 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-01T22:09:41.010 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 173 k 2023-12-01T22:09:41.010 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 649 k 2023-12-01T22:09:41.010 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2023-12-01T22:09:41.061 INFO:teuthology.orchestra.run.smithi029.stdout:rbd-nbd-16.2.4-0.el8.x86_64.rpm 3.3 MB/s | 173 kB 00:00 2023-12-01T22:09:41.062 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2023-12-01T22:09:41.062 INFO:teuthology.orchestra.run.smithi029.stdout:Total 3.2 MB/s | 173 kB 00:00 2023-12-01T22:09:41.062 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2023-12-01T22:09:41.075 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2023-12-01T22:09:41.075 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2023-12-01T22:09:41.131 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2023-12-01T22:09:41.135 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2023-12-01T22:09:41.254 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2023-12-01T22:09:41.515 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2023-12-01T22:09:41.844 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2023-12-01T22:09:42.582 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2023-12-01T22:09:42.583 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-01T22:09:42.583 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2023-12-01T22:09:42.583 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-nbd-2:16.2.4-0.el8.x86_64 2023-12-01T22:09:42.583 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-01T22:09:42.583 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-12-01T22:09:42.832 DEBUG:teuthology.parallel:result is None 2023-12-01T22:09:42.832 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2023-12-01T22:09:42.832 INFO:teuthology.packaging:ref: None 2023-12-01T22:09:42.832 INFO:teuthology.packaging:tag: v16.2.4 2023-12-01T22:09:42.832 INFO:teuthology.packaging:branch: None 2023-12-01T22:09:42.832 INFO:teuthology.packaging:sha1: 0dc074ce0b0b3b4f393f56dd32ef36884ebf45a9 2023-12-01T22:09:43.144 DEBUG:teuthology.repo_utils:git ls-remote https://github.com/ceph/ceph-ci v16.2.4^{} -> 3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-12-01T22:09:43.145 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-12-01T22:09:43.370 DEBUG:teuthology.orchestra.run.smithi029:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-12-01T22:09:43.399 INFO:teuthology.orchestra.run.smithi029.stdout:16.2.4-0.el8 2023-12-01T22:09:43.400 INFO:teuthology.packaging:The installed version of ceph is 16.2.4-0.el8 2023-12-01T22:09:43.400 INFO:teuthology.task.install:The correct ceph version 16.2.4-0 is installed. 2023-12-01T22:09:43.400 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2023-12-01T22:09:43.400 INFO:teuthology.packaging:ref: None 2023-12-01T22:09:43.400 INFO:teuthology.packaging:tag: v16.2.4 2023-12-01T22:09:43.401 INFO:teuthology.packaging:branch: None 2023-12-01T22:09:43.401 INFO:teuthology.packaging:sha1: 0dc074ce0b0b3b4f393f56dd32ef36884ebf45a9 2023-12-01T22:09:43.597 DEBUG:teuthology.repo_utils:git ls-remote https://github.com/ceph/ceph-ci v16.2.4^{} -> 3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-12-01T22:09:43.598 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-12-01T22:09:43.817 DEBUG:teuthology.orchestra.run.smithi073:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-12-01T22:09:43.845 INFO:teuthology.orchestra.run.smithi073.stdout:16.2.4-0.el8 2023-12-01T22:09:43.846 INFO:teuthology.packaging:The installed version of ceph is 16.2.4-0.el8 2023-12-01T22:09:43.846 INFO:teuthology.task.install:The correct ceph version 16.2.4-0 is installed. 2023-12-01T22:09:43.846 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-12-01T22:09:43.846 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-12-01T22:09:43.847 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-12-01T22:09:43.883 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2023-12-01T22:09:43.883 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-12-01T22:09:43.918 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-12-01T22:09:43.918 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-12-01T22:09:43.918 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/usr/bin/daemon-helper 2023-12-01T22:09:43.957 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-12-01T22:09:44.033 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2023-12-01T22:09:44.034 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd of=/usr/bin/daemon-helper 2023-12-01T22:09:44.067 DEBUG:teuthology.orchestra.run.smithi073:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-12-01T22:09:44.137 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-12-01T22:09:44.137 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-12-01T22:09:44.137 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/usr/bin/adjust-ulimits 2023-12-01T22:09:44.171 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-12-01T22:09:44.246 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2023-12-01T22:09:44.246 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd of=/usr/bin/adjust-ulimits 2023-12-01T22:09:44.277 DEBUG:teuthology.orchestra.run.smithi073:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-12-01T22:09:44.347 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-12-01T22:09:44.348 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-12-01T22:09:44.348 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/usr/bin/stdin-killer 2023-12-01T22:09:44.382 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-12-01T22:09:44.457 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2023-12-01T22:09:44.457 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd of=/usr/bin/stdin-killer 2023-12-01T22:09:44.488 DEBUG:teuthology.orchestra.run.smithi073:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-12-01T22:09:44.558 INFO:teuthology.run_tasks:Running task print... 2023-12-01T22:09:44.567 INFO:teuthology.task.print:**** done install task... 2023-12-01T22:09:44.568 INFO:teuthology.run_tasks:Running task cephadm... 2023-12-01T22:09:44.642 INFO:tasks.cephadm:Config: {'avoid_pacific_features': True, 'cephadm_branch': 'v16.2.4', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'image': 'docker.io/ceph/ceph:v16.2.4', 'roleless': True, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mon pg warn min per osd': 0}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': '0dc074ce0b0b3b4f393f56dd32ef36884ebf45a9'} 2023-12-01T22:09:44.642 INFO:tasks.cephadm:Cluster image is docker.io/ceph/ceph:v16.2.4 2023-12-01T22:09:44.644 INFO:tasks.cephadm:Cluster fsid is 5558ab6c-9096-11ee-95a2-87774f69a715 2023-12-01T22:09:44.644 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2023-12-01T22:09:44.644 INFO:tasks.cephadm:No mon roles; fabricating mons 2023-12-01T22:09:44.644 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi029': '172.21.15.29', 'mon.smithi073': '172.21.15.73'} 2023-12-01T22:09:44.644 INFO:tasks.cephadm:Normalizing hostnames... 2023-12-01T22:09:44.645 DEBUG:teuthology.orchestra.run.smithi029:> sudo hostname $(hostname -s) 2023-12-01T22:09:44.681 DEBUG:teuthology.orchestra.run.smithi073:> sudo hostname $(hostname -s) 2023-12-01T22:09:44.711 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref v16.2.4)... 2023-12-01T22:09:44.711 DEBUG:teuthology.orchestra.run.smithi029:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v16.2.4/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-12-01T22:09:44.860 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-rw-r--. 1 ubuntu ubuntu 292005 Dec 1 22:09 /home/ubuntu/cephtest/cephadm 2023-12-01T22:09:44.861 DEBUG:teuthology.orchestra.run.smithi073:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v16.2.4/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-12-01T22:09:45.079 INFO:teuthology.orchestra.run.smithi073.stdout:-rw-rw-r--. 1 ubuntu ubuntu 292005 Dec 1 22:09 /home/ubuntu/cephtest/cephadm 2023-12-01T22:09:45.080 DEBUG:teuthology.orchestra.run.smithi029:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-12-01T22:09:45.104 DEBUG:teuthology.orchestra.run.smithi073:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-12-01T22:09:45.135 DEBUG:teuthology.orchestra.run.smithi029:> sudo mkdir -p /etc/ceph 2023-12-01T22:09:45.179 DEBUG:teuthology.orchestra.run.smithi073:> sudo mkdir -p /etc/ceph 2023-12-01T22:09:45.209 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 777 /etc/ceph 2023-12-01T22:09:45.254 DEBUG:teuthology.orchestra.run.smithi073:> sudo chmod 777 /etc/ceph 2023-12-01T22:09:45.284 INFO:tasks.cephadm:Writing seed config... 2023-12-01T22:09:45.285 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2023-12-01T22:09:45.285 INFO:tasks.cephadm: override: [client] debug client = 20 2023-12-01T22:09:45.285 INFO:tasks.cephadm: override: [client] debug ms = 1 2023-12-01T22:09:45.285 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2023-12-01T22:09:45.285 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2023-12-01T22:09:45.285 INFO:tasks.cephadm: override: [global] mon pg warn min per osd = 0 2023-12-01T22:09:45.285 INFO:tasks.cephadm: override: [mds] debug mds = 20 2023-12-01T22:09:45.285 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2023-12-01T22:09:45.285 INFO:tasks.cephadm: override: [mds] debug ms = 1 2023-12-01T22:09:45.285 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2023-12-01T22:09:45.285 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2023-12-01T22:09:45.285 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2023-12-01T22:09:45.286 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2023-12-01T22:09:45.286 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2023-12-01T22:09:45.286 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2023-12-01T22:09:45.286 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2023-12-01T22:09:45.286 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2023-12-01T22:09:45.286 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2023-12-01T22:09:45.286 INFO:tasks.cephadm: override: [mon] debug mon = 20 2023-12-01T22:09:45.286 INFO:tasks.cephadm: override: [mon] debug ms = 1 2023-12-01T22:09:45.286 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2023-12-01T22:09:45.286 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2023-12-01T22:09:45.286 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2023-12-01T22:09:45.286 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2023-12-01T22:09:45.286 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2023-12-01T22:09:45.286 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2023-12-01T22:09:45.286 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2023-12-01T22:09:45.287 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2023-12-01T22:09:45.287 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2023-12-01T22:09:45.287 INFO:tasks.cephadm: override: [osd] debug ms = 1 2023-12-01T22:09:45.287 INFO:tasks.cephadm: override: [osd] debug osd = 20 2023-12-01T22:09:45.287 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2023-12-01T22:09:45.287 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2023-12-01T22:09:45.287 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2023-12-01T22:09:45.287 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2023-12-01T22:09:45.287 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2023-12-01T22:09:45.287 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2023-12-01T22:09:45.287 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2023-12-01T22:09:45.288 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-12-01T22:09:45.288 DEBUG:teuthology.orchestra.run.smithi029:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2023-12-01T22:09:45.317 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 5558ab6c-9096-11ee-95a2-87774f69a715 mon pg warn min per osd = 0 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2023-12-01T22:09:45.318 DEBUG:teuthology.orchestra.run.smithi029:mon.smithi029> sudo journalctl -f -n 0 -u ceph-5558ab6c-9096-11ee-95a2-87774f69a715@mon.smithi029.service 2023-12-01T22:09:45.360 INFO:tasks.cephadm:Bootstrapping... 2023-12-01T22:09:45.360 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 -v bootstrap --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.29 && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2023-12-01T22:09:45.394 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:-- Logs begin at Fri 2023-12-01 21:56:25 UTC. -- 2023-12-01T22:09:45.671 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2023-12-01T22:09:46.728 INFO:teuthology.orchestra.run.smithi029.stderr:/bin/podman: 4.6.1 2023-12-01T22:09:46.738 INFO:teuthology.orchestra.run.smithi029.stderr:Verifying podman|docker is present... 2023-12-01T22:09:46.739 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2023-12-01T22:09:46.764 INFO:teuthology.orchestra.run.smithi029.stderr:/bin/podman: 4.6.1 2023-12-01T22:09:46.777 INFO:teuthology.orchestra.run.smithi029.stderr:Verifying lvm2 is present... 2023-12-01T22:09:46.777 INFO:teuthology.orchestra.run.smithi029.stderr:Verifying time synchronization is in place... 2023-12-01T22:09:46.777 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: systemctl is-enabled chrony.service 2023-12-01T22:09:46.786 INFO:teuthology.orchestra.run.smithi029.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2023-12-01T22:09:46.786 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: systemctl is-active chrony.service 2023-12-01T22:09:46.793 INFO:teuthology.orchestra.run.smithi029.stderr:systemctl: inactive 2023-12-01T22:09:46.793 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: systemctl is-enabled chronyd.service 2023-12-01T22:09:46.800 INFO:teuthology.orchestra.run.smithi029.stderr:systemctl: enabled 2023-12-01T22:09:46.801 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: systemctl is-active chronyd.service 2023-12-01T22:09:46.807 INFO:teuthology.orchestra.run.smithi029.stderr:systemctl: active 2023-12-01T22:09:46.808 INFO:teuthology.orchestra.run.smithi029.stderr:Unit chronyd.service is enabled and running 2023-12-01T22:09:46.808 INFO:teuthology.orchestra.run.smithi029.stderr:Repeating the final host check... 2023-12-01T22:09:46.808 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2023-12-01T22:09:46.832 INFO:teuthology.orchestra.run.smithi029.stderr:/bin/podman: 4.6.1 2023-12-01T22:09:46.841 INFO:teuthology.orchestra.run.smithi029.stderr:podman|docker (/bin/podman) is present 2023-12-01T22:09:46.841 INFO:teuthology.orchestra.run.smithi029.stderr:systemctl is present 2023-12-01T22:09:46.842 INFO:teuthology.orchestra.run.smithi029.stderr:lvcreate is present 2023-12-01T22:09:46.842 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: systemctl is-enabled chrony.service 2023-12-01T22:09:46.848 INFO:teuthology.orchestra.run.smithi029.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2023-12-01T22:09:46.848 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: systemctl is-active chrony.service 2023-12-01T22:09:46.854 INFO:teuthology.orchestra.run.smithi029.stderr:systemctl: inactive 2023-12-01T22:09:46.855 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: systemctl is-enabled chronyd.service 2023-12-01T22:09:46.861 INFO:teuthology.orchestra.run.smithi029.stderr:systemctl: enabled 2023-12-01T22:09:46.862 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: systemctl is-active chronyd.service 2023-12-01T22:09:46.868 INFO:teuthology.orchestra.run.smithi029.stderr:systemctl: active 2023-12-01T22:09:46.868 INFO:teuthology.orchestra.run.smithi029.stderr:Unit chronyd.service is enabled and running 2023-12-01T22:09:46.868 INFO:teuthology.orchestra.run.smithi029.stderr:Host looks OK 2023-12-01T22:09:46.869 INFO:teuthology.orchestra.run.smithi029.stderr:Cluster fsid: 5558ab6c-9096-11ee-95a2-87774f69a715 2023-12-01T22:09:46.869 INFO:teuthology.orchestra.run.smithi029.stderr:Acquiring lock 140445701048808 on /run/cephadm/5558ab6c-9096-11ee-95a2-87774f69a715.lock 2023-12-01T22:09:46.869 INFO:teuthology.orchestra.run.smithi029.stderr:Lock 140445701048808 acquired on /run/cephadm/5558ab6c-9096-11ee-95a2-87774f69a715.lock 2023-12-01T22:09:46.870 INFO:teuthology.orchestra.run.smithi029.stderr:Verifying IP 172.21.15.29 port 3300 ... 2023-12-01T22:09:46.870 INFO:teuthology.orchestra.run.smithi029.stderr:Verifying IP 172.21.15.29 port 6789 ... 2023-12-01T22:09:46.870 INFO:teuthology.orchestra.run.smithi029.stderr:Base mon IP is 172.21.15.29, final addrv is [v2:172.21.15.29:3300,v1:172.21.15.29:6789] 2023-12-01T22:09:46.870 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /sbin/ip route ls 2023-12-01T22:09:46.873 INFO:teuthology.orchestra.run.smithi029.stderr:/sbin/ip: default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.29 metric 100 2023-12-01T22:09:46.873 INFO:teuthology.orchestra.run.smithi029.stderr:/sbin/ip: 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.29 metric 100 2023-12-01T22:09:46.873 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /sbin/ip -6 route ls 2023-12-01T22:09:46.876 INFO:teuthology.orchestra.run.smithi029.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2023-12-01T22:09:46.876 INFO:teuthology.orchestra.run.smithi029.stderr:/sbin/ip: fe80::/64 dev ens1f0 proto kernel metric 256 pref medium 2023-12-01T22:09:46.876 INFO:teuthology.orchestra.run.smithi029.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 1024 expires 1799sec hoplimit 64 pref medium 2023-12-01T22:09:46.876 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /sbin/ip -6 addr ls 2023-12-01T22:09:46.879 INFO:teuthology.orchestra.run.smithi029.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2023-12-01T22:09:46.879 INFO:teuthology.orchestra.run.smithi029.stderr:/sbin/ip: inet6 ::1/128 scope host 2023-12-01T22:09:46.879 INFO:teuthology.orchestra.run.smithi029.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2023-12-01T22:09:46.879 INFO:teuthology.orchestra.run.smithi029.stderr:/sbin/ip: 4: ens1f0: mtu 1500 state UP qlen 1000 2023-12-01T22:09:46.880 INFO:teuthology.orchestra.run.smithi029.stderr:/sbin/ip: inet6 fe80::ec4:7aff:febd:125a/64 scope link 2023-12-01T22:09:46.880 INFO:teuthology.orchestra.run.smithi029.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2023-12-01T22:09:46.881 INFO:teuthology.orchestra.run.smithi029.stderr:Mon IP 172.21.15.29 is in CIDR network 172.21.0.0/20 2023-12-01T22:09:46.881 INFO:teuthology.orchestra.run.smithi029.stderr:- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2023-12-01T22:09:46.882 INFO:teuthology.orchestra.run.smithi029.stderr:Pulling container image docker.io/ceph/ceph:v16.2.4... 2023-12-01T22:09:46.882 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman pull docker.io/ceph/ceph:v16.2.4 2023-12-01T22:09:46.988 INFO:teuthology.orchestra.run.smithi029.stderr:/bin/podman: Trying to pull docker.io/ceph/ceph:v16.2.4... 2023-12-01T22:09:53.462 INFO:teuthology.orchestra.run.smithi029.stderr:/bin/podman: Getting image source signatures 2023-12-01T22:09:53.468 INFO:teuthology.orchestra.run.smithi029.stderr:/bin/podman: Copying blob sha256:551101690791055467b1840be133b2e6b029d263771bc6c46a96bb7acdacac39 2023-12-01T22:09:53.468 INFO:teuthology.orchestra.run.smithi029.stderr:/bin/podman: Copying blob sha256:7a0437f04f83f084b7ed68ad9c4a4947e12fc4e1b006b38129bac89114ec3621 2023-12-01T22:10:06.051 INFO:teuthology.orchestra.run.smithi029.stderr:/bin/podman: Copying config sha256:8d91d370c2b86c07de46146aba8d36718eaefa69b1880c77fa312fda6efd7d29 2023-12-01T22:10:06.100 INFO:teuthology.orchestra.run.smithi029.stderr:/bin/podman: Writing manifest to image destination 2023-12-01T22:10:06.407 INFO:teuthology.orchestra.run.smithi029.stderr:/bin/podman: 8d91d370c2b86c07de46146aba8d36718eaefa69b1880c77fa312fda6efd7d29 2023-12-01T22:10:06.419 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi029 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --version 2023-12-01T22:10:07.056 INFO:teuthology.orchestra.run.smithi029.stderr:ceph: ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable) 2023-12-01T22:10:07.409 INFO:teuthology.orchestra.run.smithi029.stderr:Ceph version: ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable) 2023-12-01T22:10:07.409 INFO:teuthology.orchestra.run.smithi029.stderr:Extracting ceph user uid/gid from container image... 2023-12-01T22:10:07.409 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi029 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 -c %u %g /var/lib/ceph 2023-12-01T22:10:07.721 INFO:teuthology.orchestra.run.smithi029.stderr:stat: 167 167 2023-12-01T22:10:08.048 INFO:teuthology.orchestra.run.smithi029.stderr:Creating initial keys... 2023-12-01T22:10:08.048 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi029 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2023-12-01T22:10:08.358 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-authtool: AQDAWWplFdezFBAAbj1RzkTdQgf+U+OjlII06w== 2023-12-01T22:10:08.699 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi029 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2023-12-01T22:10:08.966 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-authtool: AQDAWWplLFI+ORAATBiLZf5w6+r+ycxx9rBFmw== 2023-12-01T22:10:09.316 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi029 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2023-12-01T22:10:09.616 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-authtool: AQDBWWplwxobJBAAXghjgHkNHPE7YMOMTzNBIQ== 2023-12-01T22:10:09.952 INFO:teuthology.orchestra.run.smithi029.stderr:Creating initial monmap... 2023-12-01T22:10:09.952 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/monmaptool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi029 -e CEPH_USE_RANDOM_NONCE=1 -v /tmp/ceph-tmpgsslyrwg:/tmp/monmap:z docker.io/ceph/ceph:v16.2.4 --create --clobber --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 --addv smithi029 [v2:172.21.15.29:3300,v1:172.21.15.29:6789] /tmp/monmap 2023-12-01T22:10:10.233 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2023-12-01T22:10:10.233 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to 5558ab6c-9096-11ee-95a2-87774f69a715 2023-12-01T22:10:10.233 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2023-12-01T22:10:10.577 INFO:teuthology.orchestra.run.smithi029.stderr:monmaptool for smithi029 [v2:172.21.15.29:3300,v1:172.21.15.29:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2023-12-01T22:10:10.577 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/monmaptool: set fsid to 5558ab6c-9096-11ee-95a2-87774f69a715 2023-12-01T22:10:10.577 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2023-12-01T22:10:10.577 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-12-01T22:10:10.577 INFO:teuthology.orchestra.run.smithi029.stderr:Creating mon... 2023-12-01T22:10:10.578 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-mon --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi029 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/5558ab6c-9096-11ee-95a2-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/5558ab6c-9096-11ee-95a2-87774f69a715/mon.smithi029:/var/lib/ceph/mon/ceph-smithi029:z -v /tmp/ceph-tmpsue8vkwy:/tmp/keyring:z -v /tmp/ceph-tmpgsslyrwg:/tmp/monmap:z docker.io/ceph/ceph:v16.2.4 --mkfs -i smithi029 --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -c /dev/null --monmap /tmp/monmap --keyring /tmp/keyring --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true 2023-12-01T22:10:10.851 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.849+0000 7f4a878a4700 0 set uid:gid to 167:167 (ceph:ceph) 2023-12-01T22:10:10.852 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.849+0000 7f4a878a4700 1 imported monmap: 2023-12-01T22:10:10.852 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: epoch 0 2023-12-01T22:10:10.852 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: fsid 5558ab6c-9096-11ee-95a2-87774f69a715 2023-12-01T22:10:10.852 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: last_changed 2023-12-01T22:10:10.228477+0000 2023-12-01T22:10:10.852 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: created 2023-12-01T22:10:10.228477+0000 2023-12-01T22:10:10.852 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: min_mon_release 0 (unknown) 2023-12-01T22:10:10.854 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: election_strategy: 1 2023-12-01T22:10:10.854 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: 0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.smithi029 2023-12-01T22:10:10.854 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: 2023-12-01T22:10:10.854 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.849+0000 7f4a878a4700 0 /usr/bin/ceph-mon: set fsid to 5558ab6c-9096-11ee-95a2-87774f69a715 2023-12-01T22:10:10.854 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.850+0000 7f4a878a4700 4 rocksdb: RocksDB version: 6.8.1 2023-12-01T22:10:10.855 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: 2023-12-01T22:10:10.855 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.850+0000 7f4a878a4700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2023-12-01T22:10:10.855 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.850+0000 7f4a878a4700 4 rocksdb: Compile date May 13 2021 2023-12-01T22:10:10.855 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.850+0000 7f4a878a4700 4 rocksdb: DB SUMMARY 2023-12-01T22:10:10.855 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: 2023-12-01T22:10:10.855 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.850+0000 7f4a878a4700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi029/store.db dir, Total Num: 0, files: 2023-12-01T22:10:10.855 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: 2023-12-01T22:10:10.855 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.850+0000 7f4a878a4700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi029/store.db: 2023-12-01T22:10:10.855 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: 2023-12-01T22:10:10.855 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.850+0000 7f4a878a4700 4 rocksdb: Options.error_if_exists: 0 2023-12-01T22:10:10.855 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.850+0000 7f4a878a4700 4 rocksdb: Options.create_if_missing: 1 2023-12-01T22:10:10.855 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.850+0000 7f4a878a4700 4 rocksdb: Options.paranoid_checks: 1 2023-12-01T22:10:10.855 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.850+0000 7f4a878a4700 4 rocksdb: Options.env: 0x565298b461c0 2023-12-01T22:10:10.855 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.850+0000 7f4a878a4700 4 rocksdb: Options.fs: Posix File System 2023-12-01T22:10:10.855 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.850+0000 7f4a878a4700 4 rocksdb: Options.info_log: 0x56529a5b6ec0 2023-12-01T22:10:10.856 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.850+0000 7f4a878a4700 4 rocksdb: Options.max_file_opening_threads: 16 2023-12-01T22:10:10.866 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.850+0000 7f4a878a4700 4 rocksdb: Options.statistics: (nil) 2023-12-01T22:10:10.866 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.850+0000 7f4a878a4700 4 rocksdb: Options.use_fsync: 0 2023-12-01T22:10:10.869 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.850+0000 7f4a878a4700 4 rocksdb: Options.max_log_file_size: 0 2023-12-01T22:10:10.869 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.850+0000 7f4a878a4700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2023-12-01T22:10:10.869 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.850+0000 7f4a878a4700 4 rocksdb: Options.log_file_time_to_roll: 0 2023-12-01T22:10:10.869 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.850+0000 7f4a878a4700 4 rocksdb: Options.keep_log_file_num: 1000 2023-12-01T22:10:10.869 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.850+0000 7f4a878a4700 4 rocksdb: Options.recycle_log_file_num: 0 2023-12-01T22:10:10.869 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.850+0000 7f4a878a4700 4 rocksdb: Options.allow_fallocate: 1 2023-12-01T22:10:10.869 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.850+0000 7f4a878a4700 4 rocksdb: Options.allow_mmap_reads: 0 2023-12-01T22:10:10.869 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.850+0000 7f4a878a4700 4 rocksdb: Options.allow_mmap_writes: 0 2023-12-01T22:10:10.869 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.850+0000 7f4a878a4700 4 rocksdb: Options.use_direct_reads: 0 2023-12-01T22:10:10.869 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.850+0000 7f4a878a4700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-12-01T22:10:10.869 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.850+0000 7f4a878a4700 4 rocksdb: Options.create_missing_column_families: 0 2023-12-01T22:10:10.870 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.850+0000 7f4a878a4700 4 rocksdb: Options.db_log_dir: 2023-12-01T22:10:10.870 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.850+0000 7f4a878a4700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi029/store.db 2023-12-01T22:10:10.870 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.850+0000 7f4a878a4700 4 rocksdb: Options.table_cache_numshardbits: 6 2023-12-01T22:10:10.870 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.850+0000 7f4a878a4700 4 rocksdb: Options.max_subcompactions: 1 2023-12-01T22:10:10.870 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.850+0000 7f4a878a4700 4 rocksdb: Options.max_background_flushes: -1 2023-12-01T22:10:10.870 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.850+0000 7f4a878a4700 4 rocksdb: Options.WAL_ttl_seconds: 0 2023-12-01T22:10:10.870 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.850+0000 7f4a878a4700 4 rocksdb: Options.WAL_size_limit_MB: 0 2023-12-01T22:10:10.870 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.850+0000 7f4a878a4700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-12-01T22:10:10.870 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.850+0000 7f4a878a4700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2023-12-01T22:10:10.870 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.850+0000 7f4a878a4700 4 rocksdb: Options.is_fd_close_on_exec: 1 2023-12-01T22:10:10.870 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.850+0000 7f4a878a4700 4 rocksdb: Options.advise_random_on_open: 1 2023-12-01T22:10:10.870 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.850+0000 7f4a878a4700 4 rocksdb: Options.db_write_buffer_size: 0 2023-12-01T22:10:10.870 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.850+0000 7f4a878a4700 4 rocksdb: Options.write_buffer_manager: 0x56529a5bd530 2023-12-01T22:10:10.870 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.850+0000 7f4a878a4700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2023-12-01T22:10:10.871 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.850+0000 7f4a878a4700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2023-12-01T22:10:10.871 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.850+0000 7f4a878a4700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2023-12-01T22:10:10.871 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.850+0000 7f4a878a4700 4 rocksdb: Options.use_adaptive_mutex: 0 2023-12-01T22:10:10.871 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.850+0000 7f4a878a4700 4 rocksdb: Options.rate_limiter: (nil) 2023-12-01T22:10:10.871 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.850+0000 7f4a878a4700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-12-01T22:10:10.871 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.850+0000 7f4a878a4700 4 rocksdb: Options.wal_recovery_mode: 2 2023-12-01T22:10:10.871 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.850+0000 7f4a878a4700 4 rocksdb: Options.enable_thread_tracking: 0 2023-12-01T22:10:10.871 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.850+0000 7f4a878a4700 4 rocksdb: Options.enable_pipelined_write: 0 2023-12-01T22:10:10.871 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.850+0000 7f4a878a4700 4 rocksdb: Options.unordered_write: 0 2023-12-01T22:10:10.871 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.850+0000 7f4a878a4700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2023-12-01T22:10:10.871 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.850+0000 7f4a878a4700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-12-01T22:10:10.872 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.850+0000 7f4a878a4700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2023-12-01T22:10:10.872 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.850+0000 7f4a878a4700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2023-12-01T22:10:10.872 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.850+0000 7f4a878a4700 4 rocksdb: Options.row_cache: None 2023-12-01T22:10:10.872 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.850+0000 7f4a878a4700 4 rocksdb: Options.wal_filter: None 2023-12-01T22:10:10.872 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.850+0000 7f4a878a4700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2023-12-01T22:10:10.872 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.850+0000 7f4a878a4700 4 rocksdb: Options.allow_ingest_behind: 0 2023-12-01T22:10:10.872 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.850+0000 7f4a878a4700 4 rocksdb: Options.preserve_deletes: 0 2023-12-01T22:10:10.872 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.850+0000 7f4a878a4700 4 rocksdb: Options.two_write_queues: 0 2023-12-01T22:10:10.872 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.850+0000 7f4a878a4700 4 rocksdb: Options.manual_wal_flush: 0 2023-12-01T22:10:10.873 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.850+0000 7f4a878a4700 4 rocksdb: Options.atomic_flush: 0 2023-12-01T22:10:10.873 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.850+0000 7f4a878a4700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-12-01T22:10:10.873 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.850+0000 7f4a878a4700 4 rocksdb: Options.persist_stats_to_disk: 0 2023-12-01T22:10:10.873 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.850+0000 7f4a878a4700 4 rocksdb: Options.write_dbid_to_manifest: 0 2023-12-01T22:10:10.873 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.850+0000 7f4a878a4700 4 rocksdb: Options.log_readahead_size: 0 2023-12-01T22:10:10.873 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.850+0000 7f4a878a4700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2023-12-01T22:10:10.873 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.850+0000 7f4a878a4700 4 rocksdb: Options.max_background_jobs: 2 2023-12-01T22:10:10.873 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.850+0000 7f4a878a4700 4 rocksdb: Options.max_background_compactions: -1 2023-12-01T22:10:10.873 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.850+0000 7f4a878a4700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2023-12-01T22:10:10.873 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.850+0000 7f4a878a4700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-12-01T22:10:10.873 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.850+0000 7f4a878a4700 4 rocksdb: Options.delayed_write_rate : 16777216 2023-12-01T22:10:10.873 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.850+0000 7f4a878a4700 4 rocksdb: Options.max_total_wal_size: 0 2023-12-01T22:10:10.873 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.850+0000 7f4a878a4700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-12-01T22:10:10.873 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.850+0000 7f4a878a4700 4 rocksdb: Options.stats_dump_period_sec: 600 2023-12-01T22:10:10.873 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.850+0000 7f4a878a4700 4 rocksdb: Options.stats_persist_period_sec: 600 2023-12-01T22:10:10.874 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.850+0000 7f4a878a4700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2023-12-01T22:10:10.874 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.850+0000 7f4a878a4700 4 rocksdb: Options.max_open_files: -1 2023-12-01T22:10:10.874 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.850+0000 7f4a878a4700 4 rocksdb: Options.bytes_per_sync: 0 2023-12-01T22:10:10.874 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.850+0000 7f4a878a4700 4 rocksdb: Options.wal_bytes_per_sync: 0 2023-12-01T22:10:10.874 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.850+0000 7f4a878a4700 4 rocksdb: Options.strict_bytes_per_sync: 0 2023-12-01T22:10:10.874 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.850+0000 7f4a878a4700 4 rocksdb: Options.compaction_readahead_size: 0 2023-12-01T22:10:10.874 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.850+0000 7f4a878a4700 4 rocksdb: Compression algorithms supported: 2023-12-01T22:10:10.874 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.850+0000 7f4a878a4700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2023-12-01T22:10:10.874 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.850+0000 7f4a878a4700 4 rocksdb: kZSTD supported: 0 2023-12-01T22:10:10.874 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.850+0000 7f4a878a4700 4 rocksdb: kXpressCompression supported: 0 2023-12-01T22:10:10.874 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.850+0000 7f4a878a4700 4 rocksdb: kLZ4HCCompression supported: 1 2023-12-01T22:10:10.874 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.850+0000 7f4a878a4700 4 rocksdb: kLZ4Compression supported: 1 2023-12-01T22:10:10.874 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.850+0000 7f4a878a4700 4 rocksdb: kBZip2Compression supported: 0 2023-12-01T22:10:10.875 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.850+0000 7f4a878a4700 4 rocksdb: kZlibCompression supported: 1 2023-12-01T22:10:10.875 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.850+0000 7f4a878a4700 4 rocksdb: kSnappyCompression supported: 1 2023-12-01T22:10:10.875 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.850+0000 7f4a878a4700 4 rocksdb: Fast CRC32 supported: Supported on x86 2023-12-01T22:10:10.875 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.851+0000 7f4a878a4700 4 rocksdb: [db_impl/db_impl_open.cc:273] Creating manifest 1 2023-12-01T22:10:10.875 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: 2023-12-01T22:10:10.875 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.851+0000 7f4a878a4700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi029/store.db/MANIFEST-000001 2023-12-01T22:10:10.875 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: 2023-12-01T22:10:10.875 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.851+0000 7f4a878a4700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2023-12-01T22:10:10.875 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: 2023-12-01T22:10:10.876 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.851+0000 7f4a878a4700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-12-01T22:10:10.876 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.851+0000 7f4a878a4700 4 rocksdb: Options.merge_operator: 2023-12-01T22:10:10.876 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.851+0000 7f4a878a4700 4 rocksdb: Options.compaction_filter: None 2023-12-01T22:10:10.876 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.851+0000 7f4a878a4700 4 rocksdb: Options.compaction_filter_factory: None 2023-12-01T22:10:10.876 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.851+0000 7f4a878a4700 4 rocksdb: Options.memtable_factory: SkipListFactory 2023-12-01T22:10:10.876 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.851+0000 7f4a878a4700 4 rocksdb: Options.table_factory: BlockBasedTable 2023-12-01T22:10:10.876 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.851+0000 7f4a878a4700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x56529a4ce120) 2023-12-01T22:10:10.876 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks: 1 2023-12-01T22:10:10.876 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks_with_high_priority: 0 2023-12-01T22:10:10.876 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: pin_l0_filter_and_index_blocks_in_cache: 0 2023-12-01T22:10:10.876 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: pin_top_level_index_and_filter: 1 2023-12-01T22:10:10.876 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: index_type: 0 2023-12-01T22:10:10.876 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: data_block_index_type: 0 2023-12-01T22:10:10.876 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: index_shortening: 1 2023-12-01T22:10:10.877 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: data_block_hash_table_util_ratio: 0.750000 2023-12-01T22:10:10.877 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: hash_index_allow_collision: 1 2023-12-01T22:10:10.877 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: checksum: 1 2023-12-01T22:10:10.877 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: no_block_cache: 0 2023-12-01T22:10:10.877 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: block_cache: 0x56529a504d10 2023-12-01T22:10:10.877 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: block_cache_name: BinnedLRUCache 2023-12-01T22:10:10.877 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: block_cache_options: 2023-12-01T22:10:10.877 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: capacity : 536870912 2023-12-01T22:10:10.877 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: num_shard_bits : 4 2023-12-01T22:10:10.877 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: strict_capacity_limit : 0 2023-12-01T22:10:10.877 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: high_pri_pool_ratio: 0.000 2023-12-01T22:10:10.877 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: block_cache_compressed: (nil) 2023-12-01T22:10:10.877 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: persistent_cache: (nil) 2023-12-01T22:10:10.877 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: block_size: 4096 2023-12-01T22:10:10.877 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: block_size_deviation: 10 2023-12-01T22:10:10.877 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: block_restart_interval: 16 2023-12-01T22:10:10.878 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: index_block_restart_interval: 1 2023-12-01T22:10:10.878 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: metadata_block_size: 4096 2023-12-01T22:10:10.878 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: partition_filters: 0 2023-12-01T22:10:10.878 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: use_delta_encoding: 1 2023-12-01T22:10:10.878 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: filter_policy: rocksdb.BuiltinBloomFilter 2023-12-01T22:10:10.878 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: whole_key_filtering: 1 2023-12-01T22:10:10.878 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: verify_compression: 0 2023-12-01T22:10:10.878 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: read_amp_bytes_per_bit: 0 2023-12-01T22:10:10.878 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: format_version: 2 2023-12-01T22:10:10.878 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: enable_index_compression: 1 2023-12-01T22:10:10.890 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: block_align: 0 2023-12-01T22:10:10.890 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: 2023-12-01T22:10:10.890 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.851+0000 7f4a878a4700 4 rocksdb: Options.write_buffer_size: 33554432 2023-12-01T22:10:10.890 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.851+0000 7f4a878a4700 4 rocksdb: Options.max_write_buffer_number: 2 2023-12-01T22:10:10.891 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.851+0000 7f4a878a4700 4 rocksdb: Options.compression: NoCompression 2023-12-01T22:10:10.891 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.851+0000 7f4a878a4700 4 rocksdb: Options.bottommost_compression: Disabled 2023-12-01T22:10:10.891 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.851+0000 7f4a878a4700 4 rocksdb: Options.prefix_extractor: nullptr 2023-12-01T22:10:10.891 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.851+0000 7f4a878a4700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-12-01T22:10:10.891 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.851+0000 7f4a878a4700 4 rocksdb: Options.num_levels: 7 2023-12-01T22:10:10.891 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.851+0000 7f4a878a4700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-12-01T22:10:10.891 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.851+0000 7f4a878a4700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-12-01T22:10:10.891 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.851+0000 7f4a878a4700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-12-01T22:10:10.891 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.851+0000 7f4a878a4700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-12-01T22:10:10.891 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.851+0000 7f4a878a4700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2023-12-01T22:10:10.891 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.851+0000 7f4a878a4700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-12-01T22:10:10.891 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.851+0000 7f4a878a4700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-12-01T22:10:10.891 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.851+0000 7f4a878a4700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-12-01T22:10:10.891 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.851+0000 7f4a878a4700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2023-12-01T22:10:10.891 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.851+0000 7f4a878a4700 4 rocksdb: Options.compression_opts.window_bits: -14 2023-12-01T22:10:10.892 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.851+0000 7f4a878a4700 4 rocksdb: Options.compression_opts.level: 32767 2023-12-01T22:10:10.892 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.851+0000 7f4a878a4700 4 rocksdb: Options.compression_opts.strategy: 0 2023-12-01T22:10:10.892 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.851+0000 7f4a878a4700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-12-01T22:10:10.892 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.851+0000 7f4a878a4700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-12-01T22:10:10.892 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.851+0000 7f4a878a4700 4 rocksdb: Options.compression_opts.enabled: false 2023-12-01T22:10:10.892 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.851+0000 7f4a878a4700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-12-01T22:10:10.892 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.851+0000 7f4a878a4700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-12-01T22:10:10.892 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.851+0000 7f4a878a4700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2023-12-01T22:10:10.892 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.851+0000 7f4a878a4700 4 rocksdb: Options.target_file_size_base: 67108864 2023-12-01T22:10:10.892 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.851+0000 7f4a878a4700 4 rocksdb: Options.target_file_size_multiplier: 1 2023-12-01T22:10:10.892 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.851+0000 7f4a878a4700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2023-12-01T22:10:10.892 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.851+0000 7f4a878a4700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-12-01T22:10:10.892 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.851+0000 7f4a878a4700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-12-01T22:10:10.892 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.851+0000 7f4a878a4700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-12-01T22:10:10.892 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.851+0000 7f4a878a4700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-12-01T22:10:10.893 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.851+0000 7f4a878a4700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-12-01T22:10:10.893 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.851+0000 7f4a878a4700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-12-01T22:10:10.893 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.851+0000 7f4a878a4700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-12-01T22:10:10.893 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.851+0000 7f4a878a4700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-12-01T22:10:10.893 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.851+0000 7f4a878a4700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-12-01T22:10:10.893 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.851+0000 7f4a878a4700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-12-01T22:10:10.893 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.851+0000 7f4a878a4700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2023-12-01T22:10:10.893 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.851+0000 7f4a878a4700 4 rocksdb: Options.arena_block_size: 4194304 2023-12-01T22:10:10.893 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.851+0000 7f4a878a4700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-12-01T22:10:10.893 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.851+0000 7f4a878a4700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-12-01T22:10:10.893 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.851+0000 7f4a878a4700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2023-12-01T22:10:10.893 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.851+0000 7f4a878a4700 4 rocksdb: Options.disable_auto_compactions: 0 2023-12-01T22:10:10.893 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.851+0000 7f4a878a4700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-12-01T22:10:10.893 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.851+0000 7f4a878a4700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-12-01T22:10:10.893 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.851+0000 7f4a878a4700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-12-01T22:10:10.894 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.851+0000 7f4a878a4700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-12-01T22:10:10.894 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.851+0000 7f4a878a4700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-12-01T22:10:10.894 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.851+0000 7f4a878a4700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-12-01T22:10:10.894 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.851+0000 7f4a878a4700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-12-01T22:10:10.894 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.851+0000 7f4a878a4700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-12-01T22:10:10.894 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.851+0000 7f4a878a4700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-12-01T22:10:10.894 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.851+0000 7f4a878a4700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-12-01T22:10:10.894 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.851+0000 7f4a878a4700 4 rocksdb: Options.table_properties_collectors: 2023-12-01T22:10:10.894 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.851+0000 7f4a878a4700 4 rocksdb: Options.inplace_update_support: 0 2023-12-01T22:10:10.894 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.851+0000 7f4a878a4700 4 rocksdb: Options.inplace_update_num_locks: 10000 2023-12-01T22:10:10.894 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.851+0000 7f4a878a4700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-12-01T22:10:10.894 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.851+0000 7f4a878a4700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2023-12-01T22:10:10.894 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.851+0000 7f4a878a4700 4 rocksdb: Options.memtable_huge_page_size: 0 2023-12-01T22:10:10.894 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.851+0000 7f4a878a4700 4 rocksdb: Options.bloom_locality: 0 2023-12-01T22:10:10.895 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.851+0000 7f4a878a4700 4 rocksdb: Options.max_successive_merges: 0 2023-12-01T22:10:10.895 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.851+0000 7f4a878a4700 4 rocksdb: Options.optimize_filters_for_hits: 0 2023-12-01T22:10:10.895 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.851+0000 7f4a878a4700 4 rocksdb: Options.paranoid_file_checks: 0 2023-12-01T22:10:10.895 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.851+0000 7f4a878a4700 4 rocksdb: Options.force_consistency_checks: 0 2023-12-01T22:10:10.895 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.851+0000 7f4a878a4700 4 rocksdb: Options.report_bg_io_stats: 0 2023-12-01T22:10:10.895 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.851+0000 7f4a878a4700 4 rocksdb: Options.ttl: 2592000 2023-12-01T22:10:10.895 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.851+0000 7f4a878a4700 4 rocksdb: Options.periodic_compaction_seconds: 0 2023-12-01T22:10:10.895 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.852+0000 7f4a878a4700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi029/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-12-01T22:10:10.895 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: 2023-12-01T22:10:10.895 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.852+0000 7f4a878a4700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 0 2023-12-01T22:10:10.895 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: 2023-12-01T22:10:10.895 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.853+0000 7f4a878a4700 4 rocksdb: DB pointer 0x56529a5c9800 2023-12-01T22:10:10.895 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.853+0000 7f4a7073d700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2023-12-01T22:10:10.895 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.853+0000 7f4a7073d700 4 rocksdb: [db_impl/db_impl.cc:851] 2023-12-01T22:10:10.895 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: ** DB Stats ** 2023-12-01T22:10:10.896 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2023-12-01T22:10:10.896 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-12-01T22:10:10.896 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-12-01T22:10:10.896 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-12-01T22:10:10.896 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-12-01T22:10:10.896 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2023-12-01T22:10:10.896 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-12-01T22:10:10.896 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: 2023-12-01T22:10:10.896 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2023-12-01T22:10:10.896 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-12-01T22:10:10.896 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-12-01T22:10:10.896 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-12-01T22:10:10.896 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-12-01T22:10:10.896 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: 2023-12-01T22:10:10.896 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2023-12-01T22:10:10.896 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-12-01T22:10:10.897 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-12-01T22:10:10.897 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2023-12-01T22:10:10.897 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2023-12-01T22:10:10.897 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2023-12-01T22:10:10.897 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2023-12-01T22:10:10.897 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2023-12-01T22:10:10.897 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2023-12-01T22:10:10.897 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-12-01T22:10:10.897 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-12-01T22:10:10.897 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-12-01T22:10:10.897 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: 2023-12-01T22:10:10.897 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2023-12-01T22:10:10.897 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: 2023-12-01T22:10:10.897 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2023-12-01T22:10:10.897 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-12-01T22:10:10.898 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-12-01T22:10:10.898 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-12-01T22:10:10.898 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-12-01T22:10:10.898 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: 2023-12-01T22:10:10.898 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2023-12-01T22:10:10.898 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-12-01T22:10:10.898 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-12-01T22:10:10.898 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2023-12-01T22:10:10.898 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2023-12-01T22:10:10.898 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2023-12-01T22:10:10.898 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2023-12-01T22:10:10.898 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2023-12-01T22:10:10.898 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2023-12-01T22:10:10.899 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-12-01T22:10:10.899 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-12-01T22:10:10.899 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-12-01T22:10:10.899 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: 2023-12-01T22:10:10.899 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2023-12-01T22:10:10.899 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: 2023-12-01T22:10:10.899 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.854+0000 7f4a878a4700 4 rocksdb: [db_impl/db_impl.cc:397] Shutdown: canceling all background work 2023-12-01T22:10:10.899 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.855+0000 7f4a878a4700 4 rocksdb: [db_impl/db_impl.cc:573] Shutdown complete 2023-12-01T22:10:10.899 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2023-12-01T22:10:10.855+0000 7f4a878a4700 0 /usr/bin/ceph-mon: created monfs at /var/lib/ceph/mon/ceph-smithi029 for mon.smithi029 2023-12-01T22:10:11.221 INFO:teuthology.orchestra.run.smithi029.stderr:create mon.smithi029 on 2023-12-01T22:10:11.221 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: install -d -m0770 -o 167 -g 167 /var/run/ceph/5558ab6c-9096-11ee-95a2-87774f69a715 2023-12-01T22:10:11.228 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: systemctl disable ceph.target 2023-12-01T22:10:11.237 INFO:teuthology.orchestra.run.smithi029.stderr:systemctl: Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2023-12-01T22:10:11.337 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: systemctl enable ceph.target 2023-12-01T22:10:11.346 INFO:teuthology.orchestra.run.smithi029.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2023-12-01T22:10:11.439 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: systemctl start ceph.target 2023-12-01T22:10:11.447 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: systemctl enable ceph-5558ab6c-9096-11ee-95a2-87774f69a715.target 2023-12-01T22:10:11.453 INFO:teuthology.orchestra.run.smithi029.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-5558ab6c-9096-11ee-95a2-87774f69a715.target → /etc/systemd/system/ceph-5558ab6c-9096-11ee-95a2-87774f69a715.target. 2023-12-01T22:10:11.453 INFO:teuthology.orchestra.run.smithi029.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-5558ab6c-9096-11ee-95a2-87774f69a715.target → /etc/systemd/system/ceph-5558ab6c-9096-11ee-95a2-87774f69a715.target. 2023-12-01T22:10:11.545 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: systemctl start ceph-5558ab6c-9096-11ee-95a2-87774f69a715.target 2023-12-01T22:10:11.553 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: systemctl daemon-reload 2023-12-01T22:10:11.647 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: systemctl stop ceph-5558ab6c-9096-11ee-95a2-87774f69a715@mon.smithi029 2023-12-01T22:10:11.676 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: systemctl reset-failed ceph-5558ab6c-9096-11ee-95a2-87774f69a715@mon.smithi029 2023-12-01T22:10:11.681 INFO:teuthology.orchestra.run.smithi029.stderr:systemctl: Failed to reset failed state of unit ceph-5558ab6c-9096-11ee-95a2-87774f69a715@mon.smithi029.service: Unit ceph-5558ab6c-9096-11ee-95a2-87774f69a715@mon.smithi029.service not loaded. 2023-12-01T22:10:11.682 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: systemctl enable ceph-5558ab6c-9096-11ee-95a2-87774f69a715@mon.smithi029 2023-12-01T22:10:11.689 INFO:teuthology.orchestra.run.smithi029.stderr:systemctl: Created symlink /etc/systemd/system/ceph-5558ab6c-9096-11ee-95a2-87774f69a715.target.wants/ceph-5558ab6c-9096-11ee-95a2-87774f69a715@mon.smithi029.service → /etc/systemd/system/ceph-5558ab6c-9096-11ee-95a2-87774f69a715@.service. 2023-12-01T22:10:11.785 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: systemctl start ceph-5558ab6c-9096-11ee-95a2-87774f69a715@mon.smithi029 2023-12-01T22:10:12.113 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: systemctl is-enabled firewalld.service 2023-12-01T22:10:12.124 INFO:teuthology.orchestra.run.smithi029.stderr:systemctl: disabled 2023-12-01T22:10:12.124 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: systemctl is-active firewalld.service 2023-12-01T22:10:12.133 INFO:teuthology.orchestra.run.smithi029.stderr:systemctl: inactive 2023-12-01T22:10:12.134 INFO:teuthology.orchestra.run.smithi029.stderr:firewalld.service is not enabled 2023-12-01T22:10:12.135 INFO:teuthology.orchestra.run.smithi029.stderr:Not possible to enable service . firewalld.service is not available 2023-12-01T22:10:12.135 INFO:teuthology.orchestra.run.smithi029.stderr:Not possible to open ports <[]>. firewalld.service is not available 2023-12-01T22:10:12.135 INFO:teuthology.orchestra.run.smithi029.stderr:Waiting for mon to start... 2023-12-01T22:10:12.135 INFO:teuthology.orchestra.run.smithi029.stderr:Waiting for mon... 2023-12-01T22:10:12.135 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi029 -e CEPH_USE_RANDOM_NONCE=1 -v /var/lib/ceph/5558ab6c-9096-11ee-95a2-87774f69a715/mon.smithi029:/var/lib/ceph/mon/ceph-smithi029:z -v /tmp/ceph-tmpou9lnrt4:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpxe9edifa:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status 2023-12-01T22:10:12.360 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:12 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111363]: cluster 2023-12-01T22:10:12.125788+0000 mon.smithi029 (mon.0) 1 : cluster [INF] mon.smithi029 is new leader, mons smithi029 in quorum (ranks 0) 2023-12-01T22:10:12.575 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:12.571+0000 7fe1bfbf1700 1 Processor -- start 2023-12-01T22:10:12.575 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:12.571+0000 7fe1bfbf1700 1 -- start start 2023-12-01T22:10:12.575 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:12.572+0000 7fe1bfbf1700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe1b8070430 0x7fe1b80fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-01T22:10:12.575 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:12.572+0000 7fe1bfbf1700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1b80fedd0 con 0x7fe1b8070430 2023-12-01T22:10:12.575 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:12.572+0000 7fe1bd98d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe1b8070430 0x7fe1b80fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:10:12.575 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:12.572+0000 7fe1bd98d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe1b8070430 0x7fe1b80fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37006/0 (socket says 172.21.15.29:37006) 2023-12-01T22:10:12.575 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:12.572+0000 7fe1bd98d700 1 -- 172.21.15.29:0/1621567622 learned_addr learned my addr 172.21.15.29:0/1621567622 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-12-01T22:10:12.575 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:12.572+0000 7fe1bd98d700 1 -- 172.21.15.29:0/1621567622 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1b80fef10 con 0x7fe1b8070430 2023-12-01T22:10:12.575 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:12.572+0000 7fe1bd98d700 1 --2- 172.21.15.29:0/1621567622 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe1b8070430 0x7fe1b80fe8e0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7fe1b4009960 tx=0x7fe1b40092d0).ready entity=mon.0 client_cookie=477a5bc9c657f4a9 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:10:12.575 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:12.573+0000 7fe1bc98b700 1 -- 172.21.15.29:0/1621567622 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe1b400b9c0 con 0x7fe1b8070430 2023-12-01T22:10:12.577 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:12.573+0000 7fe1bc98b700 1 -- 172.21.15.29:0/1621567622 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1b400bb20 con 0x7fe1b8070430 2023-12-01T22:10:12.577 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:12.573+0000 7fe1bc98b700 1 -- 172.21.15.29:0/1621567622 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe1b400bdf0 con 0x7fe1b8070430 2023-12-01T22:10:12.577 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:12.573+0000 7fe1bfbf1700 1 -- 172.21.15.29:0/1621567622 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe1b8070430 msgr2=0x7fe1b80fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:10:12.577 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:12.573+0000 7fe1bfbf1700 1 --2- 172.21.15.29:0/1621567622 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe1b8070430 0x7fe1b80fe8e0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7fe1b4009960 tx=0x7fe1b40092d0).stop 2023-12-01T22:10:12.577 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:12.573+0000 7fe1bfbf1700 1 -- 172.21.15.29:0/1621567622 shutdown_connections 2023-12-01T22:10:12.577 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:12.573+0000 7fe1bfbf1700 1 --2- 172.21.15.29:0/1621567622 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe1b8070430 0x7fe1b80fe8e0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:10:12.578 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:12.573+0000 7fe1bfbf1700 1 -- 172.21.15.29:0/1621567622 >> 172.21.15.29:0/1621567622 conn(0x7fe1b80f63c0 msgr2=0x7fe1b80f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:10:12.578 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:12.573+0000 7fe1bfbf1700 1 -- 172.21.15.29:0/1621567622 shutdown_connections 2023-12-01T22:10:12.578 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:12.573+0000 7fe1bfbf1700 1 -- 172.21.15.29:0/1621567622 wait complete. 2023-12-01T22:10:12.578 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:12.574+0000 7fe1bfbf1700 1 Processor -- start 2023-12-01T22:10:12.578 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:12.574+0000 7fe1bfbf1700 1 -- start start 2023-12-01T22:10:12.578 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:12.574+0000 7fe1bfbf1700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe1b8070430 0x7fe1b80fc360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:10:12.578 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:12.574+0000 7fe1bfbf1700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1b80fa7e0 con 0x7fe1b8070430 2023-12-01T22:10:12.578 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:12.574+0000 7fe1bd98d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe1b8070430 0x7fe1b80fc360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:10:12.579 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:12.574+0000 7fe1bd98d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe1b8070430 0x7fe1b80fc360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37020/0 (socket says 172.21.15.29:37020) 2023-12-01T22:10:12.579 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:12.574+0000 7fe1bd98d700 1 -- 172.21.15.29:0/4232976334 learned_addr learned my addr 172.21.15.29:0/4232976334 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-12-01T22:10:12.579 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:12.575+0000 7fe1bd98d700 1 -- 172.21.15.29:0/4232976334 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1b4009570 con 0x7fe1b8070430 2023-12-01T22:10:12.579 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:12.575+0000 7fe1bd98d700 1 --2- 172.21.15.29:0/4232976334 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe1b8070430 0x7fe1b80fc360 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7fe1b4009f20 tx=0x7fe1b4017000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:10:12.579 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:12.575+0000 7fe1a67fc700 1 -- 172.21.15.29:0/4232976334 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe1b401c420 con 0x7fe1b8070430 2023-12-01T22:10:12.579 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:12.575+0000 7fe1a67fc700 1 -- 172.21.15.29:0/4232976334 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1b401c580 con 0x7fe1b8070430 2023-12-01T22:10:12.580 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:12.575+0000 7fe1bfbf1700 1 -- 172.21.15.29:0/4232976334 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1b80fa980 con 0x7fe1b8070430 2023-12-01T22:10:12.580 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:12.575+0000 7fe1a67fc700 1 -- 172.21.15.29:0/4232976334 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe1b401c850 con 0x7fe1b8070430 2023-12-01T22:10:12.580 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:12.575+0000 7fe1bfbf1700 1 -- 172.21.15.29:0/4232976334 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1b80fad90 con 0x7fe1b8070430 2023-12-01T22:10:12.580 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:12.575+0000 7fe1a67fc700 1 -- 172.21.15.29:0/4232976334 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7fe1b402c070 con 0x7fe1b8070430 2023-12-01T22:10:12.580 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:12.575+0000 7fe1a67fc700 1 -- 172.21.15.29:0/4232976334 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fe1b40256f0 con 0x7fe1b8070430 2023-12-01T22:10:12.580 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:12.576+0000 7fe1bfbf1700 1 -- 172.21.15.29:0/4232976334 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1b8059100 con 0x7fe1b8070430 2023-12-01T22:10:12.581 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:12.579+0000 7fe1a67fc700 1 -- 172.21.15.29:0/4232976334 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7fe1b402a060 con 0x7fe1b8070430 2023-12-01T22:10:12.654 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:12.653+0000 7fe1bfbf1700 1 -- 172.21.15.29:0/4232976334 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7fe1b80ff160 con 0x7fe1b8070430 2023-12-01T22:10:12.655 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:12.653+0000 7fe1a67fc700 1 -- 172.21.15.29:0/4232976334 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+324 (secure 0 0 0) 0x7fe1b4020070 con 0x7fe1b8070430 2023-12-01T22:10:12.655 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: cluster: 2023-12-01T22:10:12.656 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: id: 5558ab6c-9096-11ee-95a2-87774f69a715 2023-12-01T22:10:12.656 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: health: HEALTH_OK 2023-12-01T22:10:12.656 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:12.656 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: services: 2023-12-01T22:10:12.657 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: mon: 1 daemons, quorum smithi029 (age 0.51264s) 2023-12-01T22:10:12.657 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: mgr: no daemons active 2023-12-01T22:10:12.657 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2023-12-01T22:10:12.657 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:12.657 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: data: 2023-12-01T22:10:12.657 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2023-12-01T22:10:12.657 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2023-12-01T22:10:12.657 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2023-12-01T22:10:12.657 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: pgs: 2023-12-01T22:10:12.657 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:12.658 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:12.654+0000 7fe1bfbf1700 1 -- 172.21.15.29:0/4232976334 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe1b8070430 msgr2=0x7fe1b80fc360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:10:12.658 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:12.654+0000 7fe1bfbf1700 1 --2- 172.21.15.29:0/4232976334 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe1b8070430 0x7fe1b80fc360 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7fe1b4009f20 tx=0x7fe1b4017000).stop 2023-12-01T22:10:12.658 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:12.654+0000 7fe1bfbf1700 1 -- 172.21.15.29:0/4232976334 shutdown_connections 2023-12-01T22:10:12.658 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:12.654+0000 7fe1bfbf1700 1 --2- 172.21.15.29:0/4232976334 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe1b8070430 0x7fe1b80fc360 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:10:12.658 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:12.654+0000 7fe1bfbf1700 1 -- 172.21.15.29:0/4232976334 >> 172.21.15.29:0/4232976334 conn(0x7fe1b80f63c0 msgr2=0x7fe1b80f8070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:10:12.658 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:12.654+0000 7fe1bfbf1700 1 -- 172.21.15.29:0/4232976334 shutdown_connections 2023-12-01T22:10:12.658 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:12.654+0000 7fe1bfbf1700 1 -- 172.21.15.29:0/4232976334 wait complete. 2023-12-01T22:10:13.010 INFO:teuthology.orchestra.run.smithi029.stderr:mon is available 2023-12-01T22:10:13.011 INFO:teuthology.orchestra.run.smithi029.stderr:Assimilating anything we can from ceph.conf... 2023-12-01T22:10:13.011 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi029 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/5558ab6c-9096-11ee-95a2-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpou9lnrt4:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpxe9edifa:/etc/ceph/ceph.conf:z -v /var/lib/ceph/5558ab6c-9096-11ee-95a2-87774f69a715/mon.smithi029:/var/lib/ceph/mon/ceph-smithi029:z docker.io/ceph/ceph:v16.2.4 config assimilate-conf -i /var/lib/ceph/mon/ceph-smithi029/config 2023-12-01T22:10:13.217 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:13 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111363]: cluster 2023-12-01T22:10:12.141645+0000 mon.smithi029 (mon.0) 2 : cluster [INF] mon.smithi029 is new leader, mons smithi029 in quorum (ranks 0) 2023-12-01T22:10:13.217 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:13 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111363]: cluster 2023-12-01T22:10:12.141888+0000 mon.smithi029 (mon.0) 3 : cluster [DBG] monmap e1: 1 mons at {smithi029=[v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0]} 2023-12-01T22:10:13.217 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:13 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111363]: cluster 2023-12-01T22:10:12.143964+0000 mon.smithi029 (mon.0) 4 : cluster [DBG] fsmap 2023-12-01T22:10:13.217 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:13 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111363]: cluster 2023-12-01T22:10:12.146129+0000 mon.smithi029 (mon.0) 5 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2023-12-01T22:10:13.217 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:13 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111363]: cluster 2023-12-01T22:10:12.146566+0000 mon.smithi029 (mon.0) 6 : cluster [DBG] mgrmap e1: no daemons active 2023-12-01T22:10:13.217 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:13 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111363]: audit 2023-12-01T22:10:12.654250+0000 mon.smithi029 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.29:0/4232976334' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2023-12-01T22:10:13.443 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:13.441+0000 7f40b508a700 1 Processor -- start 2023-12-01T22:10:13.443 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:13.441+0000 7f40b508a700 1 -- start start 2023-12-01T22:10:13.443 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:13.441+0000 7f40b508a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40b0070390 0x7f40b0100da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-01T22:10:13.443 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:13.441+0000 7f40b508a700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40b00fc790 con 0x7f40b0070390 2023-12-01T22:10:13.443 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:13.441+0000 7f40aed9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40b0070390 0x7f40b0100da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:10:13.444 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:13.441+0000 7f40aed9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40b0070390 0x7f40b0100da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37036/0 (socket says 172.21.15.29:37036) 2023-12-01T22:10:13.446 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:13.441+0000 7f40aed9d700 1 -- 172.21.15.29:0/895377512 learned_addr learned my addr 172.21.15.29:0/895377512 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-12-01T22:10:13.446 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:13.442+0000 7f40aed9d700 1 -- 172.21.15.29:0/895377512 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40b0101290 con 0x7f40b0070390 2023-12-01T22:10:13.446 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:13.442+0000 7f40aed9d700 1 --2- 172.21.15.29:0/895377512 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40b0070390 0x7f40b0100da0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7f40a000b5f0 tx=0x7f40a0006ee0).ready entity=mon.0 client_cookie=66dd2d616a29b471 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:10:13.446 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:13.442+0000 7f40add9b700 1 -- 172.21.15.29:0/895377512 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f40a000d600 con 0x7f40b0070390 2023-12-01T22:10:13.446 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:13.442+0000 7f40add9b700 1 -- 172.21.15.29:0/895377512 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40a000d760 con 0x7f40b0070390 2023-12-01T22:10:13.446 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:13.442+0000 7f40add9b700 1 -- 172.21.15.29:0/895377512 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f40a000da30 con 0x7f40b0070390 2023-12-01T22:10:13.446 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:13.442+0000 7f40b508a700 1 -- 172.21.15.29:0/895377512 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40b0070390 msgr2=0x7f40b0100da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:10:13.446 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:13.442+0000 7f40b508a700 1 --2- 172.21.15.29:0/895377512 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40b0070390 0x7f40b0100da0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7f40a000b5f0 tx=0x7f40a0006ee0).stop 2023-12-01T22:10:13.447 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:13.443+0000 7f40b508a700 1 -- 172.21.15.29:0/895377512 shutdown_connections 2023-12-01T22:10:13.447 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:13.443+0000 7f40b508a700 1 --2- 172.21.15.29:0/895377512 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40b0070390 0x7f40b0100da0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:10:13.447 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:13.443+0000 7f40b508a700 1 -- 172.21.15.29:0/895377512 >> 172.21.15.29:0/895377512 conn(0x7f40b00f63c0 msgr2=0x7f40b00f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:10:13.447 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:13.443+0000 7f40b508a700 1 -- 172.21.15.29:0/895377512 shutdown_connections 2023-12-01T22:10:13.447 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:13.443+0000 7f40b508a700 1 -- 172.21.15.29:0/895377512 wait complete. 2023-12-01T22:10:13.447 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:13.443+0000 7f40b508a700 1 Processor -- start 2023-12-01T22:10:13.447 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:13.444+0000 7f40b508a700 1 -- start start 2023-12-01T22:10:13.447 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:13.444+0000 7f40b508a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40b0070390 0x7f40b010c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:10:13.447 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:13.444+0000 7f40b508a700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40b010c9c0 con 0x7f40b0070390 2023-12-01T22:10:13.447 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:13.444+0000 7f40aed9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40b0070390 0x7f40b010c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:10:13.447 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:13.444+0000 7f40aed9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40b0070390 0x7f40b010c4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37042/0 (socket says 172.21.15.29:37042) 2023-12-01T22:10:13.447 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:13.444+0000 7f40aed9d700 1 -- 172.21.15.29:0/1956185124 learned_addr learned my addr 172.21.15.29:0/1956185124 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-12-01T22:10:13.448 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:13.444+0000 7f40aed9d700 1 -- 172.21.15.29:0/1956185124 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40a000b200 con 0x7f40b0070390 2023-12-01T22:10:13.448 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:13.444+0000 7f40aed9d700 1 --2- 172.21.15.29:0/1956185124 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40b0070390 0x7f40b010c4d0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7f40a0006c70 tx=0x7f40a0016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:10:13.448 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:13.445+0000 7f40977fe700 1 -- 172.21.15.29:0/1956185124 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f40a0011b40 con 0x7f40b0070390 2023-12-01T22:10:13.448 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:13.445+0000 7f40977fe700 1 -- 172.21.15.29:0/1956185124 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40a0011ca0 con 0x7f40b0070390 2023-12-01T22:10:13.448 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:13.445+0000 7f40b508a700 1 -- 172.21.15.29:0/1956185124 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40b010cbc0 con 0x7f40b0070390 2023-12-01T22:10:13.448 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:13.445+0000 7f40977fe700 1 -- 172.21.15.29:0/1956185124 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f40a001d430 con 0x7f40b0070390 2023-12-01T22:10:13.448 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:13.445+0000 7f40b508a700 1 -- 172.21.15.29:0/1956185124 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40b010cfd0 con 0x7f40b0070390 2023-12-01T22:10:13.448 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:13.445+0000 7f40977fe700 1 -- 172.21.15.29:0/1956185124 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f40a000f070 con 0x7f40b0070390 2023-12-01T22:10:13.448 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:13.445+0000 7f40977fe700 1 -- 172.21.15.29:0/1956185124 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f40a0027460 con 0x7f40b0070390 2023-12-01T22:10:13.449 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:13.446+0000 7f40b508a700 1 -- 172.21.15.29:0/1956185124 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40b0045ad0 con 0x7f40b0070390 2023-12-01T22:10:13.450 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:13.449+0000 7f40977fe700 1 -- 172.21.15.29:0/1956185124 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f40a0016070 con 0x7f40b0070390 2023-12-01T22:10:13.520 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:13.519+0000 7f40b508a700 1 -- 172.21.15.29:0/1956185124 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f40b0101420 con 0x7f40b0070390 2023-12-01T22:10:13.526 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:13.522+0000 7f40977fe700 1 -- 172.21.15.29:0/1956185124 <== mon.0 v2:172.21.15.29:3300/0 7 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f40a001d5e0 con 0x7f40b0070390 2023-12-01T22:10:13.526 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:13.522+0000 7f40977fe700 1 -- 172.21.15.29:0/1956185124 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+464 (secure 0 0 0) 0x7f40a0036020 con 0x7f40b0070390 2023-12-01T22:10:13.527 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:13.527 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: [global] 2023-12-01T22:10:13.527 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: fsid = 5558ab6c-9096-11ee-95a2-87774f69a715 2023-12-01T22:10:13.527 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.29:3300,v1:172.21.15.29:6789] 2023-12-01T22:10:13.527 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2023-12-01T22:10:13.527 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2023-12-01T22:10:13.527 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2023-12-01T22:10:13.527 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2023-12-01T22:10:13.527 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:13.527 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: [mgr] 2023-12-01T22:10:13.527 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2023-12-01T22:10:13.527 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:13.527 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: [osd] 2023-12-01T22:10:13.527 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: mon_osd_backfillfull_ratio = 0.85 2023-12-01T22:10:13.527 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: mon_osd_full_ratio = 0.9 2023-12-01T22:10:13.528 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: mon_osd_nearfull_ratio = 0.8 2023-12-01T22:10:13.528 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2023-12-01T22:10:13.528 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: osd_memory_target_autotune = true 2023-12-01T22:10:13.528 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2023-12-01T22:10:13.528 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:13.523+0000 7f40b508a700 1 -- 172.21.15.29:0/1956185124 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40b0070390 msgr2=0x7f40b010c4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:10:13.528 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:13.523+0000 7f40b508a700 1 --2- 172.21.15.29:0/1956185124 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40b0070390 0x7f40b010c4d0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7f40a0006c70 tx=0x7f40a0016000).stop 2023-12-01T22:10:13.528 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:13.523+0000 7f40b508a700 1 -- 172.21.15.29:0/1956185124 shutdown_connections 2023-12-01T22:10:13.528 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:13.523+0000 7f40b508a700 1 --2- 172.21.15.29:0/1956185124 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40b0070390 0x7f40b010c4d0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:10:13.528 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:13.523+0000 7f40b508a700 1 -- 172.21.15.29:0/1956185124 >> 172.21.15.29:0/1956185124 conn(0x7f40b00f63c0 msgr2=0x7f40b00f8110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:10:13.528 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:13.524+0000 7f40b508a700 1 -- 172.21.15.29:0/1956185124 shutdown_connections 2023-12-01T22:10:13.528 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:13.524+0000 7f40b508a700 1 -- 172.21.15.29:0/1956185124 wait complete. 2023-12-01T22:10:13.877 INFO:teuthology.orchestra.run.smithi029.stderr:Generating new minimal ceph.conf... 2023-12-01T22:10:13.877 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi029 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/5558ab6c-9096-11ee-95a2-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpou9lnrt4:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpxe9edifa:/etc/ceph/ceph.conf:z -v /var/lib/ceph/5558ab6c-9096-11ee-95a2-87774f69a715/mon.smithi029:/var/lib/ceph/mon/ceph-smithi029:z docker.io/ceph/ceph:v16.2.4 config generate-minimal-conf -o /var/lib/ceph/mon/ceph-smithi029/config 2023-12-01T22:10:14.262 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:14.259+0000 7f65944f7700 1 Processor -- start 2023-12-01T22:10:14.262 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:14.259+0000 7f65944f7700 1 -- start start 2023-12-01T22:10:14.263 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:14.260+0000 7f65944f7700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f658c070460 0x7f658c0fe930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-01T22:10:14.263 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:14.260+0000 7f65944f7700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f658c0fee20 con 0x7f658c070460 2023-12-01T22:10:14.263 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:14.260+0000 7f6592293700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f658c070460 0x7f658c0fe930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:10:14.263 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:14.260+0000 7f6592293700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f658c070460 0x7f658c0fe930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37044/0 (socket says 172.21.15.29:37044) 2023-12-01T22:10:14.263 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:14.260+0000 7f6592293700 1 -- 172.21.15.29:0/1538379892 learned_addr learned my addr 172.21.15.29:0/1538379892 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-12-01T22:10:14.263 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:14.260+0000 7f6592293700 1 -- 172.21.15.29:0/1538379892 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f658c0fef60 con 0x7f658c070460 2023-12-01T22:10:14.263 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:14.260+0000 7f6592293700 1 --2- 172.21.15.29:0/1538379892 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f658c070460 0x7f658c0fe930 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f658800b5f0 tx=0x7f6588006ee0).ready entity=mon.0 client_cookie=9261be15d488960e server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:10:14.263 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:14.261+0000 7f6591291700 1 -- 172.21.15.29:0/1538379892 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f658800d600 con 0x7f658c070460 2023-12-01T22:10:14.263 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:14.261+0000 7f6591291700 1 -- 172.21.15.29:0/1538379892 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f658800dbd0 con 0x7f658c070460 2023-12-01T22:10:14.263 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:14.261+0000 7f6591291700 1 -- 172.21.15.29:0/1538379892 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6588011c00 con 0x7f658c070460 2023-12-01T22:10:14.265 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:14.261+0000 7f65944f7700 1 -- 172.21.15.29:0/1538379892 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f658c070460 msgr2=0x7f658c0fe930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:10:14.265 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:14.261+0000 7f65944f7700 1 --2- 172.21.15.29:0/1538379892 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f658c070460 0x7f658c0fe930 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f658800b5f0 tx=0x7f6588006ee0).stop 2023-12-01T22:10:14.265 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:14.261+0000 7f65944f7700 1 -- 172.21.15.29:0/1538379892 shutdown_connections 2023-12-01T22:10:14.265 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:14.261+0000 7f65944f7700 1 --2- 172.21.15.29:0/1538379892 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f658c070460 0x7f658c0fe930 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:10:14.265 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:14.261+0000 7f65944f7700 1 -- 172.21.15.29:0/1538379892 >> 172.21.15.29:0/1538379892 conn(0x7f658c0f63b0 msgr2=0x7f658c0f87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:10:14.265 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:14.261+0000 7f65944f7700 1 -- 172.21.15.29:0/1538379892 shutdown_connections 2023-12-01T22:10:14.265 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:14.261+0000 7f65944f7700 1 -- 172.21.15.29:0/1538379892 wait complete. 2023-12-01T22:10:14.265 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:14.262+0000 7f65944f7700 1 Processor -- start 2023-12-01T22:10:14.265 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:14.262+0000 7f65944f7700 1 -- start start 2023-12-01T22:10:14.265 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:14.262+0000 7f65944f7700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f658c070460 0x7f658c0fc3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:10:14.265 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:14.262+0000 7f65944f7700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f658c0fa830 con 0x7f658c070460 2023-12-01T22:10:14.265 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:14.262+0000 7f6592293700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f658c070460 0x7f658c0fc3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:10:14.266 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:14.262+0000 7f6592293700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f658c070460 0x7f658c0fc3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37050/0 (socket says 172.21.15.29:37050) 2023-12-01T22:10:14.266 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:14.262+0000 7f6592293700 1 -- 172.21.15.29:0/900629925 learned_addr learned my addr 172.21.15.29:0/900629925 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-12-01T22:10:14.266 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:14.263+0000 7f6592293700 1 -- 172.21.15.29:0/900629925 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f658800b200 con 0x7f658c070460 2023-12-01T22:10:14.266 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:14.263+0000 7f6592293700 1 --2- 172.21.15.29:0/900629925 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f658c070460 0x7f658c0fc3b0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f658800e040 tx=0x7f6588006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:10:14.266 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:14.263+0000 7f657effd700 1 -- 172.21.15.29:0/900629925 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f658800d9f0 con 0x7f658c070460 2023-12-01T22:10:14.266 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:14.263+0000 7f657effd700 1 -- 172.21.15.29:0/900629925 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f658801a9f0 con 0x7f658c070460 2023-12-01T22:10:14.266 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:14.263+0000 7f657effd700 1 -- 172.21.15.29:0/900629925 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f658801b8c0 con 0x7f658c070460 2023-12-01T22:10:14.266 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:14.263+0000 7f65944f7700 1 -- 172.21.15.29:0/900629925 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f658c0fa9d0 con 0x7f658c070460 2023-12-01T22:10:14.266 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:14.263+0000 7f65944f7700 1 -- 172.21.15.29:0/900629925 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f658c0fade0 con 0x7f658c070460 2023-12-01T22:10:14.267 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:14.263+0000 7f657effd700 1 -- 172.21.15.29:0/900629925 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f658800f070 con 0x7f658c070460 2023-12-01T22:10:14.267 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:14.263+0000 7f657effd700 1 -- 172.21.15.29:0/900629925 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f6588025ae0 con 0x7f658c070460 2023-12-01T22:10:14.267 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:14.264+0000 7f65944f7700 1 -- 172.21.15.29:0/900629925 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f658c045ad0 con 0x7f658c070460 2023-12-01T22:10:14.268 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:14.267+0000 7f657effd700 1 -- 172.21.15.29:0/900629925 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f6588016070 con 0x7f658c070460 2023-12-01T22:10:14.339 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:14.337+0000 7f65944f7700 1 -- 172.21.15.29:0/900629925 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f658c0ffbe0 con 0x7f658c070460 2023-12-01T22:10:14.339 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:14.338+0000 7f657effd700 1 -- 172.21.15.29:0/900629925 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7f658c0ffbe0 con 0x7f658c070460 2023-12-01T22:10:14.340 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:14.338+0000 7f65944f7700 1 -- 172.21.15.29:0/900629925 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f658c070460 msgr2=0x7f658c0fc3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:10:14.341 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:14.338+0000 7f65944f7700 1 --2- 172.21.15.29:0/900629925 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f658c070460 0x7f658c0fc3b0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f658800e040 tx=0x7f6588006ee0).stop 2023-12-01T22:10:14.341 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:14.339+0000 7f65944f7700 1 -- 172.21.15.29:0/900629925 shutdown_connections 2023-12-01T22:10:14.341 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:14.339+0000 7f65944f7700 1 --2- 172.21.15.29:0/900629925 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f658c070460 0x7f658c0fc3b0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:10:14.341 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:14.339+0000 7f65944f7700 1 -- 172.21.15.29:0/900629925 >> 172.21.15.29:0/900629925 conn(0x7f658c0f63b0 msgr2=0x7f658c0f7ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:10:14.341 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:14.339+0000 7f65944f7700 1 -- 172.21.15.29:0/900629925 shutdown_connections 2023-12-01T22:10:14.341 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:14.339+0000 7f65944f7700 1 -- 172.21.15.29:0/900629925 wait complete. 2023-12-01T22:10:14.500 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:14 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111363]: audit 2023-12-01T22:10:13.520545+0000 mon.smithi029 (mon.0) 8 : audit [INF] from='client.? 172.21.15.29:0/1956185124' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-12-01T22:10:14.500 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:14 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111363]: audit 2023-12-01T22:10:13.523861+0000 mon.smithi029 (mon.0) 9 : audit [INF] from='client.? 172.21.15.29:0/1956185124' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2023-12-01T22:10:14.694 INFO:teuthology.orchestra.run.smithi029.stderr:Restarting the monitor... 2023-12-01T22:10:14.694 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: systemctl restart ceph-5558ab6c-9096-11ee-95a2-87774f69a715@mon.smithi029 2023-12-01T22:10:14.776 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:14 smithi029 systemd[1]: Stopping Ceph mon.smithi029 for 5558ab6c-9096-11ee-95a2-87774f69a715... 2023-12-01T22:10:15.109 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:14 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111363]: debug 2023-12-01T22:10:14.783+0000 7f83b6b87700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi029 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2023-12-01T22:10:15.109 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:14 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111363]: debug 2023-12-01T22:10:14.783+0000 7f83b6b87700 -1 mon.smithi029@0(leader) e1 *** Got Signal Terminated *** 2023-12-01T22:10:15.109 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:14 smithi029 podman[111686]: ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029 2023-12-01T22:10:15.405 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 systemd[1]: ceph-5558ab6c-9096-11ee-95a2-87774f69a715@mon.smithi029.service: Succeeded. 2023-12-01T22:10:15.405 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 systemd[1]: Stopped Ceph mon.smithi029 for 5558ab6c-9096-11ee-95a2-87774f69a715. 2023-12-01T22:10:15.406 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 systemd[1]: Starting Ceph mon.smithi029 for 5558ab6c-9096-11ee-95a2-87774f69a715... 2023-12-01T22:10:15.411 INFO:teuthology.orchestra.run.smithi029.stderr:Setting mon public_network to 172.21.0.0/20 2023-12-01T22:10:15.412 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi029 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/5558ab6c-9096-11ee-95a2-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpou9lnrt4:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpxe9edifa:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mon public_network 172.21.0.0/20 2023-12-01T22:10:15.778 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.403+0000 7f66fa451700 0 set uid:gid to 167:167 (ceph:ceph) 2023-12-01T22:10:15.778 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.403+0000 7f66fa451700 0 ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable), process ceph-mon, pid 7 2023-12-01T22:10:15.779 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.403+0000 7f66fa451700 0 pidfile_write: ignore empty --pid-file 2023-12-01T22:10:15.779 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 bash[111761]: cce6a4c2fc424f8023de6d3bf779f2c56a52516a3d7ea81ef94548d2b7ebb947 2023-12-01T22:10:15.779 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 systemd[1]: Started Ceph mon.smithi029 for 5558ab6c-9096-11ee-95a2-87774f69a715. 2023-12-01T22:10:15.779 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.408+0000 7f66fa451700 0 load: jerasure load: lrc load: isa 2023-12-01T22:10:15.779 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: RocksDB version: 6.8.1 2023-12-01T22:10:15.779 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: 2023-12-01T22:10:15.779 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2023-12-01T22:10:15.779 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Compile date May 13 2021 2023-12-01T22:10:15.779 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: DB SUMMARY 2023-12-01T22:10:15.779 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: 2023-12-01T22:10:15.779 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: CURRENT file: CURRENT 2023-12-01T22:10:15.779 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: 2023-12-01T22:10:15.779 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: IDENTITY file: IDENTITY 2023-12-01T22:10:15.779 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: 2023-12-01T22:10:15.779 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: MANIFEST file: MANIFEST-000005 size: 131 Bytes 2023-12-01T22:10:15.780 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: 2023-12-01T22:10:15.780 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi029/store.db dir, Total Num: 1, files: 000004.sst 2023-12-01T22:10:15.780 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: 2023-12-01T22:10:15.780 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi029/store.db: 000006.log size: 92918 ; 2023-12-01T22:10:15.780 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: 2023-12-01T22:10:15.780 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.error_if_exists: 0 2023-12-01T22:10:15.780 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.create_if_missing: 0 2023-12-01T22:10:15.780 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.paranoid_checks: 1 2023-12-01T22:10:15.780 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.env: 0x561d331931c0 2023-12-01T22:10:15.780 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.fs: Posix File System 2023-12-01T22:10:15.780 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.info_log: 0x561d354a5960 2023-12-01T22:10:15.780 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.max_file_opening_threads: 16 2023-12-01T22:10:15.780 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.statistics: (nil) 2023-12-01T22:10:15.781 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.use_fsync: 0 2023-12-01T22:10:15.781 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.max_log_file_size: 0 2023-12-01T22:10:15.781 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2023-12-01T22:10:15.781 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.log_file_time_to_roll: 0 2023-12-01T22:10:15.781 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.keep_log_file_num: 1000 2023-12-01T22:10:15.781 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.recycle_log_file_num: 0 2023-12-01T22:10:15.781 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.allow_fallocate: 1 2023-12-01T22:10:15.781 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.allow_mmap_reads: 0 2023-12-01T22:10:15.781 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.allow_mmap_writes: 0 2023-12-01T22:10:15.781 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.use_direct_reads: 0 2023-12-01T22:10:15.781 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-12-01T22:10:15.781 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.create_missing_column_families: 0 2023-12-01T22:10:15.781 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.db_log_dir: 2023-12-01T22:10:15.781 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi029/store.db 2023-12-01T22:10:15.781 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.table_cache_numshardbits: 6 2023-12-01T22:10:15.782 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.max_subcompactions: 1 2023-12-01T22:10:15.782 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.max_background_flushes: -1 2023-12-01T22:10:15.782 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.WAL_ttl_seconds: 0 2023-12-01T22:10:15.782 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.WAL_size_limit_MB: 0 2023-12-01T22:10:15.782 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-12-01T22:10:15.782 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2023-12-01T22:10:15.782 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.is_fd_close_on_exec: 1 2023-12-01T22:10:15.782 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.advise_random_on_open: 1 2023-12-01T22:10:15.782 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.db_write_buffer_size: 0 2023-12-01T22:10:15.782 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.write_buffer_manager: 0x561d3611a030 2023-12-01T22:10:15.782 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2023-12-01T22:10:15.782 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2023-12-01T22:10:15.782 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2023-12-01T22:10:15.782 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.use_adaptive_mutex: 0 2023-12-01T22:10:15.783 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.rate_limiter: (nil) 2023-12-01T22:10:15.783 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-12-01T22:10:15.783 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.wal_recovery_mode: 2 2023-12-01T22:10:15.783 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.enable_thread_tracking: 0 2023-12-01T22:10:15.783 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.enable_pipelined_write: 0 2023-12-01T22:10:15.783 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.unordered_write: 0 2023-12-01T22:10:15.783 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2023-12-01T22:10:15.783 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-12-01T22:10:15.783 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2023-12-01T22:10:15.783 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2023-12-01T22:10:15.783 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.row_cache: None 2023-12-01T22:10:15.783 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.wal_filter: None 2023-12-01T22:10:15.783 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2023-12-01T22:10:15.783 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.allow_ingest_behind: 0 2023-12-01T22:10:15.783 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.preserve_deletes: 0 2023-12-01T22:10:15.784 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.two_write_queues: 0 2023-12-01T22:10:15.784 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.manual_wal_flush: 0 2023-12-01T22:10:15.784 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.atomic_flush: 0 2023-12-01T22:10:15.784 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-12-01T22:10:15.784 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.persist_stats_to_disk: 0 2023-12-01T22:10:15.784 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.write_dbid_to_manifest: 0 2023-12-01T22:10:15.784 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.log_readahead_size: 0 2023-12-01T22:10:15.784 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2023-12-01T22:10:15.784 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.max_background_jobs: 2 2023-12-01T22:10:15.784 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.max_background_compactions: -1 2023-12-01T22:10:15.784 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2023-12-01T22:10:15.785 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-12-01T22:10:15.785 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.delayed_write_rate : 16777216 2023-12-01T22:10:15.785 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.max_total_wal_size: 0 2023-12-01T22:10:15.785 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-12-01T22:10:15.785 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.stats_dump_period_sec: 600 2023-12-01T22:10:15.785 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.stats_persist_period_sec: 600 2023-12-01T22:10:15.785 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2023-12-01T22:10:15.785 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.max_open_files: -1 2023-12-01T22:10:15.786 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.bytes_per_sync: 0 2023-12-01T22:10:15.786 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.wal_bytes_per_sync: 0 2023-12-01T22:10:15.786 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.strict_bytes_per_sync: 0 2023-12-01T22:10:15.786 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.compaction_readahead_size: 0 2023-12-01T22:10:15.786 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Compression algorithms supported: 2023-12-01T22:10:15.786 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2023-12-01T22:10:15.786 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: kZSTD supported: 0 2023-12-01T22:10:15.786 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: kXpressCompression supported: 0 2023-12-01T22:10:15.786 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: kLZ4HCCompression supported: 1 2023-12-01T22:10:15.787 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: kLZ4Compression supported: 1 2023-12-01T22:10:15.788 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: kBZip2Compression supported: 0 2023-12-01T22:10:15.788 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: kZlibCompression supported: 1 2023-12-01T22:10:15.788 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: kSnappyCompression supported: 1 2023-12-01T22:10:15.788 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Fast CRC32 supported: Supported on x86 2023-12-01T22:10:15.788 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi029/store.db/MANIFEST-000005 2023-12-01T22:10:15.788 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: 2023-12-01T22:10:15.788 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2023-12-01T22:10:15.788 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: 2023-12-01T22:10:15.788 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-12-01T22:10:15.788 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.merge_operator: 2023-12-01T22:10:15.788 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.compaction_filter: None 2023-12-01T22:10:15.788 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.compaction_filter_factory: None 2023-12-01T22:10:15.788 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.memtable_factory: SkipListFactory 2023-12-01T22:10:15.788 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.table_factory: BlockBasedTable 2023-12-01T22:10:15.789 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x561d353bc128) 2023-12-01T22:10:15.789 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cache_index_and_filter_blocks: 1 2023-12-01T22:10:15.789 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cache_index_and_filter_blocks_with_high_priority: 0 2023-12-01T22:10:15.789 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: pin_l0_filter_and_index_blocks_in_cache: 0 2023-12-01T22:10:15.789 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: pin_top_level_index_and_filter: 1 2023-12-01T22:10:15.789 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: index_type: 0 2023-12-01T22:10:15.789 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: data_block_index_type: 0 2023-12-01T22:10:15.789 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: index_shortening: 1 2023-12-01T22:10:15.789 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: data_block_hash_table_util_ratio: 0.750000 2023-12-01T22:10:15.789 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: hash_index_allow_collision: 1 2023-12-01T22:10:15.789 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: checksum: 1 2023-12-01T22:10:15.789 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: no_block_cache: 0 2023-12-01T22:10:15.789 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: block_cache: 0x561d353f2f10 2023-12-01T22:10:15.789 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: block_cache_name: BinnedLRUCache 2023-12-01T22:10:15.789 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: block_cache_options: 2023-12-01T22:10:15.790 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: capacity : 536870912 2023-12-01T22:10:15.790 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: num_shard_bits : 4 2023-12-01T22:10:15.790 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: strict_capacity_limit : 0 2023-12-01T22:10:15.790 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: high_pri_pool_ratio: 0.000 2023-12-01T22:10:15.790 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: block_cache_compressed: (nil) 2023-12-01T22:10:15.790 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: persistent_cache: (nil) 2023-12-01T22:10:15.790 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: block_size: 4096 2023-12-01T22:10:15.790 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: block_size_deviation: 10 2023-12-01T22:10:15.790 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: block_restart_interval: 16 2023-12-01T22:10:15.790 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: index_block_restart_interval: 1 2023-12-01T22:10:15.790 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: metadata_block_size: 4096 2023-12-01T22:10:15.790 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: partition_filters: 0 2023-12-01T22:10:15.790 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: use_delta_encoding: 1 2023-12-01T22:10:15.791 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: filter_policy: rocksdb.BuiltinBloomFilter 2023-12-01T22:10:15.791 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: whole_key_filtering: 1 2023-12-01T22:10:15.791 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: verify_compression: 0 2023-12-01T22:10:15.791 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: read_amp_bytes_per_bit: 0 2023-12-01T22:10:15.791 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: format_version: 2 2023-12-01T22:10:15.791 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: enable_index_compression: 1 2023-12-01T22:10:15.791 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: block_align: 0 2023-12-01T22:10:15.791 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: 2023-12-01T22:10:15.791 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.write_buffer_size: 33554432 2023-12-01T22:10:15.791 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.max_write_buffer_number: 2 2023-12-01T22:10:15.791 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.compression: NoCompression 2023-12-01T22:10:15.791 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.bottommost_compression: Disabled 2023-12-01T22:10:15.791 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.prefix_extractor: nullptr 2023-12-01T22:10:15.791 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-12-01T22:10:15.791 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.num_levels: 7 2023-12-01T22:10:15.792 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-12-01T22:10:15.792 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-12-01T22:10:15.792 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-12-01T22:10:15.792 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-12-01T22:10:15.792 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2023-12-01T22:10:15.792 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-12-01T22:10:15.792 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-12-01T22:10:15.792 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-12-01T22:10:15.792 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2023-12-01T22:10:15.792 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.compression_opts.window_bits: -14 2023-12-01T22:10:15.792 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.compression_opts.level: 32767 2023-12-01T22:10:15.792 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.compression_opts.strategy: 0 2023-12-01T22:10:15.792 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-12-01T22:10:15.792 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-12-01T22:10:15.793 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.compression_opts.enabled: false 2023-12-01T22:10:15.793 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-12-01T22:10:15.793 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-12-01T22:10:15.793 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2023-12-01T22:10:15.793 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.target_file_size_base: 67108864 2023-12-01T22:10:15.793 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.target_file_size_multiplier: 1 2023-12-01T22:10:15.793 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2023-12-01T22:10:15.793 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-12-01T22:10:15.793 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-12-01T22:10:15.793 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-12-01T22:10:15.793 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-12-01T22:10:15.794 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-12-01T22:10:15.794 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-12-01T22:10:15.794 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-12-01T22:10:15.794 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-12-01T22:10:15.794 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-12-01T22:10:15.794 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-12-01T22:10:15.794 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2023-12-01T22:10:15.794 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.arena_block_size: 4194304 2023-12-01T22:10:15.794 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-12-01T22:10:15.794 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-12-01T22:10:15.794 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2023-12-01T22:10:15.794 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.disable_auto_compactions: 0 2023-12-01T22:10:15.794 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.409+0000 7f66fa451700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-12-01T22:10:15.795 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.410+0000 7f66fa451700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-12-01T22:10:15.795 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.410+0000 7f66fa451700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-12-01T22:10:15.795 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.410+0000 7f66fa451700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-12-01T22:10:15.795 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.410+0000 7f66fa451700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-12-01T22:10:15.795 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.410+0000 7f66fa451700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-12-01T22:10:15.795 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.410+0000 7f66fa451700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-12-01T22:10:15.795 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.410+0000 7f66fa451700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-12-01T22:10:15.795 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.410+0000 7f66fa451700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-12-01T22:10:15.795 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.410+0000 7f66fa451700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-12-01T22:10:15.795 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.410+0000 7f66fa451700 4 rocksdb: Options.table_properties_collectors: 2023-12-01T22:10:15.795 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.410+0000 7f66fa451700 4 rocksdb: Options.inplace_update_support: 0 2023-12-01T22:10:15.795 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.410+0000 7f66fa451700 4 rocksdb: Options.inplace_update_num_locks: 10000 2023-12-01T22:10:15.795 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.410+0000 7f66fa451700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-12-01T22:10:15.795 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.410+0000 7f66fa451700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2023-12-01T22:10:15.796 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.410+0000 7f66fa451700 4 rocksdb: Options.memtable_huge_page_size: 0 2023-12-01T22:10:15.796 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.410+0000 7f66fa451700 4 rocksdb: Options.bloom_locality: 0 2023-12-01T22:10:15.796 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.410+0000 7f66fa451700 4 rocksdb: Options.max_successive_merges: 0 2023-12-01T22:10:15.796 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.410+0000 7f66fa451700 4 rocksdb: Options.optimize_filters_for_hits: 0 2023-12-01T22:10:15.796 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.410+0000 7f66fa451700 4 rocksdb: Options.paranoid_file_checks: 0 2023-12-01T22:10:15.796 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.410+0000 7f66fa451700 4 rocksdb: Options.force_consistency_checks: 0 2023-12-01T22:10:15.796 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.410+0000 7f66fa451700 4 rocksdb: Options.report_bg_io_stats: 0 2023-12-01T22:10:15.796 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.410+0000 7f66fa451700 4 rocksdb: Options.ttl: 2592000 2023-12-01T22:10:15.796 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.410+0000 7f66fa451700 4 rocksdb: Options.periodic_compaction_seconds: 0 2023-12-01T22:10:15.796 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.410+0000 7f66fa451700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi029/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 5, log_number is 4,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-12-01T22:10:15.796 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: 2023-12-01T22:10:15.796 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.410+0000 7f66fa451700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 4 2023-12-01T22:10:15.796 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: 2023-12-01T22:10:15.797 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.410+0000 7f66fa451700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1701468615412035, "job": 1, "event": "recovery_started", "log_files": [6]} 2023-12-01T22:10:15.797 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.410+0000 7f66fa451700 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #6 mode 2 2023-12-01T22:10:15.797 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.411+0000 7f66fa451700 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2023-12-01T22:10:15.797 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.412+0000 7f66fa451700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1701468615413173, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 7, "file_size": 88698, "table_properties": {"data_size": 86813, "index_size": 362, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 709, "raw_key_size": 12384, "raw_average_key_size": 49, "raw_value_size": 79954, "raw_average_value_size": 321, "num_data_blocks": 11, "num_entries": 249, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1701468615, "oldest_key_time": 3, "file_creation_time": 0}} 2023-12-01T22:10:15.797 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.412+0000 7f66fa451700 4 rocksdb: [version_set.cc:3826] Creating manifest 8 2023-12-01T22:10:15.797 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: 2023-12-01T22:10:15.797 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.412+0000 7f66fa451700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1701468615413778, "job": 1, "event": "recovery_finished"} 2023-12-01T22:10:15.797 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.413+0000 7f66fa451700 4 rocksdb: DB pointer 0x561d354b7800 2023-12-01T22:10:15.797 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.413+0000 7f66e1269700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2023-12-01T22:10:15.797 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.413+0000 7f66e1269700 4 rocksdb: [db_impl/db_impl.cc:851] 2023-12-01T22:10:15.797 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: ** DB Stats ** 2023-12-01T22:10:15.797 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: Uptime(secs): 0.0 total, 0.0 interval 2023-12-01T22:10:15.798 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-12-01T22:10:15.798 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-12-01T22:10:15.798 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-12-01T22:10:15.798 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-12-01T22:10:15.798 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2023-12-01T22:10:15.799 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-12-01T22:10:15.799 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: 2023-12-01T22:10:15.799 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: ** Compaction Stats [default] ** 2023-12-01T22:10:15.799 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-12-01T22:10:15.799 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-12-01T22:10:15.799 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: L0 2/0 88.31 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 113.8 0.00 0.00 1 0.001 0 0 2023-12-01T22:10:15.799 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: Sum 2/0 88.31 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 113.8 0.00 0.00 1 0.001 0 0 2023-12-01T22:10:15.799 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 113.8 0.00 0.00 1 0.001 0 0 2023-12-01T22:10:15.799 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: 2023-12-01T22:10:15.799 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: ** Compaction Stats [default] ** 2023-12-01T22:10:15.799 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-12-01T22:10:15.799 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-12-01T22:10:15.799 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 113.8 0.00 0.00 1 0.001 0 0 2023-12-01T22:10:15.799 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: Uptime(secs): 0.0 total, 0.0 interval 2023-12-01T22:10:15.800 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: Flush(GB): cumulative 0.000, interval 0.000 2023-12-01T22:10:15.800 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: AddFile(GB): cumulative 0.000, interval 0.000 2023-12-01T22:10:15.800 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: AddFile(Total Files): cumulative 0, interval 0 2023-12-01T22:10:15.800 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: AddFile(L0 Files): cumulative 0, interval 0 2023-12-01T22:10:15.800 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: AddFile(Keys): cumulative 0, interval 0 2023-12-01T22:10:15.800 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: Cumulative compaction: 0.00 GB write, 22.33 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-12-01T22:10:15.800 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: Interval compaction: 0.00 GB write, 22.33 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-12-01T22:10:15.800 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-12-01T22:10:15.800 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: 2023-12-01T22:10:15.800 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: ** File Read Latency Histogram By Level [default] ** 2023-12-01T22:10:15.800 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: 2023-12-01T22:10:15.800 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: ** Compaction Stats [default] ** 2023-12-01T22:10:15.800 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-12-01T22:10:15.801 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-12-01T22:10:15.801 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: L0 2/0 88.31 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 113.8 0.00 0.00 1 0.001 0 0 2023-12-01T22:10:15.801 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: Sum 2/0 88.31 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 113.8 0.00 0.00 1 0.001 0 0 2023-12-01T22:10:15.801 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-12-01T22:10:15.801 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: 2023-12-01T22:10:15.801 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: ** Compaction Stats [default] ** 2023-12-01T22:10:15.801 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-12-01T22:10:15.801 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-12-01T22:10:15.801 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 113.8 0.00 0.00 1 0.001 0 0 2023-12-01T22:10:15.801 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: Uptime(secs): 0.0 total, 0.0 interval 2023-12-01T22:10:15.801 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: Flush(GB): cumulative 0.000, interval 0.000 2023-12-01T22:10:15.801 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: AddFile(GB): cumulative 0.000, interval 0.000 2023-12-01T22:10:15.801 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: AddFile(Total Files): cumulative 0, interval 0 2023-12-01T22:10:15.801 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: AddFile(L0 Files): cumulative 0, interval 0 2023-12-01T22:10:15.802 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: AddFile(Keys): cumulative 0, interval 0 2023-12-01T22:10:15.802 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: Cumulative compaction: 0.00 GB write, 21.98 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-12-01T22:10:15.802 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-12-01T22:10:15.802 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-12-01T22:10:15.802 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: 2023-12-01T22:10:15.802 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: ** File Read Latency Histogram By Level [default] ** 2023-12-01T22:10:15.802 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: 2023-12-01T22:10:15.802 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.414+0000 7f66fa451700 0 starting mon.smithi029 rank 0 at public addrs [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] at bind addrs [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi029 fsid 5558ab6c-9096-11ee-95a2-87774f69a715 2023-12-01T22:10:15.802 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.416+0000 7f66fa451700 1 mon.smithi029@-1(???) e1 preinit fsid 5558ab6c-9096-11ee-95a2-87774f69a715 2023-12-01T22:10:15.802 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.416+0000 7f66fa451700 0 mon.smithi029@-1(???).mds e1 new map 2023-12-01T22:10:15.802 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.416+0000 7f66fa451700 0 mon.smithi029@-1(???).mds e1 print_map 2023-12-01T22:10:15.802 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: e1 2023-12-01T22:10:15.802 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: enable_multiple, ever_enabled_multiple: 1,1 2023-12-01T22:10:15.802 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-12-01T22:10:15.803 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: legacy client fscid: -1 2023-12-01T22:10:15.803 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: 2023-12-01T22:10:15.805 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: No filesystems configured 2023-12-01T22:10:15.805 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: 2023-12-01T22:10:15.805 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.417+0000 7f66fa451700 0 mon.smithi029@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2023-12-01T22:10:15.805 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.417+0000 7f66fa451700 0 mon.smithi029@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-12-01T22:10:15.805 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.417+0000 7f66fa451700 0 mon.smithi029@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-12-01T22:10:15.805 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.417+0000 7f66fa451700 0 mon.smithi029@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-12-01T22:10:15.805 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:10:13.520545+0000 mon.smithi029 (mon.0) 8 : audit [INF] from='client.? 172.21.15.29:0/1956185124' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-12-01T22:10:15.805 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:10:13.523861+0000 mon.smithi029 (mon.0) 9 : audit [INF] from='client.? 172.21.15.29:0/1956185124' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2023-12-01T22:10:15.806 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:10:15.417+0000 7f66fa451700 1 mon.smithi029@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2023-12-01T22:10:15.806 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:10:15.435098+0000 mon.smithi029 (mon.0) 1 : cluster [INF] mon.smithi029 is new leader, mons smithi029 in quorum (ranks 0) 2023-12-01T22:10:15.806 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:10:15.435223+0000 mon.smithi029 (mon.0) 2 : cluster [DBG] monmap e1: 1 mons at {smithi029=[v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0]} 2023-12-01T22:10:15.806 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:10:15.435295+0000 mon.smithi029 (mon.0) 3 : cluster [DBG] fsmap 2023-12-01T22:10:15.806 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:10:15.435347+0000 mon.smithi029 (mon.0) 4 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2023-12-01T22:10:15.806 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:10:15.435985+0000 mon.smithi029 (mon.0) 5 : cluster [DBG] mgrmap e1: no daemons active 2023-12-01T22:10:15.809 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:15.792+0000 7f9623d0d700 1 Processor -- start 2023-12-01T22:10:15.809 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:15.792+0000 7f9623d0d700 1 -- start start 2023-12-01T22:10:15.809 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:15.792+0000 7f9623d0d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f961c0fbcb0 0x7f961c0fc0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-01T22:10:15.809 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:15.792+0000 7f9623d0d700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f961c0fc5c0 con 0x7f961c0fbcb0 2023-12-01T22:10:15.809 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:15.793+0000 7f9621aa9700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f961c0fbcb0 0x7f961c0fc0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:10:15.809 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:15.793+0000 7f9621aa9700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f961c0fbcb0 0x7f961c0fc0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57076/0 (socket says 172.21.15.29:57076) 2023-12-01T22:10:15.809 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:15.793+0000 7f9621aa9700 1 -- 172.21.15.29:0/3802352179 learned_addr learned my addr 172.21.15.29:0/3802352179 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-12-01T22:10:15.809 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:15.793+0000 7f9621aa9700 1 -- 172.21.15.29:0/3802352179 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f961c0fc700 con 0x7f961c0fbcb0 2023-12-01T22:10:15.810 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:15.793+0000 7f9621aa9700 1 --2- 172.21.15.29:0/3802352179 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f961c0fbcb0 0x7f961c0fc0d0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f960c009960 tx=0x7f960c0092d0).ready entity=mon.0 client_cookie=f6bc1e3b95dcfeec server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:10:15.810 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:15.793+0000 7f9620aa7700 1 -- 172.21.15.29:0/3802352179 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f960c00b9c0 con 0x7f961c0fbcb0 2023-12-01T22:10:15.810 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:15.793+0000 7f9620aa7700 1 -- 172.21.15.29:0/3802352179 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f960c011400 con 0x7f961c0fbcb0 2023-12-01T22:10:15.810 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:15.794+0000 7f9620aa7700 1 -- 172.21.15.29:0/3802352179 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f960c0103a0 con 0x7f961c0fbcb0 2023-12-01T22:10:15.810 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:15.794+0000 7f9623d0d700 1 -- 172.21.15.29:0/3802352179 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f961c0fbcb0 msgr2=0x7f961c0fc0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:10:15.810 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:15.794+0000 7f9623d0d700 1 --2- 172.21.15.29:0/3802352179 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f961c0fbcb0 0x7f961c0fc0d0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f960c009960 tx=0x7f960c0092d0).stop 2023-12-01T22:10:15.810 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:15.794+0000 7f9623d0d700 1 -- 172.21.15.29:0/3802352179 shutdown_connections 2023-12-01T22:10:15.810 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:15.794+0000 7f9623d0d700 1 --2- 172.21.15.29:0/3802352179 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f961c0fbcb0 0x7f961c0fc0d0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:10:15.810 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:15.794+0000 7f9623d0d700 1 -- 172.21.15.29:0/3802352179 >> 172.21.15.29:0/3802352179 conn(0x7f961c0f74b0 msgr2=0x7f961c0f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:10:15.810 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:15.795+0000 7f9623d0d700 1 -- 172.21.15.29:0/3802352179 shutdown_connections 2023-12-01T22:10:15.810 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:15.795+0000 7f9623d0d700 1 -- 172.21.15.29:0/3802352179 wait complete. 2023-12-01T22:10:15.810 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:15.795+0000 7f9623d0d700 1 Processor -- start 2023-12-01T22:10:15.810 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:15.795+0000 7f9623d0d700 1 -- start start 2023-12-01T22:10:15.810 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:15.796+0000 7f9623d0d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f961c0fbcb0 0x7f961c10aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:10:15.811 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:15.796+0000 7f9623d0d700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f961c10af00 con 0x7f961c0fbcb0 2023-12-01T22:10:15.811 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:15.796+0000 7f9621aa9700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f961c0fbcb0 0x7f961c10aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:10:15.811 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:15.796+0000 7f9621aa9700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f961c0fbcb0 0x7f961c10aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57080/0 (socket says 172.21.15.29:57080) 2023-12-01T22:10:15.811 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:15.796+0000 7f9621aa9700 1 -- 172.21.15.29:0/3867770552 learned_addr learned my addr 172.21.15.29:0/3867770552 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-12-01T22:10:15.811 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:15.796+0000 7f9621aa9700 1 -- 172.21.15.29:0/3867770552 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f960c009570 con 0x7f961c0fbcb0 2023-12-01T22:10:15.811 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:15.796+0000 7f9621aa9700 1 --2- 172.21.15.29:0/3867770552 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f961c0fbcb0 0x7f961c10aa10 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f960c00ba90 tx=0x7f960c0092d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:10:15.811 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:15.797+0000 7f961a7fc700 1 -- 172.21.15.29:0/3867770552 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f960c010870 con 0x7f961c0fbcb0 2023-12-01T22:10:15.811 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:15.797+0000 7f961a7fc700 1 -- 172.21.15.29:0/3867770552 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f960c010e40 con 0x7f961c0fbcb0 2023-12-01T22:10:15.811 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:15.797+0000 7f9623d0d700 1 -- 172.21.15.29:0/3867770552 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f961c10b100 con 0x7f961c0fbcb0 2023-12-01T22:10:15.811 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:15.797+0000 7f961a7fc700 1 -- 172.21.15.29:0/3867770552 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f960c01ae10 con 0x7f961c0fbcb0 2023-12-01T22:10:15.811 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:15.797+0000 7f9623d0d700 1 -- 172.21.15.29:0/3867770552 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f961c10b510 con 0x7f961c0fbcb0 2023-12-01T22:10:15.811 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:15.797+0000 7f961a7fc700 1 -- 172.21.15.29:0/3867770552 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f960c02a070 con 0x7f961c0fbcb0 2023-12-01T22:10:15.811 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:15.797+0000 7f961a7fc700 1 -- 172.21.15.29:0/3867770552 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f960c022780 con 0x7f961c0fbcb0 2023-12-01T22:10:15.811 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:15.798+0000 7f9623d0d700 1 -- 172.21.15.29:0/3867770552 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f961c045ad0 con 0x7f961c0fbcb0 2023-12-01T22:10:15.811 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:15.801+0000 7f961a7fc700 1 -- 172.21.15.29:0/3867770552 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f960c028060 con 0x7f961c0fbcb0 2023-12-01T22:10:15.872 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:15.871+0000 7f9623d0d700 1 -- 172.21.15.29:0/3867770552 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f961c0fd9c0 con 0x7f961c0fbcb0 2023-12-01T22:10:15.875 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:15.873+0000 7f961a7fc700 1 -- 172.21.15.29:0/3867770552 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7f960c01e030 con 0x7f961c0fbcb0 2023-12-01T22:10:15.876 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:15.874+0000 7f9623d0d700 1 -- 172.21.15.29:0/3867770552 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f961c0fbcb0 msgr2=0x7f961c10aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:10:15.876 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:15.874+0000 7f9623d0d700 1 --2- 172.21.15.29:0/3867770552 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f961c0fbcb0 0x7f961c10aa10 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f960c00ba90 tx=0x7f960c0092d0).stop 2023-12-01T22:10:15.876 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:15.874+0000 7f9623d0d700 1 -- 172.21.15.29:0/3867770552 shutdown_connections 2023-12-01T22:10:15.876 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:15.874+0000 7f9623d0d700 1 --2- 172.21.15.29:0/3867770552 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f961c0fbcb0 0x7f961c10aa10 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:10:15.876 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:15.874+0000 7f9623d0d700 1 -- 172.21.15.29:0/3867770552 >> 172.21.15.29:0/3867770552 conn(0x7f961c0f74b0 msgr2=0x7f961c0f8f70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:10:15.877 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:15.874+0000 7f9623d0d700 1 -- 172.21.15.29:0/3867770552 shutdown_connections 2023-12-01T22:10:15.877 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:15.874+0000 7f9623d0d700 1 -- 172.21.15.29:0/3867770552 wait complete. 2023-12-01T22:10:16.236 INFO:teuthology.orchestra.run.smithi029.stderr:Wrote config to /etc/ceph/ceph.conf 2023-12-01T22:10:16.236 INFO:teuthology.orchestra.run.smithi029.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2023-12-01T22:10:16.236 INFO:teuthology.orchestra.run.smithi029.stderr:Creating mgr... 2023-12-01T22:10:16.237 INFO:teuthology.orchestra.run.smithi029.stderr:Verifying port 9283 ... 2023-12-01T22:10:16.239 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: systemctl daemon-reload 2023-12-01T22:10:16.355 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: systemctl stop ceph-5558ab6c-9096-11ee-95a2-87774f69a715@mgr.smithi029.loxrfu 2023-12-01T22:10:16.383 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: systemctl reset-failed ceph-5558ab6c-9096-11ee-95a2-87774f69a715@mgr.smithi029.loxrfu 2023-12-01T22:10:16.390 INFO:teuthology.orchestra.run.smithi029.stderr:systemctl: Failed to reset failed state of unit ceph-5558ab6c-9096-11ee-95a2-87774f69a715@mgr.smithi029.loxrfu.service: Unit ceph-5558ab6c-9096-11ee-95a2-87774f69a715@mgr.smithi029.loxrfu.service not loaded. 2023-12-01T22:10:16.391 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: systemctl enable ceph-5558ab6c-9096-11ee-95a2-87774f69a715@mgr.smithi029.loxrfu 2023-12-01T22:10:16.397 INFO:teuthology.orchestra.run.smithi029.stderr:systemctl: Created symlink /etc/systemd/system/ceph-5558ab6c-9096-11ee-95a2-87774f69a715.target.wants/ceph-5558ab6c-9096-11ee-95a2-87774f69a715@mgr.smithi029.loxrfu.service → /etc/systemd/system/ceph-5558ab6c-9096-11ee-95a2-87774f69a715@.service. 2023-12-01T22:10:16.491 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: systemctl start ceph-5558ab6c-9096-11ee-95a2-87774f69a715@mgr.smithi029.loxrfu 2023-12-01T22:10:16.830 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: systemctl is-enabled firewalld.service 2023-12-01T22:10:16.839 INFO:teuthology.orchestra.run.smithi029.stderr:systemctl: disabled 2023-12-01T22:10:16.840 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: systemctl is-active firewalld.service 2023-12-01T22:10:16.848 INFO:teuthology.orchestra.run.smithi029.stderr:systemctl: inactive 2023-12-01T22:10:16.849 INFO:teuthology.orchestra.run.smithi029.stderr:firewalld.service is not enabled 2023-12-01T22:10:16.849 INFO:teuthology.orchestra.run.smithi029.stderr:Not possible to enable service . firewalld.service is not available 2023-12-01T22:10:16.849 INFO:teuthology.orchestra.run.smithi029.stderr:Not possible to open ports <[]>. firewalld.service is not available 2023-12-01T22:10:16.849 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: systemctl is-enabled firewalld.service 2023-12-01T22:10:16.858 INFO:teuthology.orchestra.run.smithi029.stderr:systemctl: disabled 2023-12-01T22:10:16.858 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: systemctl is-active firewalld.service 2023-12-01T22:10:16.867 INFO:teuthology.orchestra.run.smithi029.stderr:systemctl: inactive 2023-12-01T22:10:16.868 INFO:teuthology.orchestra.run.smithi029.stderr:firewalld.service is not enabled 2023-12-01T22:10:16.868 INFO:teuthology.orchestra.run.smithi029.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2023-12-01T22:10:16.868 INFO:teuthology.orchestra.run.smithi029.stderr:Waiting for mgr to start... 2023-12-01T22:10:16.868 INFO:teuthology.orchestra.run.smithi029.stderr:Waiting for mgr... 2023-12-01T22:10:16.868 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi029 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/5558ab6c-9096-11ee-95a2-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpou9lnrt4:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpxe9edifa:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2023-12-01T22:10:17.247 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:17.245+0000 7f519185a700 1 Processor -- start 2023-12-01T22:10:17.248 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:17.245+0000 7f519185a700 1 -- start start 2023-12-01T22:10:17.248 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:17.245+0000 7f519185a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f518c070420 0x7f518c0fe970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-01T22:10:17.248 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:17.245+0000 7f519185a700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f518c0fee60 con 0x7f518c070420 2023-12-01T22:10:17.248 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:17.245+0000 7f518affd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f518c070420 0x7f518c0fe970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:10:17.249 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:17.245+0000 7f518affd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f518c070420 0x7f518c0fe970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57094/0 (socket says 172.21.15.29:57094) 2023-12-01T22:10:17.250 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:17.245+0000 7f518affd700 1 -- 172.21.15.29:0/1305144339 learned_addr learned my addr 172.21.15.29:0/1305144339 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-12-01T22:10:17.250 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:17.246+0000 7f518affd700 1 -- 172.21.15.29:0/1305144339 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f518c0fefa0 con 0x7f518c070420 2023-12-01T22:10:17.250 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:17.246+0000 7f518affd700 1 --2- 172.21.15.29:0/1305144339 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f518c070420 0x7f518c0fe970 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f517c00b5f0 tx=0x7f517c006ee0).ready entity=mon.0 client_cookie=8e645e428541a453 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:10:17.250 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:17.246+0000 7f5189ffb700 1 -- 172.21.15.29:0/1305144339 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f517c00d600 con 0x7f518c070420 2023-12-01T22:10:17.250 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:17.246+0000 7f5189ffb700 1 -- 172.21.15.29:0/1305144339 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f517c00dbd0 con 0x7f518c070420 2023-12-01T22:10:17.250 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:17.246+0000 7f5189ffb700 1 -- 172.21.15.29:0/1305144339 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f517c011c00 con 0x7f518c070420 2023-12-01T22:10:17.250 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:17.247+0000 7f519185a700 1 -- 172.21.15.29:0/1305144339 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f518c070420 msgr2=0x7f518c0fe970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:10:17.251 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:17.247+0000 7f519185a700 1 --2- 172.21.15.29:0/1305144339 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f518c070420 0x7f518c0fe970 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f517c00b5f0 tx=0x7f517c006ee0).stop 2023-12-01T22:10:17.251 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:17.249+0000 7f519185a700 1 -- 172.21.15.29:0/1305144339 shutdown_connections 2023-12-01T22:10:17.251 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:17.249+0000 7f519185a700 1 --2- 172.21.15.29:0/1305144339 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f518c070420 0x7f518c0fe970 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:10:17.251 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:17.249+0000 7f519185a700 1 -- 172.21.15.29:0/1305144339 >> 172.21.15.29:0/1305144339 conn(0x7f518c0f63b0 msgr2=0x7f518c0f87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:10:17.251 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:17.249+0000 7f519185a700 1 -- 172.21.15.29:0/1305144339 shutdown_connections 2023-12-01T22:10:17.251 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:17.249+0000 7f519185a700 1 -- 172.21.15.29:0/1305144339 wait complete. 2023-12-01T22:10:17.252 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:17.250+0000 7f519185a700 1 Processor -- start 2023-12-01T22:10:17.254 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:17.250+0000 7f519185a700 1 -- start start 2023-12-01T22:10:17.254 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:17.250+0000 7f519185a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f518c070420 0x7f518c10a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:10:17.254 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:17.250+0000 7f519185a700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f518c10aed0 con 0x7f518c070420 2023-12-01T22:10:17.254 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:17.250+0000 7f518affd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f518c070420 0x7f518c10a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:10:17.254 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:17.250+0000 7f518affd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f518c070420 0x7f518c10a9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57110/0 (socket says 172.21.15.29:57110) 2023-12-01T22:10:17.254 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:17.250+0000 7f518affd700 1 -- 172.21.15.29:0/3089530932 learned_addr learned my addr 172.21.15.29:0/3089530932 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-12-01T22:10:17.254 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:17.251+0000 7f518affd700 1 -- 172.21.15.29:0/3089530932 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f517c00b200 con 0x7f518c070420 2023-12-01T22:10:17.255 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:17.251+0000 7f518affd700 1 --2- 172.21.15.29:0/3089530932 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f518c070420 0x7f518c10a9e0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f517c00e040 tx=0x7f517c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:10:17.255 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:17.251+0000 7f51737fe700 1 -- 172.21.15.29:0/3089530932 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f517c00d9f0 con 0x7f518c070420 2023-12-01T22:10:17.255 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:17.251+0000 7f51737fe700 1 -- 172.21.15.29:0/3089530932 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f517c01a9f0 con 0x7f518c070420 2023-12-01T22:10:17.255 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:17.251+0000 7f519185a700 1 -- 172.21.15.29:0/3089530932 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f518c10b0d0 con 0x7f518c070420 2023-12-01T22:10:17.255 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:17.251+0000 7f51737fe700 1 -- 172.21.15.29:0/3089530932 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f517c01b890 con 0x7f518c070420 2023-12-01T22:10:17.255 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:17.251+0000 7f519185a700 1 -- 172.21.15.29:0/3089530932 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f518c10b4e0 con 0x7f518c070420 2023-12-01T22:10:17.255 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:17.251+0000 7f51737fe700 1 -- 172.21.15.29:0/3089530932 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f517c00f070 con 0x7f518c070420 2023-12-01T22:10:17.255 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:17.251+0000 7f51737fe700 1 -- 172.21.15.29:0/3089530932 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f517c025b30 con 0x7f518c070420 2023-12-01T22:10:17.255 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:17.252+0000 7f519185a700 1 -- 172.21.15.29:0/3089530932 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f518c045ad0 con 0x7f518c070420 2023-12-01T22:10:17.256 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:17.255+0000 7f51737fe700 1 -- 172.21.15.29:0/3089530932 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f517c016070 con 0x7f518c070420 2023-12-01T22:10:17.276 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:16 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:10:15.874588+0000 mon.smithi029 (mon.0) 6 : audit [INF] from='client.? 172.21.15.29:0/3867770552' entity='client.admin' 2023-12-01T22:10:17.330 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:17.328+0000 7f519185a700 1 -- 172.21.15.29:0/3089530932 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f518c1047a0 con 0x7f518c070420 2023-12-01T22:10:17.330 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:17.328+0000 7f51737fe700 1 -- 172.21.15.29:0/3089530932 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7f517c025cf0 con 0x7f518c070420 2023-12-01T22:10:17.330 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:17.330 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: { 2023-12-01T22:10:17.331 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "fsid": "5558ab6c-9096-11ee-95a2-87774f69a715", 2023-12-01T22:10:17.332 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "health": { 2023-12-01T22:10:17.332 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2023-12-01T22:10:17.332 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "checks": {}, 2023-12-01T22:10:17.332 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "mutes": [] 2023-12-01T22:10:17.332 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: }, 2023-12-01T22:10:17.332 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "election_epoch": 5, 2023-12-01T22:10:17.332 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "quorum": [ 2023-12-01T22:10:17.332 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 0 2023-12-01T22:10:17.333 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: ], 2023-12-01T22:10:17.333 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "quorum_names": [ 2023-12-01T22:10:17.333 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "smithi029" 2023-12-01T22:10:17.333 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: ], 2023-12-01T22:10:17.334 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "quorum_age": 1, 2023-12-01T22:10:17.334 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "monmap": { 2023-12-01T22:10:17.334 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "epoch": 1, 2023-12-01T22:10:17.334 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2023-12-01T22:10:17.334 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "num_mons": 1 2023-12-01T22:10:17.334 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: }, 2023-12-01T22:10:17.334 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "osdmap": { 2023-12-01T22:10:17.334 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "epoch": 1, 2023-12-01T22:10:17.335 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "num_osds": 0, 2023-12-01T22:10:17.336 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "num_up_osds": 0, 2023-12-01T22:10:17.336 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "osd_up_since": 0, 2023-12-01T22:10:17.336 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "num_in_osds": 0, 2023-12-01T22:10:17.336 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "osd_in_since": 0, 2023-12-01T22:10:17.336 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2023-12-01T22:10:17.336 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: }, 2023-12-01T22:10:17.336 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "pgmap": { 2023-12-01T22:10:17.336 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "pgs_by_state": [], 2023-12-01T22:10:17.336 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "num_pgs": 0, 2023-12-01T22:10:17.336 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "num_pools": 0, 2023-12-01T22:10:17.336 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "num_objects": 0, 2023-12-01T22:10:17.336 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "data_bytes": 0, 2023-12-01T22:10:17.336 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "bytes_used": 0, 2023-12-01T22:10:17.338 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "bytes_avail": 0, 2023-12-01T22:10:17.338 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "bytes_total": 0 2023-12-01T22:10:17.338 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: }, 2023-12-01T22:10:17.338 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "fsmap": { 2023-12-01T22:10:17.338 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "epoch": 1, 2023-12-01T22:10:17.338 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "by_rank": [], 2023-12-01T22:10:17.338 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "up:standby": 0 2023-12-01T22:10:17.338 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: }, 2023-12-01T22:10:17.338 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "mgrmap": { 2023-12-01T22:10:17.338 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "available": false, 2023-12-01T22:10:17.338 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "num_standbys": 0, 2023-12-01T22:10:17.338 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "modules": [ 2023-12-01T22:10:17.338 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "iostat", 2023-12-01T22:10:17.339 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "restful" 2023-12-01T22:10:17.340 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: ], 2023-12-01T22:10:17.340 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "services": {} 2023-12-01T22:10:17.340 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: }, 2023-12-01T22:10:17.340 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "servicemap": { 2023-12-01T22:10:17.340 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "epoch": 1, 2023-12-01T22:10:17.340 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "modified": "2023-12-01T22:10:12.142858+0000", 2023-12-01T22:10:17.340 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "services": {} 2023-12-01T22:10:17.340 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: }, 2023-12-01T22:10:17.340 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "progress_events": {} 2023-12-01T22:10:17.341 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: } 2023-12-01T22:10:17.341 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:17.329+0000 7f519185a700 1 -- 172.21.15.29:0/3089530932 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f518c070420 msgr2=0x7f518c10a9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:10:17.341 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:17.329+0000 7f519185a700 1 --2- 172.21.15.29:0/3089530932 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f518c070420 0x7f518c10a9e0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f517c00e040 tx=0x7f517c006ee0).stop 2023-12-01T22:10:17.341 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:17.329+0000 7f519185a700 1 -- 172.21.15.29:0/3089530932 shutdown_connections 2023-12-01T22:10:17.341 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:17.329+0000 7f519185a700 1 --2- 172.21.15.29:0/3089530932 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f518c070420 0x7f518c10a9e0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:10:17.341 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:17.329+0000 7f519185a700 1 -- 172.21.15.29:0/3089530932 >> 172.21.15.29:0/3089530932 conn(0x7f518c0f63b0 msgr2=0x7f518c0f7eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:10:17.341 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:17.329+0000 7f519185a700 1 -- 172.21.15.29:0/3089530932 shutdown_connections 2023-12-01T22:10:17.341 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:17.329+0000 7f519185a700 1 -- 172.21.15.29:0/3089530932 wait complete. 2023-12-01T22:10:17.701 INFO:teuthology.orchestra.run.smithi029.stderr:mgr not available, waiting (1/15)... 2023-12-01T22:10:18.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:17 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:10:17.329609+0000 mon.smithi029 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.29:0/3089530932' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-12-01T22:10:19.704 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi029 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/5558ab6c-9096-11ee-95a2-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpou9lnrt4:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpxe9edifa:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2023-12-01T22:10:20.097 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:20.094+0000 7f65aae17700 1 Processor -- start 2023-12-01T22:10:20.097 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:20.095+0000 7f65aae17700 1 -- start start 2023-12-01T22:10:20.097 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:20.095+0000 7f65aae17700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65a40703c0 0x7f65a40fe9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-01T22:10:20.098 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:20.095+0000 7f65aae17700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65a40707e0 con 0x7f65a40703c0 2023-12-01T22:10:20.098 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:20.095+0000 7f65a8bb3700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65a40703c0 0x7f65a40fe9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:10:20.098 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:20.095+0000 7f65a8bb3700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65a40703c0 0x7f65a40fe9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57126/0 (socket says 172.21.15.29:57126) 2023-12-01T22:10:20.098 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:20.095+0000 7f65a8bb3700 1 -- 172.21.15.29:0/331247240 learned_addr learned my addr 172.21.15.29:0/331247240 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-12-01T22:10:20.099 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:20.095+0000 7f65a8bb3700 1 -- 172.21.15.29:0/331247240 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65a40feec0 con 0x7f65a40703c0 2023-12-01T22:10:20.099 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:20.095+0000 7f65a8bb3700 1 --2- 172.21.15.29:0/331247240 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65a40703c0 0x7f65a40fe9d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f659800b5f0 tx=0x7f6598006ee0).ready entity=mon.0 client_cookie=73effed7d2c9f19b server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:10:20.099 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:20.096+0000 7f65a37fe700 1 -- 172.21.15.29:0/331247240 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f659800d600 con 0x7f65a40703c0 2023-12-01T22:10:20.100 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:20.096+0000 7f65a37fe700 1 -- 172.21.15.29:0/331247240 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f659800dbd0 con 0x7f65a40703c0 2023-12-01T22:10:20.100 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:20.096+0000 7f65a37fe700 1 -- 172.21.15.29:0/331247240 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6598011c00 con 0x7f65a40703c0 2023-12-01T22:10:20.100 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:20.096+0000 7f65aae17700 1 -- 172.21.15.29:0/331247240 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65a40703c0 msgr2=0x7f65a40fe9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:10:20.100 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:20.096+0000 7f65aae17700 1 --2- 172.21.15.29:0/331247240 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65a40703c0 0x7f65a40fe9d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f659800b5f0 tx=0x7f6598006ee0).stop 2023-12-01T22:10:20.100 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:20.096+0000 7f65aae17700 1 -- 172.21.15.29:0/331247240 shutdown_connections 2023-12-01T22:10:20.100 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:20.096+0000 7f65aae17700 1 --2- 172.21.15.29:0/331247240 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65a40703c0 0x7f65a40fe9d0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:10:20.100 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:20.096+0000 7f65aae17700 1 -- 172.21.15.29:0/331247240 >> 172.21.15.29:0/331247240 conn(0x7f65a40f63b0 msgr2=0x7f65a40f87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:10:20.100 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:20.096+0000 7f65aae17700 1 -- 172.21.15.29:0/331247240 shutdown_connections 2023-12-01T22:10:20.100 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:20.097+0000 7f65aae17700 1 -- 172.21.15.29:0/331247240 wait complete. 2023-12-01T22:10:20.100 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:20.097+0000 7f65aae17700 1 Processor -- start 2023-12-01T22:10:20.100 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:20.097+0000 7f65aae17700 1 -- start start 2023-12-01T22:10:20.102 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:20.097+0000 7f65aae17700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65a40703c0 0x7f65a40fc450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:10:20.102 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:20.097+0000 7f65aae17700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65a40fa8d0 con 0x7f65a40703c0 2023-12-01T22:10:20.102 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:20.098+0000 7f65a8bb3700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65a40703c0 0x7f65a40fc450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:10:20.102 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:20.098+0000 7f65a8bb3700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65a40703c0 0x7f65a40fc450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57138/0 (socket says 172.21.15.29:57138) 2023-12-01T22:10:20.102 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:20.098+0000 7f65a8bb3700 1 -- 172.21.15.29:0/370088032 learned_addr learned my addr 172.21.15.29:0/370088032 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-12-01T22:10:20.102 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:20.098+0000 7f65a8bb3700 1 -- 172.21.15.29:0/370088032 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f659800b200 con 0x7f65a40703c0 2023-12-01T22:10:20.102 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:20.098+0000 7f65a8bb3700 1 --2- 172.21.15.29:0/370088032 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65a40703c0 0x7f65a40fc450 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f659800b190 tx=0x7f6598006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:10:20.103 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:20.098+0000 7f65a17fa700 1 -- 172.21.15.29:0/370088032 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f659801b410 con 0x7f65a40703c0 2023-12-01T22:10:20.103 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:20.098+0000 7f65a17fa700 1 -- 172.21.15.29:0/370088032 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f659801b9e0 con 0x7f65a40703c0 2023-12-01T22:10:20.103 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:20.098+0000 7f65aae17700 1 -- 172.21.15.29:0/370088032 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65a40faa70 con 0x7f65a40703c0 2023-12-01T22:10:20.103 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:20.098+0000 7f65a17fa700 1 -- 172.21.15.29:0/370088032 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f659801a8f0 con 0x7f65a40703c0 2023-12-01T22:10:20.103 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:20.098+0000 7f65aae17700 1 -- 172.21.15.29:0/370088032 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65a40fae80 con 0x7f65a40703c0 2023-12-01T22:10:20.103 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:20.099+0000 7f65a17fa700 1 -- 172.21.15.29:0/370088032 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f659800f070 con 0x7f65a40703c0 2023-12-01T22:10:20.104 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:20.099+0000 7f65a17fa700 1 -- 172.21.15.29:0/370088032 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f6598025cf0 con 0x7f65a40703c0 2023-12-01T22:10:20.104 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:20.099+0000 7f65aae17700 1 -- 172.21.15.29:0/370088032 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65a4045ad0 con 0x7f65a40703c0 2023-12-01T22:10:20.105 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:20.102+0000 7f65a17fa700 1 -- 172.21.15.29:0/370088032 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f6598016070 con 0x7f65a40703c0 2023-12-01T22:10:20.187 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:20.178+0000 7f65aae17700 1 -- 172.21.15.29:0/370088032 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f65a40008d0 con 0x7f65a40703c0 2023-12-01T22:10:20.187 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:20.178+0000 7f65a17fa700 1 -- 172.21.15.29:0/370088032 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7f6598025460 con 0x7f65a40703c0 2023-12-01T22:10:20.187 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:20.187 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: { 2023-12-01T22:10:20.187 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "fsid": "5558ab6c-9096-11ee-95a2-87774f69a715", 2023-12-01T22:10:20.187 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "health": { 2023-12-01T22:10:20.187 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2023-12-01T22:10:20.187 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "checks": {}, 2023-12-01T22:10:20.187 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "mutes": [] 2023-12-01T22:10:20.187 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: }, 2023-12-01T22:10:20.188 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "election_epoch": 5, 2023-12-01T22:10:20.188 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "quorum": [ 2023-12-01T22:10:20.188 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 0 2023-12-01T22:10:20.188 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: ], 2023-12-01T22:10:20.188 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "quorum_names": [ 2023-12-01T22:10:20.188 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "smithi029" 2023-12-01T22:10:20.188 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: ], 2023-12-01T22:10:20.188 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "quorum_age": 4, 2023-12-01T22:10:20.188 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "monmap": { 2023-12-01T22:10:20.188 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "epoch": 1, 2023-12-01T22:10:20.188 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2023-12-01T22:10:20.188 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "num_mons": 1 2023-12-01T22:10:20.189 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: }, 2023-12-01T22:10:20.189 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "osdmap": { 2023-12-01T22:10:20.189 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "epoch": 1, 2023-12-01T22:10:20.189 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "num_osds": 0, 2023-12-01T22:10:20.189 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "num_up_osds": 0, 2023-12-01T22:10:20.189 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "osd_up_since": 0, 2023-12-01T22:10:20.189 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "num_in_osds": 0, 2023-12-01T22:10:20.189 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "osd_in_since": 0, 2023-12-01T22:10:20.189 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2023-12-01T22:10:20.189 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: }, 2023-12-01T22:10:20.189 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "pgmap": { 2023-12-01T22:10:20.189 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "pgs_by_state": [], 2023-12-01T22:10:20.189 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "num_pgs": 0, 2023-12-01T22:10:20.189 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "num_pools": 0, 2023-12-01T22:10:20.189 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "num_objects": 0, 2023-12-01T22:10:20.190 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "data_bytes": 0, 2023-12-01T22:10:20.190 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "bytes_used": 0, 2023-12-01T22:10:20.190 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "bytes_avail": 0, 2023-12-01T22:10:20.190 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "bytes_total": 0 2023-12-01T22:10:20.190 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: }, 2023-12-01T22:10:20.190 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "fsmap": { 2023-12-01T22:10:20.191 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "epoch": 1, 2023-12-01T22:10:20.191 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "by_rank": [], 2023-12-01T22:10:20.191 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "up:standby": 0 2023-12-01T22:10:20.191 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: }, 2023-12-01T22:10:20.191 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "mgrmap": { 2023-12-01T22:10:20.191 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "available": false, 2023-12-01T22:10:20.191 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "num_standbys": 0, 2023-12-01T22:10:20.191 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "modules": [ 2023-12-01T22:10:20.191 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "iostat", 2023-12-01T22:10:20.191 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "restful" 2023-12-01T22:10:20.191 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: ], 2023-12-01T22:10:20.191 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "services": {} 2023-12-01T22:10:20.191 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: }, 2023-12-01T22:10:20.192 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "servicemap": { 2023-12-01T22:10:20.192 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "epoch": 1, 2023-12-01T22:10:20.192 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "modified": "2023-12-01T22:10:12.142858+0000", 2023-12-01T22:10:20.192 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "services": {} 2023-12-01T22:10:20.192 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: }, 2023-12-01T22:10:20.192 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "progress_events": {} 2023-12-01T22:10:20.192 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: } 2023-12-01T22:10:20.192 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:20.179+0000 7f65aae17700 1 -- 172.21.15.29:0/370088032 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65a40703c0 msgr2=0x7f65a40fc450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:10:20.192 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:20.179+0000 7f65aae17700 1 --2- 172.21.15.29:0/370088032 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65a40703c0 0x7f65a40fc450 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f659800b190 tx=0x7f6598006ee0).stop 2023-12-01T22:10:20.192 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:20.179+0000 7f65aae17700 1 -- 172.21.15.29:0/370088032 shutdown_connections 2023-12-01T22:10:20.192 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:20.179+0000 7f65aae17700 1 --2- 172.21.15.29:0/370088032 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65a40703c0 0x7f65a40fc450 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:10:20.192 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:20.179+0000 7f65aae17700 1 -- 172.21.15.29:0/370088032 >> 172.21.15.29:0/370088032 conn(0x7f65a40f63b0 msgr2=0x7f65a40f7ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:10:20.192 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:20.180+0000 7f65aae17700 1 -- 172.21.15.29:0/370088032 shutdown_connections 2023-12-01T22:10:20.192 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:20.180+0000 7f65aae17700 1 -- 172.21.15.29:0/370088032 wait complete. 2023-12-01T22:10:20.487 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:20 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:10:20.179620+0000 mon.smithi029 (mon.0) 8 : audit [DBG] from='client.? 172.21.15.29:0/370088032' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-12-01T22:10:20.542 INFO:teuthology.orchestra.run.smithi029.stderr:mgr not available, waiting (2/15)... 2023-12-01T22:10:22.545 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi029 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/5558ab6c-9096-11ee-95a2-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpou9lnrt4:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpxe9edifa:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2023-12-01T22:10:22.777 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:22 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:10:22.467337+0000 mon.smithi029 (mon.0) 9 : cluster [INF] Activating manager daemon smithi029.loxrfu 2023-12-01T22:10:22.777 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:22 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:10:22.470192+0000 mon.smithi029 (mon.0) 10 : cluster [DBG] mgrmap e2: smithi029.loxrfu(active, starting, since 0.00298583s) 2023-12-01T22:10:22.777 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:22 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:10:22.470924+0000 mon.smithi029 (mon.0) 11 : audit [DBG] from='mgr.14100 172.21.15.29:0/3155763299' entity='mgr.smithi029.loxrfu' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-12-01T22:10:22.778 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:22 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:10:22.471166+0000 mon.smithi029 (mon.0) 12 : audit [DBG] from='mgr.14100 172.21.15.29:0/3155763299' entity='mgr.smithi029.loxrfu' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-12-01T22:10:22.778 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:22 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:10:22.471290+0000 mon.smithi029 (mon.0) 13 : audit [DBG] from='mgr.14100 172.21.15.29:0/3155763299' entity='mgr.smithi029.loxrfu' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-12-01T22:10:22.778 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:22 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:10:22.471433+0000 mon.smithi029 (mon.0) 14 : audit [DBG] from='mgr.14100 172.21.15.29:0/3155763299' entity='mgr.smithi029.loxrfu' cmd=[{"prefix": "mon metadata", "id": "smithi029"}]: dispatch 2023-12-01T22:10:22.778 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:22 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:10:22.471916+0000 mon.smithi029 (mon.0) 15 : audit [DBG] from='mgr.14100 172.21.15.29:0/3155763299' entity='mgr.smithi029.loxrfu' cmd=[{"prefix": "mgr metadata", "who": "smithi029.loxrfu", "id": "smithi029.loxrfu"}]: dispatch 2023-12-01T22:10:22.778 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:22 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:10:22.485830+0000 mon.smithi029 (mon.0) 16 : cluster [INF] Manager daemon smithi029.loxrfu is now available 2023-12-01T22:10:22.778 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:22 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:10:22.493896+0000 mon.smithi029 (mon.0) 17 : audit [INF] from='mgr.14100 172.21.15.29:0/3155763299' entity='mgr.smithi029.loxrfu' 2023-12-01T22:10:22.778 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:22 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:10:22.499219+0000 mon.smithi029 (mon.0) 18 : audit [INF] from='mgr.14100 172.21.15.29:0/3155763299' entity='mgr.smithi029.loxrfu' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi029.loxrfu/mirror_snapshot_schedule"}]: dispatch 2023-12-01T22:10:22.778 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:22 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:10:22.500651+0000 mon.smithi029 (mon.0) 19 : audit [INF] from='mgr.14100 172.21.15.29:0/3155763299' entity='mgr.smithi029.loxrfu' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi029.loxrfu/trash_purge_schedule"}]: dispatch 2023-12-01T22:10:22.779 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:22 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:10:22.505999+0000 mon.smithi029 (mon.0) 20 : audit [INF] from='mgr.14100 172.21.15.29:0/3155763299' entity='mgr.smithi029.loxrfu' 2023-12-01T22:10:22.779 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:22 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:10:22.508266+0000 mon.smithi029 (mon.0) 21 : audit [INF] from='mgr.14100 172.21.15.29:0/3155763299' entity='mgr.smithi029.loxrfu' 2023-12-01T22:10:22.884 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:22.880+0000 7fa9652ea700 1 Processor -- start 2023-12-01T22:10:22.884 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:22.880+0000 7fa9652ea700 1 -- start start 2023-12-01T22:10:22.884 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:22.880+0000 7fa9652ea700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa9600703f0 0x7fa960100ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-01T22:10:22.885 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:22.880+0000 7fa9652ea700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9600fc590 con 0x7fa9600703f0 2023-12-01T22:10:22.885 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:22.881+0000 7fa95effd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa9600703f0 0x7fa960100ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:10:22.885 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:22.881+0000 7fa95effd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa9600703f0 0x7fa960100ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57194/0 (socket says 172.21.15.29:57194) 2023-12-01T22:10:22.885 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:22.881+0000 7fa95effd700 1 -- 172.21.15.29:0/391786401 learned_addr learned my addr 172.21.15.29:0/391786401 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-12-01T22:10:22.885 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:22.881+0000 7fa95effd700 1 -- 172.21.15.29:0/391786401 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa960101090 con 0x7fa9600703f0 2023-12-01T22:10:22.885 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:22.881+0000 7fa95effd700 1 --2- 172.21.15.29:0/391786401 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa9600703f0 0x7fa960100ba0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7fa95000b5f0 tx=0x7fa950006ee0).ready entity=mon.0 client_cookie=3bf452165aa1fecd server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:10:22.885 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:22.881+0000 7fa95e7fc700 1 -- 172.21.15.29:0/391786401 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa95000d600 con 0x7fa9600703f0 2023-12-01T22:10:22.885 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:22.881+0000 7fa95e7fc700 1 -- 172.21.15.29:0/391786401 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fa95000dbd0 con 0x7fa9600703f0 2023-12-01T22:10:22.885 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:22.881+0000 7fa95e7fc700 1 -- 172.21.15.29:0/391786401 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa950011c00 con 0x7fa9600703f0 2023-12-01T22:10:22.885 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:22.882+0000 7fa9652ea700 1 -- 172.21.15.29:0/391786401 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa9600703f0 msgr2=0x7fa960100ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:10:22.885 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:22.882+0000 7fa9652ea700 1 --2- 172.21.15.29:0/391786401 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa9600703f0 0x7fa960100ba0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7fa95000b5f0 tx=0x7fa950006ee0).stop 2023-12-01T22:10:22.885 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:22.882+0000 7fa9652ea700 1 -- 172.21.15.29:0/391786401 shutdown_connections 2023-12-01T22:10:22.886 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:22.882+0000 7fa9652ea700 1 --2- 172.21.15.29:0/391786401 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa9600703f0 0x7fa960100ba0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:10:22.886 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:22.882+0000 7fa9652ea700 1 -- 172.21.15.29:0/391786401 >> 172.21.15.29:0/391786401 conn(0x7fa9600f6240 msgr2=0x7fa9600f8660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:10:22.886 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:22.882+0000 7fa9652ea700 1 -- 172.21.15.29:0/391786401 shutdown_connections 2023-12-01T22:10:22.886 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:22.882+0000 7fa9652ea700 1 -- 172.21.15.29:0/391786401 wait complete. 2023-12-01T22:10:22.886 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:22.883+0000 7fa9652ea700 1 Processor -- start 2023-12-01T22:10:22.887 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:22.883+0000 7fa9652ea700 1 -- start start 2023-12-01T22:10:22.887 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:22.883+0000 7fa9652ea700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa9600703f0 0x7fa96010ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:10:22.887 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:22.883+0000 7fa9652ea700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa96010cf30 con 0x7fa9600703f0 2023-12-01T22:10:22.887 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:22.884+0000 7fa95effd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa9600703f0 0x7fa96010ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:10:22.888 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:22.884+0000 7fa95effd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa9600703f0 0x7fa96010ca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57204/0 (socket says 172.21.15.29:57204) 2023-12-01T22:10:22.888 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:22.884+0000 7fa95effd700 1 -- 172.21.15.29:0/2058984965 learned_addr learned my addr 172.21.15.29:0/2058984965 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-12-01T22:10:22.888 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:22.884+0000 7fa95effd700 1 -- 172.21.15.29:0/2058984965 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa95000b200 con 0x7fa9600703f0 2023-12-01T22:10:22.888 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:22.884+0000 7fa95effd700 1 --2- 172.21.15.29:0/2058984965 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa9600703f0 0x7fa96010ca40 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7fa95000b190 tx=0x7fa950006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:10:22.888 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:22.884+0000 7fa957fff700 1 -- 172.21.15.29:0/2058984965 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa95001b410 con 0x7fa9600703f0 2023-12-01T22:10:22.888 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:22.884+0000 7fa957fff700 1 -- 172.21.15.29:0/2058984965 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fa95001b9e0 con 0x7fa9600703f0 2023-12-01T22:10:22.888 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:22.884+0000 7fa9652ea700 1 -- 172.21.15.29:0/2058984965 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa96010d130 con 0x7fa9600703f0 2023-12-01T22:10:22.888 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:22.884+0000 7fa957fff700 1 -- 172.21.15.29:0/2058984965 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa95001a880 con 0x7fa9600703f0 2023-12-01T22:10:22.888 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:22.884+0000 7fa9652ea700 1 -- 172.21.15.29:0/2058984965 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa96010d540 con 0x7fa9600703f0 2023-12-01T22:10:22.888 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:22.885+0000 7fa957fff700 1 -- 172.21.15.29:0/2058984965 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 2) v1 ==== 35959+0+0 (secure 0 0 0) 0x7fa95000f070 con 0x7fa9600703f0 2023-12-01T22:10:22.888 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:22.885+0000 7fa957fff700 1 -- 172.21.15.29:0/2058984965 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fa950040ff0 con 0x7fa9600703f0 2023-12-01T22:10:22.889 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:22.886+0000 7fa9652ea700 1 -- 172.21.15.29:0/2058984965 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa960045ad0 con 0x7fa9600703f0 2023-12-01T22:10:22.889 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:22.888+0000 7fa957fff700 1 -- 172.21.15.29:0/2058984965 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7fa950016070 con 0x7fa9600703f0 2023-12-01T22:10:22.969 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:22.960+0000 7fa9652ea700 1 -- 172.21.15.29:0/2058984965 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fa9601068a0 con 0x7fa9600703f0 2023-12-01T22:10:22.969 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:22.961+0000 7fa957fff700 1 -- 172.21.15.29:0/2058984965 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7fa9500224f0 con 0x7fa9600703f0 2023-12-01T22:10:22.969 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:22.969 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: { 2023-12-01T22:10:22.969 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "fsid": "5558ab6c-9096-11ee-95a2-87774f69a715", 2023-12-01T22:10:22.969 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "health": { 2023-12-01T22:10:22.970 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2023-12-01T22:10:22.970 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "checks": {}, 2023-12-01T22:10:22.970 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "mutes": [] 2023-12-01T22:10:22.970 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: }, 2023-12-01T22:10:22.970 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "election_epoch": 5, 2023-12-01T22:10:22.970 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "quorum": [ 2023-12-01T22:10:22.970 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 0 2023-12-01T22:10:22.970 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: ], 2023-12-01T22:10:22.970 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "quorum_names": [ 2023-12-01T22:10:22.970 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "smithi029" 2023-12-01T22:10:22.970 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: ], 2023-12-01T22:10:22.970 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "quorum_age": 7, 2023-12-01T22:10:22.971 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "monmap": { 2023-12-01T22:10:22.971 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "epoch": 1, 2023-12-01T22:10:22.971 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2023-12-01T22:10:22.971 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "num_mons": 1 2023-12-01T22:10:22.971 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: }, 2023-12-01T22:10:22.971 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "osdmap": { 2023-12-01T22:10:22.971 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "epoch": 1, 2023-12-01T22:10:22.971 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "num_osds": 0, 2023-12-01T22:10:22.971 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "num_up_osds": 0, 2023-12-01T22:10:22.971 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "osd_up_since": 0, 2023-12-01T22:10:22.971 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "num_in_osds": 0, 2023-12-01T22:10:22.971 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "osd_in_since": 0, 2023-12-01T22:10:22.972 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2023-12-01T22:10:22.972 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: }, 2023-12-01T22:10:22.972 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "pgmap": { 2023-12-01T22:10:22.972 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "pgs_by_state": [], 2023-12-01T22:10:22.972 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "num_pgs": 0, 2023-12-01T22:10:22.972 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "num_pools": 0, 2023-12-01T22:10:22.972 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "num_objects": 0, 2023-12-01T22:10:22.972 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "data_bytes": 0, 2023-12-01T22:10:22.972 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "bytes_used": 0, 2023-12-01T22:10:22.973 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "bytes_avail": 0, 2023-12-01T22:10:22.973 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "bytes_total": 0 2023-12-01T22:10:22.973 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: }, 2023-12-01T22:10:22.973 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "fsmap": { 2023-12-01T22:10:22.973 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "epoch": 1, 2023-12-01T22:10:22.973 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "by_rank": [], 2023-12-01T22:10:22.973 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "up:standby": 0 2023-12-01T22:10:22.973 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: }, 2023-12-01T22:10:22.973 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "mgrmap": { 2023-12-01T22:10:22.973 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "available": false, 2023-12-01T22:10:22.973 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "num_standbys": 0, 2023-12-01T22:10:22.974 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "modules": [ 2023-12-01T22:10:22.974 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "iostat", 2023-12-01T22:10:22.974 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "restful" 2023-12-01T22:10:22.974 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: ], 2023-12-01T22:10:22.974 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "services": {} 2023-12-01T22:10:22.974 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: }, 2023-12-01T22:10:22.974 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "servicemap": { 2023-12-01T22:10:22.974 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "epoch": 1, 2023-12-01T22:10:22.974 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "modified": "2023-12-01T22:10:12.142858+0000", 2023-12-01T22:10:22.974 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "services": {} 2023-12-01T22:10:22.974 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: }, 2023-12-01T22:10:22.975 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "progress_events": {} 2023-12-01T22:10:22.975 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: } 2023-12-01T22:10:22.975 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:22.962+0000 7fa9652ea700 1 -- 172.21.15.29:0/2058984965 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa9600703f0 msgr2=0x7fa96010ca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:10:22.975 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:22.962+0000 7fa9652ea700 1 --2- 172.21.15.29:0/2058984965 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa9600703f0 0x7fa96010ca40 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7fa95000b190 tx=0x7fa950006ee0).stop 2023-12-01T22:10:22.975 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:22.962+0000 7fa9652ea700 1 -- 172.21.15.29:0/2058984965 shutdown_connections 2023-12-01T22:10:22.975 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:22.962+0000 7fa9652ea700 1 --2- 172.21.15.29:0/2058984965 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa9600703f0 0x7fa96010ca40 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:10:22.975 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:22.962+0000 7fa9652ea700 1 -- 172.21.15.29:0/2058984965 >> 172.21.15.29:0/2058984965 conn(0x7fa9600f6240 msgr2=0x7fa9600f7cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:10:22.975 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:22.962+0000 7fa9652ea700 1 -- 172.21.15.29:0/2058984965 shutdown_connections 2023-12-01T22:10:22.975 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:22.962+0000 7fa9652ea700 1 -- 172.21.15.29:0/2058984965 wait complete. 2023-12-01T22:10:23.301 INFO:teuthology.orchestra.run.smithi029.stderr:mgr not available, waiting (3/15)... 2023-12-01T22:10:23.776 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:23 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:10:22.962120+0000 mon.smithi029 (mon.0) 22 : audit [DBG] from='client.? 172.21.15.29:0/2058984965' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-12-01T22:10:23.777 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:23 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:10:23.473867+0000 mon.smithi029 (mon.0) 23 : cluster [DBG] mgrmap e3: smithi029.loxrfu(active, since 1.00666s) 2023-12-01T22:10:25.303 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi029 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/5558ab6c-9096-11ee-95a2-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpou9lnrt4:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpxe9edifa:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2023-12-01T22:10:25.698 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:25.694+0000 7f1ba38df700 1 Processor -- start 2023-12-01T22:10:25.698 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:25.695+0000 7f1ba38df700 1 -- start start 2023-12-01T22:10:25.698 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:25.695+0000 7f1ba38df700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b9c0703d0 0x7f1b9c100d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-01T22:10:25.698 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:25.695+0000 7f1ba38df700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b9c0fc730 con 0x7f1b9c0703d0 2023-12-01T22:10:25.698 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:25.695+0000 7f1ba167b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b9c0703d0 0x7f1b9c100d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:10:25.698 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:25.695+0000 7f1ba167b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b9c0703d0 0x7f1b9c100d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48936/0 (socket says 172.21.15.29:48936) 2023-12-01T22:10:25.698 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:25.695+0000 7f1ba167b700 1 -- 172.21.15.29:0/4014681139 learned_addr learned my addr 172.21.15.29:0/4014681139 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-12-01T22:10:25.698 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:25.695+0000 7f1ba167b700 1 -- 172.21.15.29:0/4014681139 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b9c101230 con 0x7f1b9c0703d0 2023-12-01T22:10:25.699 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:25.695+0000 7f1ba167b700 1 --2- 172.21.15.29:0/4014681139 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b9c0703d0 0x7f1b9c100d40 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f1b9800b5f0 tx=0x7f1b98006ee0).ready entity=mon.0 client_cookie=cf2f2f7908f30fa server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:10:25.699 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:25.696+0000 7f1b8bfff700 1 -- 172.21.15.29:0/4014681139 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1b9800d600 con 0x7f1b9c0703d0 2023-12-01T22:10:25.701 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:25.696+0000 7f1b8bfff700 1 -- 172.21.15.29:0/4014681139 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f1b9800dbd0 con 0x7f1b9c0703d0 2023-12-01T22:10:25.702 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:25.696+0000 7f1b8bfff700 1 -- 172.21.15.29:0/4014681139 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1b98011c00 con 0x7f1b9c0703d0 2023-12-01T22:10:25.702 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:25.696+0000 7f1ba38df700 1 -- 172.21.15.29:0/4014681139 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b9c0703d0 msgr2=0x7f1b9c100d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:10:25.702 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:25.696+0000 7f1ba38df700 1 --2- 172.21.15.29:0/4014681139 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b9c0703d0 0x7f1b9c100d40 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f1b9800b5f0 tx=0x7f1b98006ee0).stop 2023-12-01T22:10:25.702 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:25.696+0000 7f1ba38df700 1 -- 172.21.15.29:0/4014681139 shutdown_connections 2023-12-01T22:10:25.702 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:25.696+0000 7f1ba38df700 1 --2- 172.21.15.29:0/4014681139 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b9c0703d0 0x7f1b9c100d40 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:10:25.702 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:25.696+0000 7f1ba38df700 1 -- 172.21.15.29:0/4014681139 >> 172.21.15.29:0/4014681139 conn(0x7f1b9c0f63c0 msgr2=0x7f1b9c0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:10:25.702 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:25.696+0000 7f1ba38df700 1 -- 172.21.15.29:0/4014681139 shutdown_connections 2023-12-01T22:10:25.702 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:25.696+0000 7f1ba38df700 1 -- 172.21.15.29:0/4014681139 wait complete. 2023-12-01T22:10:25.703 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:25.697+0000 7f1ba38df700 1 Processor -- start 2023-12-01T22:10:25.703 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:25.697+0000 7f1ba38df700 1 -- start start 2023-12-01T22:10:25.703 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:25.697+0000 7f1ba38df700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b9c0703d0 0x7f1b9c10cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:10:25.703 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:25.698+0000 7f1ba38df700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b9c10d100 con 0x7f1b9c0703d0 2023-12-01T22:10:25.703 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:25.698+0000 7f1ba167b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b9c0703d0 0x7f1b9c10cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:10:25.703 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:25.698+0000 7f1ba167b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b9c0703d0 0x7f1b9c10cc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48948/0 (socket says 172.21.15.29:48948) 2023-12-01T22:10:25.703 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:25.698+0000 7f1ba167b700 1 -- 172.21.15.29:0/2599181374 learned_addr learned my addr 172.21.15.29:0/2599181374 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-12-01T22:10:25.703 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:25.698+0000 7f1ba167b700 1 -- 172.21.15.29:0/2599181374 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b9800b200 con 0x7f1b9c0703d0 2023-12-01T22:10:25.703 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:25.698+0000 7f1ba167b700 1 --2- 172.21.15.29:0/2599181374 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b9c0703d0 0x7f1b9c10cc10 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f1b9800e040 tx=0x7f1b98006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:10:25.703 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:25.698+0000 7f1b89ffb700 1 -- 172.21.15.29:0/2599181374 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1b9800d9f0 con 0x7f1b9c0703d0 2023-12-01T22:10:25.703 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:25.698+0000 7f1b89ffb700 1 -- 172.21.15.29:0/2599181374 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f1b9801a9f0 con 0x7f1b9c0703d0 2023-12-01T22:10:25.703 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:25.698+0000 7f1ba38df700 1 -- 172.21.15.29:0/2599181374 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b9c10d300 con 0x7f1b9c0703d0 2023-12-01T22:10:25.703 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:25.698+0000 7f1b89ffb700 1 -- 172.21.15.29:0/2599181374 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1b9801b8c0 con 0x7f1b9c0703d0 2023-12-01T22:10:25.704 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:25.699+0000 7f1ba38df700 1 -- 172.21.15.29:0/2599181374 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b9c10d710 con 0x7f1b9c0703d0 2023-12-01T22:10:25.704 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:25.699+0000 7f1b89ffb700 1 -- 172.21.15.29:0/2599181374 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 4) v1 ==== 36109+0+0 (secure 0 0 0) 0x7f1b9800f070 con 0x7f1b9c0703d0 2023-12-01T22:10:25.704 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:25.700+0000 7f1b89ffb700 1 --2- 172.21.15.29:0/2599181374 >> [v2:172.21.15.29:6800/204751073,v1:172.21.15.29:6801/204751073] conn(0x7f1b8402fd50 0x7f1b84032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:10:25.704 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:25.700+0000 7f1b89ffb700 1 -- 172.21.15.29:0/2599181374 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f1b98042000 con 0x7f1b9c0703d0 2023-12-01T22:10:25.704 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:25.700+0000 7f1ba0e7a700 1 --2- 172.21.15.29:0/2599181374 >> [v2:172.21.15.29:6800/204751073,v1:172.21.15.29:6801/204751073] conn(0x7f1b8402fd50 0x7f1b84032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:10:25.704 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:25.700+0000 7f1ba38df700 1 -- 172.21.15.29:0/2599181374 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b9c059100 con 0x7f1b9c0703d0 2023-12-01T22:10:25.704 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:25.700+0000 7f1ba0e7a700 1 --2- 172.21.15.29:0/2599181374 >> [v2:172.21.15.29:6800/204751073,v1:172.21.15.29:6801/204751073] conn(0x7f1b8402fd50 0x7f1b84032210 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f1b8c0096a0 tx=0x7f1b8c006b40).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:10:25.706 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:25.705+0000 7f1b89ffb700 1 -- 172.21.15.29:0/2599181374 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1b98016070 con 0x7f1b9c0703d0 2023-12-01T22:10:25.776 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:25 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:10:24.522071+0000 mon.smithi029 (mon.0) 24 : cluster [DBG] mgrmap e4: smithi029.loxrfu(active, since 2s) 2023-12-01T22:10:25.939 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:25.931+0000 7f1ba38df700 1 -- 172.21.15.29:0/2599181374 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f1b9c045ad0 con 0x7f1b9c0703d0 2023-12-01T22:10:25.940 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:25.931+0000 7f1b89ffb700 1 -- 172.21.15.29:0/2599181374 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7f1b9801bcb0 con 0x7f1b9c0703d0 2023-12-01T22:10:25.940 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:25.940 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: { 2023-12-01T22:10:25.940 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "fsid": "5558ab6c-9096-11ee-95a2-87774f69a715", 2023-12-01T22:10:25.940 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "health": { 2023-12-01T22:10:25.940 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2023-12-01T22:10:25.940 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "checks": {}, 2023-12-01T22:10:25.941 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "mutes": [] 2023-12-01T22:10:25.941 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: }, 2023-12-01T22:10:25.941 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "election_epoch": 5, 2023-12-01T22:10:25.941 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "quorum": [ 2023-12-01T22:10:25.941 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 0 2023-12-01T22:10:25.941 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: ], 2023-12-01T22:10:25.941 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "quorum_names": [ 2023-12-01T22:10:25.941 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "smithi029" 2023-12-01T22:10:25.941 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: ], 2023-12-01T22:10:25.941 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "quorum_age": 10, 2023-12-01T22:10:25.941 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "monmap": { 2023-12-01T22:10:25.942 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "epoch": 1, 2023-12-01T22:10:25.942 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2023-12-01T22:10:25.942 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "num_mons": 1 2023-12-01T22:10:25.942 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: }, 2023-12-01T22:10:25.942 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "osdmap": { 2023-12-01T22:10:25.942 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "epoch": 1, 2023-12-01T22:10:25.942 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "num_osds": 0, 2023-12-01T22:10:25.942 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "num_up_osds": 0, 2023-12-01T22:10:25.942 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "osd_up_since": 0, 2023-12-01T22:10:25.942 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "num_in_osds": 0, 2023-12-01T22:10:25.942 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "osd_in_since": 0, 2023-12-01T22:10:25.942 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2023-12-01T22:10:25.943 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: }, 2023-12-01T22:10:25.943 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "pgmap": { 2023-12-01T22:10:25.943 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "pgs_by_state": [], 2023-12-01T22:10:25.943 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "num_pgs": 0, 2023-12-01T22:10:25.943 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "num_pools": 0, 2023-12-01T22:10:25.943 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "num_objects": 0, 2023-12-01T22:10:25.943 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "data_bytes": 0, 2023-12-01T22:10:25.943 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "bytes_used": 0, 2023-12-01T22:10:25.943 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "bytes_avail": 0, 2023-12-01T22:10:25.943 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "bytes_total": 0 2023-12-01T22:10:25.943 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: }, 2023-12-01T22:10:25.943 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "fsmap": { 2023-12-01T22:10:25.943 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "epoch": 1, 2023-12-01T22:10:25.943 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "by_rank": [], 2023-12-01T22:10:25.944 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "up:standby": 0 2023-12-01T22:10:25.944 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: }, 2023-12-01T22:10:25.944 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "mgrmap": { 2023-12-01T22:10:25.944 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "available": true, 2023-12-01T22:10:25.944 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "num_standbys": 0, 2023-12-01T22:10:25.944 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "modules": [ 2023-12-01T22:10:25.944 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "iostat", 2023-12-01T22:10:25.944 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "restful" 2023-12-01T22:10:25.944 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: ], 2023-12-01T22:10:25.944 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "services": {} 2023-12-01T22:10:25.944 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: }, 2023-12-01T22:10:25.944 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "servicemap": { 2023-12-01T22:10:25.944 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "epoch": 1, 2023-12-01T22:10:25.944 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "modified": "2023-12-01T22:10:12.142858+0000", 2023-12-01T22:10:25.945 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "services": {} 2023-12-01T22:10:25.945 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: }, 2023-12-01T22:10:25.945 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "progress_events": {} 2023-12-01T22:10:25.945 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: } 2023-12-01T22:10:25.945 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:25.933+0000 7f1ba38df700 1 -- 172.21.15.29:0/2599181374 >> [v2:172.21.15.29:6800/204751073,v1:172.21.15.29:6801/204751073] conn(0x7f1b8402fd50 msgr2=0x7f1b84032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:10:25.945 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:25.933+0000 7f1ba38df700 1 --2- 172.21.15.29:0/2599181374 >> [v2:172.21.15.29:6800/204751073,v1:172.21.15.29:6801/204751073] conn(0x7f1b8402fd50 0x7f1b84032210 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f1b8c0096a0 tx=0x7f1b8c006b40).stop 2023-12-01T22:10:25.945 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:25.933+0000 7f1ba38df700 1 -- 172.21.15.29:0/2599181374 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b9c0703d0 msgr2=0x7f1b9c10cc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:10:25.945 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:25.933+0000 7f1ba38df700 1 --2- 172.21.15.29:0/2599181374 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b9c0703d0 0x7f1b9c10cc10 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f1b9800e040 tx=0x7f1b98006ee0).stop 2023-12-01T22:10:25.945 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:25.933+0000 7f1ba38df700 1 -- 172.21.15.29:0/2599181374 shutdown_connections 2023-12-01T22:10:25.945 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:25.933+0000 7f1ba38df700 1 --2- 172.21.15.29:0/2599181374 >> [v2:172.21.15.29:6800/204751073,v1:172.21.15.29:6801/204751073] conn(0x7f1b8402fd50 0x7f1b84032210 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:10:25.945 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:25.933+0000 7f1ba38df700 1 --2- 172.21.15.29:0/2599181374 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b9c0703d0 0x7f1b9c10cc10 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:10:25.945 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:25.933+0000 7f1ba38df700 1 -- 172.21.15.29:0/2599181374 >> 172.21.15.29:0/2599181374 conn(0x7f1b9c0f63c0 msgr2=0x7f1b9c0f7ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:10:25.945 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:25.933+0000 7f1ba38df700 1 -- 172.21.15.29:0/2599181374 shutdown_connections 2023-12-01T22:10:25.945 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:25.933+0000 7f1ba38df700 1 -- 172.21.15.29:0/2599181374 wait complete. 2023-12-01T22:10:26.293 INFO:teuthology.orchestra.run.smithi029.stderr:mgr is available 2023-12-01T22:10:26.293 INFO:teuthology.orchestra.run.smithi029.stderr:Enabling cephadm module... 2023-12-01T22:10:26.293 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi029 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/5558ab6c-9096-11ee-95a2-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpou9lnrt4:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpxe9edifa:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable cephadm 2023-12-01T22:10:26.655 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:26.652+0000 7febead4a700 1 Processor -- start 2023-12-01T22:10:26.655 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:26.652+0000 7febead4a700 1 -- start start 2023-12-01T22:10:26.655 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:26.652+0000 7febead4a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7febe40703b0 0x7febe4100d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-01T22:10:26.655 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:26.652+0000 7febead4a700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febe40fc750 con 0x7febe40703b0 2023-12-01T22:10:26.655 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:26.652+0000 7febe8ae6700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7febe40703b0 0x7febe4100d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:10:26.655 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:26.652+0000 7febe8ae6700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7febe40703b0 0x7febe4100d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48964/0 (socket says 172.21.15.29:48964) 2023-12-01T22:10:26.655 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:26.652+0000 7febe8ae6700 1 -- 172.21.15.29:0/3407445403 learned_addr learned my addr 172.21.15.29:0/3407445403 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-12-01T22:10:26.655 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:26.652+0000 7febe8ae6700 1 -- 172.21.15.29:0/3407445403 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febe4101250 con 0x7febe40703b0 2023-12-01T22:10:26.655 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:26.652+0000 7febe8ae6700 1 --2- 172.21.15.29:0/3407445403 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7febe40703b0 0x7febe4100d60 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7febd400b5f0 tx=0x7febd4006ee0).ready entity=mon.0 client_cookie=d3c6012689d67258 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:10:26.655 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:26.653+0000 7febe37fe700 1 -- 172.21.15.29:0/3407445403 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7febd400d600 con 0x7febe40703b0 2023-12-01T22:10:26.655 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:26.653+0000 7febe37fe700 1 -- 172.21.15.29:0/3407445403 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7febd400dbd0 con 0x7febe40703b0 2023-12-01T22:10:26.655 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:26.653+0000 7febe37fe700 1 -- 172.21.15.29:0/3407445403 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7febd4011c00 con 0x7febe40703b0 2023-12-01T22:10:26.660 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:26.653+0000 7febead4a700 1 -- 172.21.15.29:0/3407445403 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7febe40703b0 msgr2=0x7febe4100d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:10:26.660 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:26.653+0000 7febead4a700 1 --2- 172.21.15.29:0/3407445403 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7febe40703b0 0x7febe4100d60 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7febd400b5f0 tx=0x7febd4006ee0).stop 2023-12-01T22:10:26.660 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:26.653+0000 7febead4a700 1 -- 172.21.15.29:0/3407445403 shutdown_connections 2023-12-01T22:10:26.660 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:26.653+0000 7febead4a700 1 --2- 172.21.15.29:0/3407445403 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7febe40703b0 0x7febe4100d60 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:10:26.660 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:26.653+0000 7febead4a700 1 -- 172.21.15.29:0/3407445403 >> 172.21.15.29:0/3407445403 conn(0x7febe40f63c0 msgr2=0x7febe40f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:10:26.660 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:26.654+0000 7febead4a700 1 -- 172.21.15.29:0/3407445403 shutdown_connections 2023-12-01T22:10:26.660 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:26.654+0000 7febead4a700 1 -- 172.21.15.29:0/3407445403 wait complete. 2023-12-01T22:10:26.660 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:26.654+0000 7febead4a700 1 Processor -- start 2023-12-01T22:10:26.660 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:26.654+0000 7febead4a700 1 -- start start 2023-12-01T22:10:26.660 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:26.654+0000 7febead4a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7febe40703b0 0x7febe410ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:10:26.661 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:26.654+0000 7febead4a700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febe410d1e0 con 0x7febe40703b0 2023-12-01T22:10:26.661 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:26.655+0000 7febe8ae6700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7febe40703b0 0x7febe410ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:10:26.661 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:26.655+0000 7febe8ae6700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7febe40703b0 0x7febe410ccf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48980/0 (socket says 172.21.15.29:48980) 2023-12-01T22:10:26.661 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:26.655+0000 7febe8ae6700 1 -- 172.21.15.29:0/2853971141 learned_addr learned my addr 172.21.15.29:0/2853971141 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-12-01T22:10:26.661 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:26.655+0000 7febe8ae6700 1 -- 172.21.15.29:0/2853971141 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febd400b200 con 0x7febe40703b0 2023-12-01T22:10:26.662 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:26.655+0000 7febe8ae6700 1 --2- 172.21.15.29:0/2853971141 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7febe40703b0 0x7febe410ccf0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7febd400e040 tx=0x7febd4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:10:26.662 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:26.655+0000 7febe17fa700 1 -- 172.21.15.29:0/2853971141 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7febd400d9f0 con 0x7febe40703b0 2023-12-01T22:10:26.662 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:26.655+0000 7febe17fa700 1 -- 172.21.15.29:0/2853971141 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7febd401a9f0 con 0x7febe40703b0 2023-12-01T22:10:26.662 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:26.655+0000 7febe17fa700 1 -- 172.21.15.29:0/2853971141 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7febd401b8c0 con 0x7febe40703b0 2023-12-01T22:10:26.662 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:26.655+0000 7febead4a700 1 -- 172.21.15.29:0/2853971141 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febe410d3e0 con 0x7febe40703b0 2023-12-01T22:10:26.662 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:26.655+0000 7febead4a700 1 -- 172.21.15.29:0/2853971141 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febe410d7f0 con 0x7febe40703b0 2023-12-01T22:10:26.662 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:26.656+0000 7febe17fa700 1 -- 172.21.15.29:0/2853971141 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 4) v1 ==== 36109+0+0 (secure 0 0 0) 0x7febd400f070 con 0x7febe40703b0 2023-12-01T22:10:26.662 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:26.656+0000 7febead4a700 1 -- 172.21.15.29:0/2853971141 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febe4045ad0 con 0x7febe40703b0 2023-12-01T22:10:26.662 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:26.656+0000 7febe17fa700 1 --2- 172.21.15.29:0/2853971141 >> [v2:172.21.15.29:6800/204751073,v1:172.21.15.29:6801/204751073] conn(0x7febd002fd00 0x7febd00321c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:10:26.663 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:26.657+0000 7febe17fa700 1 -- 172.21.15.29:0/2853971141 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7febd40412e0 con 0x7febe40703b0 2023-12-01T22:10:26.663 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:26.657+0000 7febe3fff700 1 --2- 172.21.15.29:0/2853971141 >> [v2:172.21.15.29:6800/204751073,v1:172.21.15.29:6801/204751073] conn(0x7febd002fd00 0x7febd00321c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:10:26.663 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:26.657+0000 7febe3fff700 1 --2- 172.21.15.29:0/2853971141 >> [v2:172.21.15.29:6800/204751073,v1:172.21.15.29:6801/204751073] conn(0x7febd002fd00 0x7febd00321c0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7febd80096a0 tx=0x7febd8006b40).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:10:26.663 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:26.661+0000 7febe17fa700 1 -- 172.21.15.29:0/2853971141 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7febd40224f0 con 0x7febe40703b0 2023-12-01T22:10:26.776 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:26 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:10:25.932424+0000 mon.smithi029 (mon.0) 25 : audit [DBG] from='client.? 172.21.15.29:0/2599181374' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-12-01T22:10:26.870 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:26.868+0000 7febead4a700 1 -- 172.21.15.29:0/2853971141 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7febe4101db0 con 0x7febe40703b0 2023-12-01T22:10:27.526 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:27.524+0000 7febe17fa700 1 -- 172.21.15.29:0/2853971141 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 5) v1 ==== 36120+0+0 (secure 0 0 0) 0x7febd4022d80 con 0x7febe40703b0 2023-12-01T22:10:27.527 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:27.525+0000 7febe17fa700 1 -- 172.21.15.29:0/2853971141 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7febd4022310 con 0x7febe40703b0 2023-12-01T22:10:27.530 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:27.527+0000 7febead4a700 1 -- 172.21.15.29:0/2853971141 >> [v2:172.21.15.29:6800/204751073,v1:172.21.15.29:6801/204751073] conn(0x7febd002fd00 msgr2=0x7febd00321c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:10:27.530 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:27.527+0000 7febead4a700 1 --2- 172.21.15.29:0/2853971141 >> [v2:172.21.15.29:6800/204751073,v1:172.21.15.29:6801/204751073] conn(0x7febd002fd00 0x7febd00321c0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7febd80096a0 tx=0x7febd8006b40).stop 2023-12-01T22:10:27.530 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:27.527+0000 7febead4a700 1 -- 172.21.15.29:0/2853971141 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7febe40703b0 msgr2=0x7febe410ccf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:10:27.531 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:27.527+0000 7febead4a700 1 --2- 172.21.15.29:0/2853971141 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7febe40703b0 0x7febe410ccf0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7febd400e040 tx=0x7febd4006ee0).stop 2023-12-01T22:10:27.531 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:27.528+0000 7febead4a700 1 -- 172.21.15.29:0/2853971141 shutdown_connections 2023-12-01T22:10:27.531 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:27.528+0000 7febead4a700 1 --2- 172.21.15.29:0/2853971141 >> [v2:172.21.15.29:6800/204751073,v1:172.21.15.29:6801/204751073] conn(0x7febd002fd00 0x7febd00321c0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:10:27.531 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:27.528+0000 7febead4a700 1 --2- 172.21.15.29:0/2853971141 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7febe40703b0 0x7febe410ccf0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:10:27.531 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:27.528+0000 7febead4a700 1 -- 172.21.15.29:0/2853971141 >> 172.21.15.29:0/2853971141 conn(0x7febe40f63c0 msgr2=0x7febe40f7f50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:10:27.531 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:27.529+0000 7febead4a700 1 -- 172.21.15.29:0/2853971141 shutdown_connections 2023-12-01T22:10:27.531 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:27.529+0000 7febead4a700 1 -- 172.21.15.29:0/2853971141 wait complete. 2023-12-01T22:10:27.776 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:27 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:10:26.869876+0000 mon.smithi029 (mon.0) 26 : audit [INF] from='client.? 172.21.15.29:0/2853971141' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2023-12-01T22:10:27.883 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi029 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/5558ab6c-9096-11ee-95a2-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpou9lnrt4:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpxe9edifa:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr stat 2023-12-01T22:10:28.776 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:28 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:10:27.526484+0000 mon.smithi029 (mon.0) 27 : audit [INF] from='client.? 172.21.15.29:0/2853971141' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2023-12-01T22:10:28.776 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:28 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:10:27.526546+0000 mon.smithi029 (mon.0) 28 : cluster [DBG] mgrmap e5: smithi029.loxrfu(active, since 5s) 2023-12-01T22:10:28.923 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:28.754+0000 7f9bdfc3e700 1 Processor -- start 2023-12-01T22:10:28.923 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:28.754+0000 7f9bdfc3e700 1 -- start start 2023-12-01T22:10:28.923 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:28.754+0000 7f9bdfc3e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9bd8070430 0x7f9bd80fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-01T22:10:28.923 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:28.754+0000 7f9bdfc3e700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bd80fedd0 con 0x7f9bd8070430 2023-12-01T22:10:28.924 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:28.755+0000 7f9bdd9da700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9bd8070430 0x7f9bd80fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:10:28.924 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:28.755+0000 7f9bdd9da700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9bd8070430 0x7f9bd80fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49016/0 (socket says 172.21.15.29:49016) 2023-12-01T22:10:28.924 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:28.755+0000 7f9bdd9da700 1 -- 172.21.15.29:0/3976361224 learned_addr learned my addr 172.21.15.29:0/3976361224 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-12-01T22:10:28.924 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:28.755+0000 7f9bdd9da700 1 -- 172.21.15.29:0/3976361224 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bd80fef10 con 0x7f9bd8070430 2023-12-01T22:10:28.924 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:28.755+0000 7f9bdd9da700 1 --2- 172.21.15.29:0/3976361224 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9bd8070430 0x7f9bd80fe8e0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f9bd400b5f0 tx=0x7f9bd4006ee0).ready entity=mon.0 client_cookie=dedd38dda00f7f44 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:10:28.924 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:28.755+0000 7f9bdc9d8700 1 -- 172.21.15.29:0/3976361224 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9bd400d600 con 0x7f9bd8070430 2023-12-01T22:10:28.924 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:28.755+0000 7f9bdc9d8700 1 -- 172.21.15.29:0/3976361224 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f9bd400dbd0 con 0x7f9bd8070430 2023-12-01T22:10:28.925 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:28.755+0000 7f9bdc9d8700 1 -- 172.21.15.29:0/3976361224 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9bd4011c00 con 0x7f9bd8070430 2023-12-01T22:10:28.925 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:28.756+0000 7f9bdfc3e700 1 -- 172.21.15.29:0/3976361224 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9bd8070430 msgr2=0x7f9bd80fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:10:28.926 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:28.756+0000 7f9bdfc3e700 1 --2- 172.21.15.29:0/3976361224 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9bd8070430 0x7f9bd80fe8e0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f9bd400b5f0 tx=0x7f9bd4006ee0).stop 2023-12-01T22:10:28.926 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:28.756+0000 7f9bdfc3e700 1 -- 172.21.15.29:0/3976361224 shutdown_connections 2023-12-01T22:10:28.926 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:28.756+0000 7f9bdfc3e700 1 --2- 172.21.15.29:0/3976361224 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9bd8070430 0x7f9bd80fe8e0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:10:28.926 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:28.756+0000 7f9bdfc3e700 1 -- 172.21.15.29:0/3976361224 >> 172.21.15.29:0/3976361224 conn(0x7f9bd80f63c0 msgr2=0x7f9bd80f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:10:28.927 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:28.756+0000 7f9bdfc3e700 1 -- 172.21.15.29:0/3976361224 shutdown_connections 2023-12-01T22:10:28.927 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:28.756+0000 7f9bdfc3e700 1 -- 172.21.15.29:0/3976361224 wait complete. 2023-12-01T22:10:28.927 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:28.757+0000 7f9bdfc3e700 1 Processor -- start 2023-12-01T22:10:28.927 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:28.757+0000 7f9bdfc3e700 1 -- start start 2023-12-01T22:10:28.927 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:28.757+0000 7f9bdfc3e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9bd8070430 0x7f9bd810ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:10:28.927 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:28.757+0000 7f9bdfc3e700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bd810d1c0 con 0x7f9bd8070430 2023-12-01T22:10:28.927 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:28.757+0000 7f9bdd9da700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9bd8070430 0x7f9bd810ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:10:28.927 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:28.757+0000 7f9bdd9da700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9bd8070430 0x7f9bd810ccd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49028/0 (socket says 172.21.15.29:49028) 2023-12-01T22:10:28.927 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:28.757+0000 7f9bdd9da700 1 -- 172.21.15.29:0/3275053941 learned_addr learned my addr 172.21.15.29:0/3275053941 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-12-01T22:10:28.927 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:28.758+0000 7f9bdd9da700 1 -- 172.21.15.29:0/3275053941 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bd400b200 con 0x7f9bd8070430 2023-12-01T22:10:28.927 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:28.758+0000 7f9bdd9da700 1 --2- 172.21.15.29:0/3275053941 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9bd8070430 0x7f9bd810ccd0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f9bd400e040 tx=0x7f9bd4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:10:28.928 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:28.758+0000 7f9bc67fc700 1 -- 172.21.15.29:0/3275053941 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9bd400d9f0 con 0x7f9bd8070430 2023-12-01T22:10:28.929 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:28.758+0000 7f9bc67fc700 1 -- 172.21.15.29:0/3275053941 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f9bd401a9f0 con 0x7f9bd8070430 2023-12-01T22:10:28.929 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:28.758+0000 7f9bdfc3e700 1 -- 172.21.15.29:0/3275053941 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bd810d3c0 con 0x7f9bd8070430 2023-12-01T22:10:28.929 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:28.758+0000 7f9bc67fc700 1 -- 172.21.15.29:0/3275053941 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9bd401b8c0 con 0x7f9bd8070430 2023-12-01T22:10:28.929 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:28.758+0000 7f9bdfc3e700 1 -- 172.21.15.29:0/3275053941 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bd810d7d0 con 0x7f9bd8070430 2023-12-01T22:10:28.929 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:28.759+0000 7f9bc67fc700 1 -- 172.21.15.29:0/3275053941 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f9bd400f070 con 0x7f9bd8070430 2023-12-01T22:10:28.929 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:28.759+0000 7f9bc67fc700 1 --2- 172.21.15.29:0/3275053941 >> [v2:172.21.15.29:6800/204751073,v1:172.21.15.29:6801/204751073] conn(0x7f9bc002fd50 0x7f9bc0032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:10:28.929 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:28.759+0000 7f9bdfc3e700 1 -- 172.21.15.29:0/3275053941 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bd8045ad0 con 0x7f9bd8070430 2023-12-01T22:10:28.929 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:28.759+0000 7f9bc67fc700 1 -- 172.21.15.29:0/3275053941 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f9bd4041230 con 0x7f9bd8070430 2023-12-01T22:10:28.929 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:28.759+0000 7f9bdd1d9700 1 -- 172.21.15.29:0/3275053941 >> [v2:172.21.15.29:6800/204751073,v1:172.21.15.29:6801/204751073] conn(0x7f9bc002fd50 msgr2=0x7f9bc0032210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/204751073 2023-12-01T22:10:28.929 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:28.759+0000 7f9bdd1d9700 1 --2- 172.21.15.29:0/3275053941 >> [v2:172.21.15.29:6800/204751073,v1:172.21.15.29:6801/204751073] conn(0x7f9bc002fd50 0x7f9bc0032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-12-01T22:10:28.929 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:28.764+0000 7f9bc67fc700 1 -- 172.21.15.29:0/3275053941 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9bd4016080 con 0x7f9bd8070430 2023-12-01T22:10:28.962 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:28.960+0000 7f9bdd1d9700 1 -- 172.21.15.29:0/3275053941 >> [v2:172.21.15.29:6800/204751073,v1:172.21.15.29:6801/204751073] conn(0x7f9bc002fd50 msgr2=0x7f9bc0032210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/204751073 2023-12-01T22:10:28.962 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:28.960+0000 7f9bdd1d9700 1 --2- 172.21.15.29:0/3275053941 >> [v2:172.21.15.29:6800/204751073,v1:172.21.15.29:6801/204751073] conn(0x7f9bc002fd50 0x7f9bc0032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-12-01T22:10:29.024 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:29.021+0000 7f9bdfc3e700 1 -- 172.21.15.29:0/3275053941 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f9bd810e670 con 0x7f9bd8070430 2023-12-01T22:10:29.024 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:29.022+0000 7f9bc67fc700 1 -- 172.21.15.29:0/3275053941 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+103 (secure 0 0 0) 0x7f9bd4022d20 con 0x7f9bd8070430 2023-12-01T22:10:29.024 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: { 2023-12-01T22:10:29.024 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "epoch": 5, 2023-12-01T22:10:29.024 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "available": true, 2023-12-01T22:10:29.024 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "active_name": "smithi029.loxrfu", 2023-12-01T22:10:29.024 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "num_standby": 0 2023-12-01T22:10:29.025 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: } 2023-12-01T22:10:29.026 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:29.024+0000 7f9bdfc3e700 1 -- 172.21.15.29:0/3275053941 >> [v2:172.21.15.29:6800/204751073,v1:172.21.15.29:6801/204751073] conn(0x7f9bc002fd50 msgr2=0x7f9bc0032210 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-01T22:10:29.026 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:29.024+0000 7f9bdfc3e700 1 --2- 172.21.15.29:0/3275053941 >> [v2:172.21.15.29:6800/204751073,v1:172.21.15.29:6801/204751073] conn(0x7f9bc002fd50 0x7f9bc0032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-12-01T22:10:29.026 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:29.024+0000 7f9bdfc3e700 1 -- 172.21.15.29:0/3275053941 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9bd8070430 msgr2=0x7f9bd810ccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:10:29.026 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:29.024+0000 7f9bdfc3e700 1 --2- 172.21.15.29:0/3275053941 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9bd8070430 0x7f9bd810ccd0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f9bd400e040 tx=0x7f9bd4006ee0).stop 2023-12-01T22:10:29.026 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:29.024+0000 7f9bdfc3e700 1 -- 172.21.15.29:0/3275053941 shutdown_connections 2023-12-01T22:10:29.027 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:29.024+0000 7f9bdfc3e700 1 --2- 172.21.15.29:0/3275053941 >> [v2:172.21.15.29:6800/204751073,v1:172.21.15.29:6801/204751073] conn(0x7f9bc002fd50 0x7f9bc0032210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-12-01T22:10:29.027 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:29.024+0000 7f9bdfc3e700 1 --2- 172.21.15.29:0/3275053941 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9bd8070430 0x7f9bd810ccd0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:10:29.027 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:29.024+0000 7f9bdfc3e700 1 -- 172.21.15.29:0/3275053941 >> 172.21.15.29:0/3275053941 conn(0x7f9bd80f63c0 msgr2=0x7f9bd80f7f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:10:29.027 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:29.025+0000 7f9bdfc3e700 1 -- 172.21.15.29:0/3275053941 shutdown_connections 2023-12-01T22:10:29.027 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:29.025+0000 7f9bdfc3e700 1 -- 172.21.15.29:0/3275053941 wait complete. 2023-12-01T22:10:29.776 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:29 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:10:29.023038+0000 mon.smithi029 (mon.0) 29 : audit [DBG] from='client.? 172.21.15.29:0/3275053941' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-12-01T22:10:31.166 INFO:teuthology.orchestra.run.smithi029.stderr:Waiting for the mgr to restart... 2023-12-01T22:10:31.167 INFO:teuthology.orchestra.run.smithi029.stderr:Waiting for mgr epoch 5... 2023-12-01T22:10:31.167 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi029 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/5558ab6c-9096-11ee-95a2-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpou9lnrt4:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpxe9edifa:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 tell mgr mgr_status 2023-12-01T22:10:32.500 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:32.290+0000 7f63bdb80700 1 Processor -- start 2023-12-01T22:10:32.500 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:32.291+0000 7f63bdb80700 1 -- start start 2023-12-01T22:10:32.500 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:32.291+0000 7f63bdb80700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63b80703f0 0x7f63b8100d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-01T22:10:32.501 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:32.291+0000 7f63bdb80700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63b80fc710 con 0x7f63b80703f0 2023-12-01T22:10:32.501 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:32.291+0000 7f63b77fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63b80703f0 0x7f63b8100d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:10:32.501 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:32.291+0000 7f63b77fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63b80703f0 0x7f63b8100d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49044/0 (socket says 172.21.15.29:49044) 2023-12-01T22:10:32.501 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:32.291+0000 7f63b77fe700 1 -- 172.21.15.29:0/4061031296 learned_addr learned my addr 172.21.15.29:0/4061031296 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-12-01T22:10:32.501 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:32.291+0000 7f63b77fe700 1 -- 172.21.15.29:0/4061031296 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63b8101210 con 0x7f63b80703f0 2023-12-01T22:10:32.501 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:32.291+0000 7f63b77fe700 1 --2- 172.21.15.29:0/4061031296 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63b80703f0 0x7f63b8100d20 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7f63a800b5f0 tx=0x7f63a8006ee0).ready entity=mon.0 client_cookie=e280c7765f4ec494 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:10:32.501 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:32.291+0000 7f63b67fc700 1 -- 172.21.15.29:0/4061031296 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f63a800d600 con 0x7f63b80703f0 2023-12-01T22:10:32.501 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:32.292+0000 7f63b67fc700 1 -- 172.21.15.29:0/4061031296 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f63a800dbd0 con 0x7f63b80703f0 2023-12-01T22:10:32.501 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:32.292+0000 7f63b67fc700 1 -- 172.21.15.29:0/4061031296 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f63a8011c00 con 0x7f63b80703f0 2023-12-01T22:10:32.501 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:32.292+0000 7f63bdb80700 1 -- 172.21.15.29:0/4061031296 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63b80703f0 msgr2=0x7f63b8100d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:10:32.501 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:32.292+0000 7f63bdb80700 1 --2- 172.21.15.29:0/4061031296 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63b80703f0 0x7f63b8100d20 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7f63a800b5f0 tx=0x7f63a8006ee0).stop 2023-12-01T22:10:32.501 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:32.292+0000 7f63bdb80700 1 -- 172.21.15.29:0/4061031296 shutdown_connections 2023-12-01T22:10:32.501 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:32.292+0000 7f63bdb80700 1 --2- 172.21.15.29:0/4061031296 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63b80703f0 0x7f63b8100d20 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:10:32.501 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:32.292+0000 7f63bdb80700 1 -- 172.21.15.29:0/4061031296 >> 172.21.15.29:0/4061031296 conn(0x7f63b80f63c0 msgr2=0x7f63b80f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:10:32.501 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:32.293+0000 7f63bdb80700 1 -- 172.21.15.29:0/4061031296 shutdown_connections 2023-12-01T22:10:32.502 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:32.293+0000 7f63bdb80700 1 -- 172.21.15.29:0/4061031296 wait complete. 2023-12-01T22:10:32.502 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:32.293+0000 7f63bdb80700 1 Processor -- start 2023-12-01T22:10:32.502 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:32.294+0000 7f63bdb80700 1 -- start start 2023-12-01T22:10:32.502 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:32.294+0000 7f63bdb80700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63b80703f0 0x7f63b810ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:10:32.502 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:32.294+0000 7f63bdb80700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63b810d1e0 con 0x7f63b80703f0 2023-12-01T22:10:32.502 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:32.294+0000 7f63b77fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63b80703f0 0x7f63b810ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:10:32.502 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:32.294+0000 7f63b77fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63b80703f0 0x7f63b810ccf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49050/0 (socket says 172.21.15.29:49050) 2023-12-01T22:10:32.502 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:32.294+0000 7f63b77fe700 1 -- 172.21.15.29:0/1746657493 learned_addr learned my addr 172.21.15.29:0/1746657493 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-12-01T22:10:32.502 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:32.294+0000 7f63b77fe700 1 -- 172.21.15.29:0/1746657493 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63a800b200 con 0x7f63b80703f0 2023-12-01T22:10:32.502 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:32.294+0000 7f63b77fe700 1 --2- 172.21.15.29:0/1746657493 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63b80703f0 0x7f63b810ccf0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f63a800e040 tx=0x7f63a8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:10:32.502 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:32.295+0000 7f639ffff700 1 -- 172.21.15.29:0/1746657493 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f63a800d9f0 con 0x7f63b80703f0 2023-12-01T22:10:32.502 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:32.295+0000 7f639ffff700 1 -- 172.21.15.29:0/1746657493 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f63a801a9f0 con 0x7f63b80703f0 2023-12-01T22:10:32.502 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:32.295+0000 7f639ffff700 1 -- 172.21.15.29:0/1746657493 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f63a801b8c0 con 0x7f63b80703f0 2023-12-01T22:10:32.503 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:32.295+0000 7f63bdb80700 1 -- 172.21.15.29:0/1746657493 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63b810d3e0 con 0x7f63b80703f0 2023-12-01T22:10:32.503 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:32.295+0000 7f63bdb80700 1 -- 172.21.15.29:0/1746657493 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63b810d7f0 con 0x7f63b80703f0 2023-12-01T22:10:32.503 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:32.296+0000 7f639ffff700 1 -- 172.21.15.29:0/1746657493 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f63a800f070 con 0x7f63b80703f0 2023-12-01T22:10:32.503 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:32.296+0000 7f639ffff700 1 --2- 172.21.15.29:0/1746657493 >> [v2:172.21.15.29:6800/204751073,v1:172.21.15.29:6801/204751073] conn(0x7f63a402fdb0 0x7f63a4032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:10:32.503 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:32.296+0000 7f63bdb80700 1 -- 172.21.15.29:0/1746657493 --> [v2:172.21.15.29:6800/204751073,v1:172.21.15.29:6801/204751073] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f63b8059100 con 0x7f63a402fdb0 2023-12-01T22:10:32.503 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:32.296+0000 7f639ffff700 1 -- 172.21.15.29:0/1746657493 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f63a8042020 con 0x7f63b80703f0 2023-12-01T22:10:32.503 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:32.296+0000 7f63b6ffd700 1 -- 172.21.15.29:0/1746657493 >> [v2:172.21.15.29:6800/204751073,v1:172.21.15.29:6801/204751073] conn(0x7f63a402fdb0 msgr2=0x7f63a4032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/204751073 2023-12-01T22:10:32.503 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:32.296+0000 7f63b6ffd700 1 --2- 172.21.15.29:0/1746657493 >> [v2:172.21.15.29:6800/204751073,v1:172.21.15.29:6801/204751073] conn(0x7f63a402fdb0 0x7f63a4032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-12-01T22:10:32.503 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:32.497+0000 7f63b6ffd700 1 -- 172.21.15.29:0/1746657493 >> [v2:172.21.15.29:6800/204751073,v1:172.21.15.29:6801/204751073] conn(0x7f63a402fdb0 msgr2=0x7f63a4032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/204751073 2023-12-01T22:10:32.503 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:32.497+0000 7f63b6ffd700 1 --2- 172.21.15.29:0/1746657493 >> [v2:172.21.15.29:6800/204751073,v1:172.21.15.29:6801/204751073] conn(0x7f63a402fdb0 0x7f63a4032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-12-01T22:10:32.900 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:32.898+0000 7f63b6ffd700 1 -- 172.21.15.29:0/1746657493 >> [v2:172.21.15.29:6800/204751073,v1:172.21.15.29:6801/204751073] conn(0x7f63a402fdb0 msgr2=0x7f63a4032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/204751073 2023-12-01T22:10:32.900 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:32.898+0000 7f63b6ffd700 1 --2- 172.21.15.29:0/1746657493 >> [v2:172.21.15.29:6800/204751073,v1:172.21.15.29:6801/204751073] conn(0x7f63a402fdb0 0x7f63a4032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-12-01T22:10:33.172 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:33.170+0000 7f639ffff700 1 -- 172.21.15.29:0/1746657493 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mgrmap(e 6) v1 ==== 35970+0+0 (secure 0 0 0) 0x7f63a80257e0 con 0x7f63b80703f0 2023-12-01T22:10:33.172 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:33.170+0000 7f639ffff700 1 -- 172.21.15.29:0/1746657493 >> [v2:172.21.15.29:6800/204751073,v1:172.21.15.29:6801/204751073] conn(0x7f63a402fdb0 msgr2=0x7f63a4032270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-01T22:10:33.172 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:33.170+0000 7f639ffff700 1 --2- 172.21.15.29:0/1746657493 >> [v2:172.21.15.29:6800/204751073,v1:172.21.15.29:6801/204751073] conn(0x7f63a402fdb0 0x7f63a4032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-12-01T22:10:34.174 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:34.171+0000 7f639ffff700 1 -- 172.21.15.29:0/1746657493 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 7) v1 ==== 36040+0+0 (secure 0 0 0) 0x7f63a801ab60 con 0x7f63b80703f0 2023-12-01T22:10:34.174 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:34.172+0000 7f639ffff700 1 --2- 172.21.15.29:0/1746657493 >> [v2:172.21.15.29:6800/3122114772,v1:172.21.15.29:6801/3122114772] conn(0x7f63a40336d0 0x7f63a4035ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:10:34.174 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:34.172+0000 7f639ffff700 1 -- 172.21.15.29:0/1746657493 --> [v2:172.21.15.29:6800/3122114772,v1:172.21.15.29:6801/3122114772] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f63b8059100 con 0x7f63a40336d0 2023-12-01T22:10:34.174 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:34.172+0000 7f63b6ffd700 1 --2- 172.21.15.29:0/1746657493 >> [v2:172.21.15.29:6800/3122114772,v1:172.21.15.29:6801/3122114772] conn(0x7f63a40336d0 0x7f63a4035ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:10:34.175 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:34.173+0000 7f63b6ffd700 1 --2- 172.21.15.29:0/1746657493 >> [v2:172.21.15.29:6800/3122114772,v1:172.21.15.29:6801/3122114772] conn(0x7f63a40336d0 0x7f63a4035ac0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f63ac0036d0 tx=0x7f63ac006c50).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:10:34.175 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:34.173+0000 7f639ffff700 1 -- 172.21.15.29:0/1746657493 <== mgr.14116 v2:172.21.15.29:6800/3122114772 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (secure 0 0 0) 0x7f63b8059100 con 0x7f63a40336d0 2023-12-01T22:10:34.182 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:34.181+0000 7f63bdb80700 1 -- 172.21.15.29:0/1746657493 --> [v2:172.21.15.29:6800/3122114772,v1:172.21.15.29:6801/3122114772] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f63b8045ad0 con 0x7f63a40336d0 2023-12-01T22:10:34.185 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:34.181+0000 7f639ffff700 1 -- 172.21.15.29:0/1746657493 <== mgr.14116 v2:172.21.15.29:6800/3122114772 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f63b8045ad0 con 0x7f63a40336d0 2023-12-01T22:10:34.186 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: { 2023-12-01T22:10:34.187 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "mgrmap_epoch": 7, 2023-12-01T22:10:34.187 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "initialized": true 2023-12-01T22:10:34.187 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: } 2023-12-01T22:10:34.187 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:34.181+0000 7f63bdb80700 1 -- 172.21.15.29:0/1746657493 >> [v2:172.21.15.29:6800/3122114772,v1:172.21.15.29:6801/3122114772] conn(0x7f63a40336d0 msgr2=0x7f63a4035ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:10:34.187 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:34.181+0000 7f63bdb80700 1 --2- 172.21.15.29:0/1746657493 >> [v2:172.21.15.29:6800/3122114772,v1:172.21.15.29:6801/3122114772] conn(0x7f63a40336d0 0x7f63a4035ac0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f63ac0036d0 tx=0x7f63ac006c50).stop 2023-12-01T22:10:34.187 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:34.181+0000 7f63bdb80700 1 -- 172.21.15.29:0/1746657493 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63b80703f0 msgr2=0x7f63b810ccf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:10:34.187 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:34.181+0000 7f63bdb80700 1 --2- 172.21.15.29:0/1746657493 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63b80703f0 0x7f63b810ccf0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f63a800e040 tx=0x7f63a8006ee0).stop 2023-12-01T22:10:34.187 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:34.181+0000 7f63bdb80700 1 -- 172.21.15.29:0/1746657493 shutdown_connections 2023-12-01T22:10:34.187 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:34.182+0000 7f63bdb80700 1 --2- 172.21.15.29:0/1746657493 >> [v2:172.21.15.29:6800/3122114772,v1:172.21.15.29:6801/3122114772] conn(0x7f63a40336d0 0x7f63a4035ac0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:10:34.187 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:34.182+0000 7f63bdb80700 1 --2- 172.21.15.29:0/1746657493 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63b80703f0 0x7f63b810ccf0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:10:34.187 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:34.182+0000 7f63bdb80700 1 --2- 172.21.15.29:0/1746657493 >> [v2:172.21.15.29:6800/204751073,v1:172.21.15.29:6801/204751073] conn(0x7f63a402fdb0 0x7f63a4032270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-12-01T22:10:34.187 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:34.182+0000 7f63bdb80700 1 -- 172.21.15.29:0/1746657493 >> 172.21.15.29:0/1746657493 conn(0x7f63b80f63c0 msgr2=0x7f63b80f7ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:10:34.187 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:34.182+0000 7f63bdb80700 1 -- 172.21.15.29:0/1746657493 shutdown_connections 2023-12-01T22:10:34.187 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:34.182+0000 7f63bdb80700 1 -- 172.21.15.29:0/1746657493 wait complete. 2023-12-01T22:10:34.527 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:34 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:10:33.118970+0000 mon.smithi029 (mon.0) 30 : cluster [INF] Active manager daemon smithi029.loxrfu restarted 2023-12-01T22:10:34.527 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:34 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:10:33.119377+0000 mon.smithi029 (mon.0) 31 : cluster [INF] Activating manager daemon smithi029.loxrfu 2023-12-01T22:10:34.527 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:34 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:10:33.121106+0000 mon.smithi029 (mon.0) 32 : cluster [DBG] osdmap e2: 0 total, 0 up, 0 in 2023-12-01T22:10:34.527 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:34 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:10:33.172281+0000 mon.smithi029 (mon.0) 33 : cluster [DBG] mgrmap e6: smithi029.loxrfu(active, starting, since 0.0529788s) 2023-12-01T22:10:34.527 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:34 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:10:33.173489+0000 mon.smithi029 (mon.0) 34 : audit [DBG] from='mgr.14116 172.21.15.29:0/3573596141' entity='mgr.smithi029.loxrfu' cmd=[{"prefix": "mon metadata", "id": "smithi029"}]: dispatch 2023-12-01T22:10:34.527 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:34 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:10:33.174178+0000 mon.smithi029 (mon.0) 35 : audit [DBG] from='mgr.14116 172.21.15.29:0/3573596141' entity='mgr.smithi029.loxrfu' cmd=[{"prefix": "mgr metadata", "who": "smithi029.loxrfu", "id": "smithi029.loxrfu"}]: dispatch 2023-12-01T22:10:34.527 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:34 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:10:33.174974+0000 mon.smithi029 (mon.0) 36 : audit [DBG] from='mgr.14116 172.21.15.29:0/3573596141' entity='mgr.smithi029.loxrfu' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-12-01T22:10:34.527 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:34 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:10:33.175152+0000 mon.smithi029 (mon.0) 37 : audit [DBG] from='mgr.14116 172.21.15.29:0/3573596141' entity='mgr.smithi029.loxrfu' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-12-01T22:10:34.528 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:34 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:10:33.175314+0000 mon.smithi029 (mon.0) 38 : audit [DBG] from='mgr.14116 172.21.15.29:0/3573596141' entity='mgr.smithi029.loxrfu' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-12-01T22:10:34.528 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:34 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:10:33.188607+0000 mon.smithi029 (mon.0) 39 : cluster [INF] Manager daemon smithi029.loxrfu is now available 2023-12-01T22:10:34.528 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:34 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:10:33.204194+0000 mon.smithi029 (mon.0) 40 : audit [INF] from='mgr.14116 172.21.15.29:0/3573596141' entity='mgr.smithi029.loxrfu' 2023-12-01T22:10:34.528 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:34 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:10:33.207156+0000 mon.smithi029 (mon.0) 41 : audit [INF] from='mgr.14116 172.21.15.29:0/3573596141' entity='mgr.smithi029.loxrfu' 2023-12-01T22:10:34.528 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:34 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:10:33.209862+0000 mon.smithi029 (mon.0) 42 : audit [INF] from='mgr.14116 172.21.15.29:0/3573596141' entity='mgr.smithi029.loxrfu' 2023-12-01T22:10:34.528 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:34 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:10:33.312209+0000 mon.smithi029 (mon.0) 43 : audit [INF] from='mgr.14116 172.21.15.29:0/3573596141' entity='mgr.smithi029.loxrfu' 2023-12-01T22:10:34.528 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:34 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:10:33.313014+0000 mon.smithi029 (mon.0) 44 : audit [DBG] from='mgr.14116 172.21.15.29:0/3573596141' entity='mgr.smithi029.loxrfu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-01T22:10:34.528 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:34 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:10:33.314296+0000 mon.smithi029 (mon.0) 45 : audit [DBG] from='mgr.14116 172.21.15.29:0/3573596141' entity='mgr.smithi029.loxrfu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-01T22:10:34.528 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:34 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:10:33.316177+0000 mon.smithi029 (mon.0) 46 : audit [DBG] from='mgr.14116 172.21.15.29:0/3573596141' entity='mgr.smithi029.loxrfu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-01T22:10:34.528 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:34 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:10:33.328182+0000 mon.smithi029 (mon.0) 47 : audit [INF] from='mgr.14116 172.21.15.29:0/3573596141' entity='mgr.smithi029.loxrfu' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi029.loxrfu/mirror_snapshot_schedule"}]: dispatch 2023-12-01T22:10:34.528 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:34 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:10:33.329615+0000 mon.smithi029 (mon.0) 48 : audit [INF] from='mgr.14116 172.21.15.29:0/3573596141' entity='mgr.smithi029.loxrfu' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi029.loxrfu/trash_purge_schedule"}]: dispatch 2023-12-01T22:10:35.527 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:35 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:10:34.174537+0000 mgr.smithi029.loxrfu (mgr.14116) 1 : audit [DBG] from='client.14120 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2023-12-01T22:10:35.527 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:35 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:10:34.175430+0000 mon.smithi029 (mon.0) 49 : cluster [DBG] mgrmap e7: smithi029.loxrfu(active, since 1.05612s) 2023-12-01T22:10:35.527 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:35 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:10:34.182368+0000 mgr.smithi029.loxrfu (mgr.14116) 2 : audit [DBG] from='client.14120 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2023-12-01T22:10:36.526 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:36 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:10:35.177400+0000 mon.smithi029 (mon.0) 50 : cluster [DBG] mgrmap e8: smithi029.loxrfu(active, since 2s) 2023-12-01T22:10:36.624 INFO:teuthology.orchestra.run.smithi029.stderr:mgr epoch 5 is available 2023-12-01T22:10:36.624 INFO:teuthology.orchestra.run.smithi029.stderr:Setting orchestrator backend to cephadm... 2023-12-01T22:10:36.624 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi029 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/5558ab6c-9096-11ee-95a2-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpou9lnrt4:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpxe9edifa:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch set backend cephadm 2023-12-01T22:10:38.307 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:37.843+0000 7f83af8f0700 1 Processor -- start 2023-12-01T22:10:38.366 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:37.844+0000 7f83af8f0700 1 -- start start 2023-12-01T22:10:38.366 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:37.844+0000 7f83af8f0700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f83a80703b0 0x7f83a8100d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-01T22:10:38.367 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:37.844+0000 7f83af8f0700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83a80fc750 con 0x7f83a80703b0 2023-12-01T22:10:38.367 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:37.844+0000 7f83ad68c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f83a80703b0 0x7f83a8100d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:10:38.367 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:37.844+0000 7f83ad68c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f83a80703b0 0x7f83a8100d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49612/0 (socket says 172.21.15.29:49612) 2023-12-01T22:10:38.367 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:37.844+0000 7f83ad68c700 1 -- 172.21.15.29:0/2332796088 learned_addr learned my addr 172.21.15.29:0/2332796088 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-12-01T22:10:38.367 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:37.844+0000 7f83ad68c700 1 -- 172.21.15.29:0/2332796088 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83a8101250 con 0x7f83a80703b0 2023-12-01T22:10:38.367 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:37.844+0000 7f83ad68c700 1 --2- 172.21.15.29:0/2332796088 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f83a80703b0 0x7f83a8100d60 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f83a400b5f0 tx=0x7f83a4006ee0).ready entity=mon.0 client_cookie=9b61a9976405f738 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:10:38.368 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:37.845+0000 7f8397fff700 1 -- 172.21.15.29:0/2332796088 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f83a400d600 con 0x7f83a80703b0 2023-12-01T22:10:38.368 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:37.845+0000 7f8397fff700 1 -- 172.21.15.29:0/2332796088 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f83a400dbd0 con 0x7f83a80703b0 2023-12-01T22:10:38.368 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:37.845+0000 7f8397fff700 1 -- 172.21.15.29:0/2332796088 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f83a4011c00 con 0x7f83a80703b0 2023-12-01T22:10:38.368 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:37.845+0000 7f83af8f0700 1 -- 172.21.15.29:0/2332796088 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f83a80703b0 msgr2=0x7f83a8100d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:10:38.368 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:37.845+0000 7f83af8f0700 1 --2- 172.21.15.29:0/2332796088 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f83a80703b0 0x7f83a8100d60 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f83a400b5f0 tx=0x7f83a4006ee0).stop 2023-12-01T22:10:38.368 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:37.845+0000 7f83af8f0700 1 -- 172.21.15.29:0/2332796088 shutdown_connections 2023-12-01T22:10:38.368 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:37.845+0000 7f83af8f0700 1 --2- 172.21.15.29:0/2332796088 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f83a80703b0 0x7f83a8100d60 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:10:38.369 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:37.845+0000 7f83af8f0700 1 -- 172.21.15.29:0/2332796088 >> 172.21.15.29:0/2332796088 conn(0x7f83a80f63c0 msgr2=0x7f83a80f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:10:38.369 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:37.845+0000 7f83af8f0700 1 -- 172.21.15.29:0/2332796088 shutdown_connections 2023-12-01T22:10:38.369 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:37.845+0000 7f83af8f0700 1 -- 172.21.15.29:0/2332796088 wait complete. 2023-12-01T22:10:38.369 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:37.846+0000 7f83af8f0700 1 Processor -- start 2023-12-01T22:10:38.369 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:37.846+0000 7f83af8f0700 1 -- start start 2023-12-01T22:10:38.369 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:37.846+0000 7f83af8f0700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f83a80703b0 0x7f83a810ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:10:38.369 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:37.846+0000 7f83af8f0700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83a810d1e0 con 0x7f83a80703b0 2023-12-01T22:10:38.369 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:37.847+0000 7f83ad68c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f83a80703b0 0x7f83a810ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:10:38.369 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:37.847+0000 7f83ad68c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f83a80703b0 0x7f83a810ccf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49614/0 (socket says 172.21.15.29:49614) 2023-12-01T22:10:38.369 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:37.847+0000 7f83ad68c700 1 -- 172.21.15.29:0/2956858521 learned_addr learned my addr 172.21.15.29:0/2956858521 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-12-01T22:10:38.369 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:37.847+0000 7f83ad68c700 1 -- 172.21.15.29:0/2956858521 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83a400b200 con 0x7f83a80703b0 2023-12-01T22:10:38.369 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:37.847+0000 7f83ad68c700 1 --2- 172.21.15.29:0/2956858521 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f83a80703b0 0x7f83a810ccf0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f83a400e040 tx=0x7f83a4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:10:38.370 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:37.847+0000 7f8395ffb700 1 -- 172.21.15.29:0/2956858521 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f83a400d9f0 con 0x7f83a80703b0 2023-12-01T22:10:38.370 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:37.847+0000 7f8395ffb700 1 -- 172.21.15.29:0/2956858521 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f83a401a9f0 con 0x7f83a80703b0 2023-12-01T22:10:38.370 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:37.847+0000 7f83af8f0700 1 -- 172.21.15.29:0/2956858521 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83a810d3e0 con 0x7f83a80703b0 2023-12-01T22:10:38.370 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:37.847+0000 7f8395ffb700 1 -- 172.21.15.29:0/2956858521 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f83a401b8c0 con 0x7f83a80703b0 2023-12-01T22:10:38.370 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:37.847+0000 7f83af8f0700 1 -- 172.21.15.29:0/2956858521 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83a810d7f0 con 0x7f83a80703b0 2023-12-01T22:10:38.370 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:37.848+0000 7f8395ffb700 1 -- 172.21.15.29:0/2956858521 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f83a400f070 con 0x7f83a80703b0 2023-12-01T22:10:38.370 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:37.848+0000 7f8395ffb700 1 --2- 172.21.15.29:0/2956858521 >> [v2:172.21.15.29:6800/3122114772,v1:172.21.15.29:6801/3122114772] conn(0x7f839002fdb0 0x7f8390032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:10:38.370 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:37.849+0000 7f8395ffb700 1 -- 172.21.15.29:0/2956858521 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f83a40413a0 con 0x7f83a80703b0 2023-12-01T22:10:38.370 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:37.849+0000 7f83af8f0700 1 -- 172.21.15.29:0/2956858521 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83a8045ad0 con 0x7f83a80703b0 2023-12-01T22:10:38.370 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:37.849+0000 7f83ace8b700 1 --2- 172.21.15.29:0/2956858521 >> [v2:172.21.15.29:6800/3122114772,v1:172.21.15.29:6801/3122114772] conn(0x7f839002fdb0 0x7f8390032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:10:38.371 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:37.849+0000 7f83ace8b700 1 --2- 172.21.15.29:0/2956858521 >> [v2:172.21.15.29:6800/3122114772,v1:172.21.15.29:6801/3122114772] conn(0x7f839002fdb0 0x7f8390032270 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f83980096a0 tx=0x7f8398006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:10:38.371 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:37.853+0000 7f8395ffb700 1 -- 172.21.15.29:0/2956858521 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f83a4016070 con 0x7f83a80703b0 2023-12-01T22:10:38.371 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:38.053+0000 7f83af8f0700 1 -- 172.21.15.29:0/2956858521 --> [v2:172.21.15.29:6800/3122114772,v1:172.21.15.29:6801/3122114772] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7f83a810ed50 con 0x7f839002fdb0 2023-12-01T22:10:38.371 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:38.058+0000 7f8395ffb700 1 -- 172.21.15.29:0/2956858521 <== mgr.14116 v2:172.21.15.29:6800/3122114772 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f83a810ed50 con 0x7f839002fdb0 2023-12-01T22:10:38.371 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:38.060+0000 7f83af8f0700 1 -- 172.21.15.29:0/2956858521 >> [v2:172.21.15.29:6800/3122114772,v1:172.21.15.29:6801/3122114772] conn(0x7f839002fdb0 msgr2=0x7f8390032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:10:38.371 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:38.060+0000 7f83af8f0700 1 --2- 172.21.15.29:0/2956858521 >> [v2:172.21.15.29:6800/3122114772,v1:172.21.15.29:6801/3122114772] conn(0x7f839002fdb0 0x7f8390032270 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f83980096a0 tx=0x7f8398006b40).stop 2023-12-01T22:10:38.371 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:38.060+0000 7f83af8f0700 1 -- 172.21.15.29:0/2956858521 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f83a80703b0 msgr2=0x7f83a810ccf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:10:38.371 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:38.060+0000 7f83af8f0700 1 --2- 172.21.15.29:0/2956858521 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f83a80703b0 0x7f83a810ccf0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f83a400e040 tx=0x7f83a4006ee0).stop 2023-12-01T22:10:38.371 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:38.060+0000 7f83af8f0700 1 -- 172.21.15.29:0/2956858521 shutdown_connections 2023-12-01T22:10:38.371 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:38.060+0000 7f83af8f0700 1 --2- 172.21.15.29:0/2956858521 >> [v2:172.21.15.29:6800/3122114772,v1:172.21.15.29:6801/3122114772] conn(0x7f839002fdb0 0x7f8390032270 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:10:38.371 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:38.060+0000 7f83af8f0700 1 --2- 172.21.15.29:0/2956858521 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f83a80703b0 0x7f83a810ccf0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:10:38.371 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:38.060+0000 7f83af8f0700 1 -- 172.21.15.29:0/2956858521 >> 172.21.15.29:0/2956858521 conn(0x7f83a80f63c0 msgr2=0x7f83a80f7f50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:10:38.371 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:38.060+0000 7f83af8f0700 1 -- 172.21.15.29:0/2956858521 shutdown_connections 2023-12-01T22:10:38.371 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:38.060+0000 7f83af8f0700 1 -- 172.21.15.29:0/2956858521 wait complete. 2023-12-01T22:10:39.527 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:39 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:10:38.054812+0000 mgr.smithi029.loxrfu (mgr.14116) 3 : audit [DBG] from='client.14126 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2023-12-01T22:10:39.527 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:39 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:10:38.058887+0000 mon.smithi029 (mon.0) 51 : audit [INF] from='mgr.14116 172.21.15.29:0/3573596141' entity='mgr.smithi029.loxrfu' 2023-12-01T22:10:39.527 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:39 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:10:38.063362+0000 mon.smithi029 (mon.0) 52 : audit [DBG] from='mgr.14116 172.21.15.29:0/3573596141' entity='mgr.smithi029.loxrfu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-01T22:10:41.665 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi029 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/5558ab6c-9096-11ee-95a2-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpou9lnrt4:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpxe9edifa:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm set-user root 2023-12-01T22:10:42.978 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:42.816+0000 7fb327cd2700 1 Processor -- start 2023-12-01T22:10:42.978 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:42.816+0000 7fb327cd2700 1 -- start start 2023-12-01T22:10:42.979 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:42.816+0000 7fb327cd2700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3200703f0 0x7fb3200fe920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-01T22:10:42.979 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:42.816+0000 7fb327cd2700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3200fee10 con 0x7fb3200703f0 2023-12-01T22:10:42.979 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:42.816+0000 7fb325a6e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3200703f0 0x7fb3200fe920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:10:42.979 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:42.816+0000 7fb325a6e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3200703f0 0x7fb3200fe920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49630/0 (socket says 172.21.15.29:49630) 2023-12-01T22:10:42.979 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:42.816+0000 7fb325a6e700 1 -- 172.21.15.29:0/1363342232 learned_addr learned my addr 172.21.15.29:0/1363342232 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-12-01T22:10:42.979 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:42.817+0000 7fb325a6e700 1 -- 172.21.15.29:0/1363342232 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3200fef50 con 0x7fb3200703f0 2023-12-01T22:10:42.979 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:42.817+0000 7fb325a6e700 1 --2- 172.21.15.29:0/1363342232 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3200703f0 0x7fb3200fe920 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7fb31c00b5f0 tx=0x7fb31c006ee0).ready entity=mon.0 client_cookie=50281a93b13281f4 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:10:42.979 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:42.817+0000 7fb324a6c700 1 -- 172.21.15.29:0/1363342232 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb31c00d600 con 0x7fb3200703f0 2023-12-01T22:10:42.979 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:42.817+0000 7fb324a6c700 1 -- 172.21.15.29:0/1363342232 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fb31c00dbd0 con 0x7fb3200703f0 2023-12-01T22:10:42.979 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:42.817+0000 7fb324a6c700 1 -- 172.21.15.29:0/1363342232 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb31c011c00 con 0x7fb3200703f0 2023-12-01T22:10:42.979 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:42.817+0000 7fb327cd2700 1 -- 172.21.15.29:0/1363342232 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3200703f0 msgr2=0x7fb3200fe920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:10:42.979 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:42.817+0000 7fb327cd2700 1 --2- 172.21.15.29:0/1363342232 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3200703f0 0x7fb3200fe920 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7fb31c00b5f0 tx=0x7fb31c006ee0).stop 2023-12-01T22:10:42.979 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:42.817+0000 7fb327cd2700 1 -- 172.21.15.29:0/1363342232 shutdown_connections 2023-12-01T22:10:42.980 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:42.817+0000 7fb327cd2700 1 --2- 172.21.15.29:0/1363342232 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3200703f0 0x7fb3200fe920 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:10:42.980 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:42.817+0000 7fb327cd2700 1 -- 172.21.15.29:0/1363342232 >> 172.21.15.29:0/1363342232 conn(0x7fb3200f63c0 msgr2=0x7fb3200f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:10:42.980 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:42.818+0000 7fb327cd2700 1 -- 172.21.15.29:0/1363342232 shutdown_connections 2023-12-01T22:10:42.980 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:42.818+0000 7fb327cd2700 1 -- 172.21.15.29:0/1363342232 wait complete. 2023-12-01T22:10:42.980 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:42.818+0000 7fb327cd2700 1 Processor -- start 2023-12-01T22:10:42.980 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:42.818+0000 7fb327cd2700 1 -- start start 2023-12-01T22:10:42.980 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:42.818+0000 7fb327cd2700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3200703f0 0x7fb32010a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:10:42.980 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:42.818+0000 7fb327cd2700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb32010aeb0 con 0x7fb3200703f0 2023-12-01T22:10:42.980 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:42.818+0000 7fb325a6e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3200703f0 0x7fb32010a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:10:42.980 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:42.818+0000 7fb325a6e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3200703f0 0x7fb32010a9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49638/0 (socket says 172.21.15.29:49638) 2023-12-01T22:10:42.980 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:42.818+0000 7fb325a6e700 1 -- 172.21.15.29:0/3854895401 learned_addr learned my addr 172.21.15.29:0/3854895401 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-12-01T22:10:42.980 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:42.819+0000 7fb325a6e700 1 -- 172.21.15.29:0/3854895401 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb31c00b200 con 0x7fb3200703f0 2023-12-01T22:10:42.980 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:42.819+0000 7fb325a6e700 1 --2- 172.21.15.29:0/3854895401 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3200703f0 0x7fb32010a9c0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7fb31c00b190 tx=0x7fb31c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:10:42.980 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:42.819+0000 7fb30e7fc700 1 -- 172.21.15.29:0/3854895401 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb31c01b410 con 0x7fb3200703f0 2023-12-01T22:10:42.981 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:42.819+0000 7fb30e7fc700 1 -- 172.21.15.29:0/3854895401 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fb31c01b9e0 con 0x7fb3200703f0 2023-12-01T22:10:42.981 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:42.819+0000 7fb327cd2700 1 -- 172.21.15.29:0/3854895401 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb32010b0b0 con 0x7fb3200703f0 2023-12-01T22:10:42.981 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:42.819+0000 7fb30e7fc700 1 -- 172.21.15.29:0/3854895401 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb31c01a880 con 0x7fb3200703f0 2023-12-01T22:10:42.981 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:42.819+0000 7fb327cd2700 1 -- 172.21.15.29:0/3854895401 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb32010b4c0 con 0x7fb3200703f0 2023-12-01T22:10:42.981 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:42.820+0000 7fb30e7fc700 1 -- 172.21.15.29:0/3854895401 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fb31c00f070 con 0x7fb3200703f0 2023-12-01T22:10:42.981 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:42.820+0000 7fb327cd2700 1 -- 172.21.15.29:0/3854895401 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb320045ad0 con 0x7fb3200703f0 2023-12-01T22:10:42.981 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:42.820+0000 7fb30e7fc700 1 --2- 172.21.15.29:0/3854895401 >> [v2:172.21.15.29:6800/3122114772,v1:172.21.15.29:6801/3122114772] conn(0x7fb30802fdb0 0x7fb308032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:10:42.981 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:42.820+0000 7fb30e7fc700 1 -- 172.21.15.29:0/3854895401 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fb31c0411d0 con 0x7fb3200703f0 2023-12-01T22:10:42.981 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:42.820+0000 7fb32526d700 1 --2- 172.21.15.29:0/3854895401 >> [v2:172.21.15.29:6800/3122114772,v1:172.21.15.29:6801/3122114772] conn(0x7fb30802fdb0 0x7fb308032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:10:42.981 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:42.821+0000 7fb32526d700 1 --2- 172.21.15.29:0/3854895401 >> [v2:172.21.15.29:6800/3122114772,v1:172.21.15.29:6801/3122114772] conn(0x7fb30802fdb0 0x7fb308032270 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fb3100096a0 tx=0x7fb310006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:10:42.981 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:42.825+0000 7fb30e7fc700 1 -- 172.21.15.29:0/3854895401 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb31c0224f0 con 0x7fb3200703f0 2023-12-01T22:10:42.992 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:42.990+0000 7fb327cd2700 1 -- 172.21.15.29:0/3854895401 --> [v2:172.21.15.29:6800/3122114772,v1:172.21.15.29:6801/3122114772] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7fb32010bda0 con 0x7fb30802fdb0 2023-12-01T22:10:42.992 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:42.991+0000 7fb30e7fc700 1 -- 172.21.15.29:0/3854895401 <== mgr.14116 v2:172.21.15.29:6800/3122114772 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7fb32010bda0 con 0x7fb30802fdb0 2023-12-01T22:10:42.992 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: value unchanged 2023-12-01T22:10:42.995 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:42.993+0000 7fb327cd2700 1 -- 172.21.15.29:0/3854895401 >> [v2:172.21.15.29:6800/3122114772,v1:172.21.15.29:6801/3122114772] conn(0x7fb30802fdb0 msgr2=0x7fb308032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:10:42.995 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:42.993+0000 7fb327cd2700 1 --2- 172.21.15.29:0/3854895401 >> [v2:172.21.15.29:6800/3122114772,v1:172.21.15.29:6801/3122114772] conn(0x7fb30802fdb0 0x7fb308032270 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fb3100096a0 tx=0x7fb310006b40).stop 2023-12-01T22:10:42.995 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:42.993+0000 7fb327cd2700 1 -- 172.21.15.29:0/3854895401 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3200703f0 msgr2=0x7fb32010a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:10:42.995 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:42.993+0000 7fb327cd2700 1 --2- 172.21.15.29:0/3854895401 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3200703f0 0x7fb32010a9c0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7fb31c00b190 tx=0x7fb31c006ee0).stop 2023-12-01T22:10:42.995 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:42.993+0000 7fb327cd2700 1 -- 172.21.15.29:0/3854895401 shutdown_connections 2023-12-01T22:10:42.995 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:42.993+0000 7fb327cd2700 1 --2- 172.21.15.29:0/3854895401 >> [v2:172.21.15.29:6800/3122114772,v1:172.21.15.29:6801/3122114772] conn(0x7fb30802fdb0 0x7fb308032270 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:10:42.996 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:42.993+0000 7fb327cd2700 1 --2- 172.21.15.29:0/3854895401 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3200703f0 0x7fb32010a9c0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:10:42.996 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:42.993+0000 7fb327cd2700 1 -- 172.21.15.29:0/3854895401 >> 172.21.15.29:0/3854895401 conn(0x7fb3200f63c0 msgr2=0x7fb3200f7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:10:42.996 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:42.993+0000 7fb327cd2700 1 -- 172.21.15.29:0/3854895401 shutdown_connections 2023-12-01T22:10:42.996 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:42.993+0000 7fb327cd2700 1 -- 172.21.15.29:0/3854895401 wait complete. 2023-12-01T22:10:43.905 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:43 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:10:42.992125+0000 mgr.smithi029.loxrfu (mgr.14116) 4 : audit [DBG] from='client.14128 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2023-12-01T22:10:45.115 INFO:teuthology.orchestra.run.smithi029.stderr:Generating ssh key... 2023-12-01T22:10:45.115 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi029 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/5558ab6c-9096-11ee-95a2-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpou9lnrt4:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpxe9edifa:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm generate-key 2023-12-01T22:10:46.726 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:46.725+0000 7f903040e700 1 Processor -- start 2023-12-01T22:10:46.726 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:46.725+0000 7f903040e700 1 -- start start 2023-12-01T22:10:46.726 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:46.725+0000 7f903040e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9028070390 0x7f9028100d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-01T22:10:46.727 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:46.725+0000 7f903040e700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90280fc770 con 0x7f9028070390 2023-12-01T22:10:46.728 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:46.726+0000 7f902e1aa700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9028070390 0x7f9028100d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:10:46.728 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:46.726+0000 7f902e1aa700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9028070390 0x7f9028100d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60622/0 (socket says 172.21.15.29:60622) 2023-12-01T22:10:46.728 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:46.726+0000 7f902e1aa700 1 -- 172.21.15.29:0/380865858 learned_addr learned my addr 172.21.15.29:0/380865858 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-12-01T22:10:46.728 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:46.726+0000 7f902e1aa700 1 -- 172.21.15.29:0/380865858 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9028101270 con 0x7f9028070390 2023-12-01T22:10:46.728 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:46.726+0000 7f902e1aa700 1 --2- 172.21.15.29:0/380865858 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9028070390 0x7f9028100d80 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f902400b5f0 tx=0x7f9024006ee0).ready entity=mon.0 client_cookie=ac184087a80abb62 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:10:46.728 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:46.726+0000 7f902d1a8700 1 -- 172.21.15.29:0/380865858 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f902400d600 con 0x7f9028070390 2023-12-01T22:10:46.728 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:46.727+0000 7f902d1a8700 1 -- 172.21.15.29:0/380865858 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f902400dbd0 con 0x7f9028070390 2023-12-01T22:10:46.728 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:46.727+0000 7f902d1a8700 1 -- 172.21.15.29:0/380865858 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9024011c00 con 0x7f9028070390 2023-12-01T22:10:46.729 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:46.727+0000 7f903040e700 1 -- 172.21.15.29:0/380865858 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9028070390 msgr2=0x7f9028100d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:10:46.729 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:46.727+0000 7f903040e700 1 --2- 172.21.15.29:0/380865858 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9028070390 0x7f9028100d80 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f902400b5f0 tx=0x7f9024006ee0).stop 2023-12-01T22:10:46.729 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:46.727+0000 7f903040e700 1 -- 172.21.15.29:0/380865858 shutdown_connections 2023-12-01T22:10:46.729 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:46.727+0000 7f903040e700 1 --2- 172.21.15.29:0/380865858 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9028070390 0x7f9028100d80 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:10:46.730 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:46.727+0000 7f903040e700 1 -- 172.21.15.29:0/380865858 >> 172.21.15.29:0/380865858 conn(0x7f90280f63c0 msgr2=0x7f90280f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:10:46.730 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:46.727+0000 7f903040e700 1 -- 172.21.15.29:0/380865858 shutdown_connections 2023-12-01T22:10:46.730 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:46.727+0000 7f903040e700 1 -- 172.21.15.29:0/380865858 wait complete. 2023-12-01T22:10:46.730 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:46.728+0000 7f903040e700 1 Processor -- start 2023-12-01T22:10:46.730 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:46.728+0000 7f903040e700 1 -- start start 2023-12-01T22:10:46.730 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:46.728+0000 7f903040e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9028070390 0x7f902810cc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:10:46.730 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:46.728+0000 7f903040e700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f902810d160 con 0x7f9028070390 2023-12-01T22:10:46.732 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:46.728+0000 7f902e1aa700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9028070390 0x7f902810cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:10:46.732 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:46.729+0000 7f902e1aa700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9028070390 0x7f902810cc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60634/0 (socket says 172.21.15.29:60634) 2023-12-01T22:10:46.732 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:46.729+0000 7f902e1aa700 1 -- 172.21.15.29:0/1650777697 learned_addr learned my addr 172.21.15.29:0/1650777697 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-12-01T22:10:46.732 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:46.729+0000 7f902e1aa700 1 -- 172.21.15.29:0/1650777697 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f902400b200 con 0x7f9028070390 2023-12-01T22:10:46.732 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:46.729+0000 7f902e1aa700 1 --2- 172.21.15.29:0/1650777697 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9028070390 0x7f902810cc70 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f902400e040 tx=0x7f9024006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:10:46.732 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:46.729+0000 7f9016ffd700 1 -- 172.21.15.29:0/1650777697 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f902400d9f0 con 0x7f9028070390 2023-12-01T22:10:46.732 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:46.729+0000 7f9016ffd700 1 -- 172.21.15.29:0/1650777697 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f902401a9f0 con 0x7f9028070390 2023-12-01T22:10:46.732 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:46.729+0000 7f903040e700 1 -- 172.21.15.29:0/1650777697 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f902810d360 con 0x7f9028070390 2023-12-01T22:10:46.732 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:46.729+0000 7f9016ffd700 1 -- 172.21.15.29:0/1650777697 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f902401b8c0 con 0x7f9028070390 2023-12-01T22:10:46.732 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:46.729+0000 7f903040e700 1 -- 172.21.15.29:0/1650777697 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f902810d770 con 0x7f9028070390 2023-12-01T22:10:46.732 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:46.730+0000 7f9016ffd700 1 -- 172.21.15.29:0/1650777697 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f902400f070 con 0x7f9028070390 2023-12-01T22:10:46.733 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:46.730+0000 7f9016ffd700 1 --2- 172.21.15.29:0/1650777697 >> [v2:172.21.15.29:6800/3122114772,v1:172.21.15.29:6801/3122114772] conn(0x7f901002fdb0 0x7f9010032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:10:46.733 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:46.730+0000 7f9016ffd700 1 -- 172.21.15.29:0/1650777697 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f9024041260 con 0x7f9028070390 2023-12-01T22:10:46.733 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:46.730+0000 7f902d9a9700 1 --2- 172.21.15.29:0/1650777697 >> [v2:172.21.15.29:6800/3122114772,v1:172.21.15.29:6801/3122114772] conn(0x7f901002fdb0 0x7f9010032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:10:46.733 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:46.731+0000 7f902d9a9700 1 --2- 172.21.15.29:0/1650777697 >> [v2:172.21.15.29:6800/3122114772,v1:172.21.15.29:6801/3122114772] conn(0x7f901002fdb0 0x7f9010032270 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f90180096a0 tx=0x7f9018006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:10:46.733 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:46.731+0000 7f903040e700 1 -- 172.21.15.29:0/1650777697 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9028045ad0 con 0x7f9028070390 2023-12-01T22:10:46.736 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:46.735+0000 7f9016ffd700 1 -- 172.21.15.29:0/1650777697 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f902401a580 con 0x7f9028070390 2023-12-01T22:10:46.903 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:46.902+0000 7f903040e700 1 -- 172.21.15.29:0/1650777697 --> [v2:172.21.15.29:6800/3122114772,v1:172.21.15.29:6801/3122114772] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f9028106b00 con 0x7f901002fdb0 2023-12-01T22:10:46.990 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:46.990+0000 7f9016ffd700 1 -- 172.21.15.29:0/1650777697 <== mgr.14116 v2:172.21.15.29:6800/3122114772 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f9028106b00 con 0x7f901002fdb0 2023-12-01T22:10:46.993 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:46.991+0000 7f903040e700 1 -- 172.21.15.29:0/1650777697 >> [v2:172.21.15.29:6800/3122114772,v1:172.21.15.29:6801/3122114772] conn(0x7f901002fdb0 msgr2=0x7f9010032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:10:46.993 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:46.991+0000 7f903040e700 1 --2- 172.21.15.29:0/1650777697 >> [v2:172.21.15.29:6800/3122114772,v1:172.21.15.29:6801/3122114772] conn(0x7f901002fdb0 0x7f9010032270 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f90180096a0 tx=0x7f9018006b40).stop 2023-12-01T22:10:46.993 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:46.991+0000 7f903040e700 1 -- 172.21.15.29:0/1650777697 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9028070390 msgr2=0x7f902810cc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:10:46.993 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:46.991+0000 7f903040e700 1 --2- 172.21.15.29:0/1650777697 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9028070390 0x7f902810cc70 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f902400e040 tx=0x7f9024006ee0).stop 2023-12-01T22:10:46.993 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:46.992+0000 7f903040e700 1 -- 172.21.15.29:0/1650777697 shutdown_connections 2023-12-01T22:10:46.993 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:46.992+0000 7f903040e700 1 --2- 172.21.15.29:0/1650777697 >> [v2:172.21.15.29:6800/3122114772,v1:172.21.15.29:6801/3122114772] conn(0x7f901002fdb0 0x7f9010032270 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:10:46.993 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:46.992+0000 7f903040e700 1 --2- 172.21.15.29:0/1650777697 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9028070390 0x7f902810cc70 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:10:46.994 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:46.992+0000 7f903040e700 1 -- 172.21.15.29:0/1650777697 >> 172.21.15.29:0/1650777697 conn(0x7f90280f63c0 msgr2=0x7f90280f7f80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:10:46.994 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:46.992+0000 7f903040e700 1 -- 172.21.15.29:0/1650777697 shutdown_connections 2023-12-01T22:10:46.994 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:46.992+0000 7f903040e700 1 -- 172.21.15.29:0/1650777697 wait complete. 2023-12-01T22:10:47.372 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi029 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/5558ab6c-9096-11ee-95a2-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpou9lnrt4:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpxe9edifa:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm get-pub-key 2023-12-01T22:10:47.755 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:47.754+0000 7fdfbd3d8700 1 Processor -- start 2023-12-01T22:10:47.756 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:47.754+0000 7fdfbd3d8700 1 -- start start 2023-12-01T22:10:47.756 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:47.754+0000 7fdfbd3d8700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdfb8070400 0x7fdfb8100d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-01T22:10:47.756 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:47.754+0000 7fdfbd3d8700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfb80fc780 con 0x7fdfb8070400 2023-12-01T22:10:47.756 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:47.754+0000 7fdfb6ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdfb8070400 0x7fdfb8100d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:10:47.756 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:47.754+0000 7fdfb6ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdfb8070400 0x7fdfb8100d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60650/0 (socket says 172.21.15.29:60650) 2023-12-01T22:10:47.756 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:47.754+0000 7fdfb6ffd700 1 -- 172.21.15.29:0/886342348 learned_addr learned my addr 172.21.15.29:0/886342348 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-12-01T22:10:47.756 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:47.754+0000 7fdfb6ffd700 1 -- 172.21.15.29:0/886342348 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfb8101280 con 0x7fdfb8070400 2023-12-01T22:10:47.756 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:47.755+0000 7fdfb6ffd700 1 --2- 172.21.15.29:0/886342348 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdfb8070400 0x7fdfb8100d90 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7fdfa800b5f0 tx=0x7fdfa8006ee0).ready entity=mon.0 client_cookie=8d33c989d11c3c46 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:10:47.756 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:47.755+0000 7fdfb5ffb700 1 -- 172.21.15.29:0/886342348 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdfa800d600 con 0x7fdfb8070400 2023-12-01T22:10:47.756 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:47.755+0000 7fdfb5ffb700 1 -- 172.21.15.29:0/886342348 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fdfa800dbd0 con 0x7fdfb8070400 2023-12-01T22:10:47.763 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:47.755+0000 7fdfb5ffb700 1 -- 172.21.15.29:0/886342348 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdfa8011c00 con 0x7fdfb8070400 2023-12-01T22:10:47.763 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:47.755+0000 7fdfbd3d8700 1 -- 172.21.15.29:0/886342348 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdfb8070400 msgr2=0x7fdfb8100d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:10:47.763 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:47.755+0000 7fdfbd3d8700 1 --2- 172.21.15.29:0/886342348 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdfb8070400 0x7fdfb8100d90 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7fdfa800b5f0 tx=0x7fdfa8006ee0).stop 2023-12-01T22:10:47.763 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:47.755+0000 7fdfbd3d8700 1 -- 172.21.15.29:0/886342348 shutdown_connections 2023-12-01T22:10:47.763 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:47.755+0000 7fdfbd3d8700 1 --2- 172.21.15.29:0/886342348 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdfb8070400 0x7fdfb8100d90 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:10:47.763 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:47.755+0000 7fdfbd3d8700 1 -- 172.21.15.29:0/886342348 >> 172.21.15.29:0/886342348 conn(0x7fdfb80f63b0 msgr2=0x7fdfb80f87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:10:47.763 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:47.755+0000 7fdfbd3d8700 1 -- 172.21.15.29:0/886342348 shutdown_connections 2023-12-01T22:10:47.764 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:47.755+0000 7fdfbd3d8700 1 -- 172.21.15.29:0/886342348 wait complete. 2023-12-01T22:10:47.764 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:47.756+0000 7fdfbd3d8700 1 Processor -- start 2023-12-01T22:10:47.764 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:47.756+0000 7fdfbd3d8700 1 -- start start 2023-12-01T22:10:47.764 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:47.756+0000 7fdfbd3d8700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdfb8070400 0x7fdfb810cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:10:47.764 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:47.756+0000 7fdfbd3d8700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfb810d240 con 0x7fdfb8070400 2023-12-01T22:10:47.764 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:47.757+0000 7fdfb6ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdfb8070400 0x7fdfb810cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:10:47.764 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:47.757+0000 7fdfb6ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdfb8070400 0x7fdfb810cd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60656/0 (socket says 172.21.15.29:60656) 2023-12-01T22:10:47.764 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:47.757+0000 7fdfb6ffd700 1 -- 172.21.15.29:0/1034097838 learned_addr learned my addr 172.21.15.29:0/1034097838 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-12-01T22:10:47.764 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:47.757+0000 7fdfb6ffd700 1 -- 172.21.15.29:0/1034097838 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfa800b200 con 0x7fdfb8070400 2023-12-01T22:10:47.764 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:47.757+0000 7fdfb6ffd700 1 --2- 172.21.15.29:0/1034097838 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdfb8070400 0x7fdfb810cd50 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7fdfa8002f40 tx=0x7fdfa8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:10:47.764 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:47.757+0000 7fdf9f7fe700 1 -- 172.21.15.29:0/1034097838 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdfa8011690 con 0x7fdfb8070400 2023-12-01T22:10:47.765 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:47.757+0000 7fdf9f7fe700 1 -- 172.21.15.29:0/1034097838 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fdfa801b880 con 0x7fdfb8070400 2023-12-01T22:10:47.765 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:47.757+0000 7fdfbd3d8700 1 -- 172.21.15.29:0/1034097838 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfb810d440 con 0x7fdfb8070400 2023-12-01T22:10:47.765 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:47.757+0000 7fdf9f7fe700 1 -- 172.21.15.29:0/1034097838 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdfa801a6b0 con 0x7fdfb8070400 2023-12-01T22:10:47.765 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:47.757+0000 7fdfbd3d8700 1 -- 172.21.15.29:0/1034097838 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfb810d850 con 0x7fdfb8070400 2023-12-01T22:10:47.765 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:47.758+0000 7fdf9f7fe700 1 -- 172.21.15.29:0/1034097838 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fdfa800f070 con 0x7fdfb8070400 2023-12-01T22:10:47.765 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:47.758+0000 7fdf9f7fe700 1 --2- 172.21.15.29:0/1034097838 >> [v2:172.21.15.29:6800/3122114772,v1:172.21.15.29:6801/3122114772] conn(0x7fdfa402fdb0 0x7fdfa4032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:10:47.765 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:47.759+0000 7fdf9f7fe700 1 -- 172.21.15.29:0/1034097838 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fdfa80411f0 con 0x7fdfb8070400 2023-12-01T22:10:47.765 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:47.759+0000 7fdfb67fc700 1 --2- 172.21.15.29:0/1034097838 >> [v2:172.21.15.29:6800/3122114772,v1:172.21.15.29:6801/3122114772] conn(0x7fdfa402fdb0 0x7fdfa4032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:10:47.765 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:47.759+0000 7fdfb67fc700 1 --2- 172.21.15.29:0/1034097838 >> [v2:172.21.15.29:6800/3122114772,v1:172.21.15.29:6801/3122114772] conn(0x7fdfa402fdb0 0x7fdfa4032270 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7fdfac0096a0 tx=0x7fdfac006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:10:47.765 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:47.759+0000 7fdfbd3d8700 1 -- 172.21.15.29:0/1034097838 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfb8045ad0 con 0x7fdfb8070400 2023-12-01T22:10:47.765 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:47.764+0000 7fdf9f7fe700 1 -- 172.21.15.29:0/1034097838 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdfa8016070 con 0x7fdfb8070400 2023-12-01T22:10:47.931 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:47.930+0000 7fdfbd3d8700 1 -- 172.21.15.29:0/1034097838 --> [v2:172.21.15.29:6800/3122114772,v1:172.21.15.29:6801/3122114772] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7fdfb8106b50 con 0x7fdfa402fdb0 2023-12-01T22:10:47.932 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:47.931+0000 7fdf9f7fe700 1 -- 172.21.15.29:0/1034097838 <== mgr.14116 v2:172.21.15.29:6800/3122114772 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7fdfb8106b50 con 0x7fdfa402fdb0 2023-12-01T22:10:47.932 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDGfrW1MXb6ua3LhHOmlc1sD/DX4/0YK/khj7Z6/nnegRYI0nWfxKFNguc9/TKDAr9QM7A2NH4gmyTorc7ZSq1KjPv8XwgTaRdEZSFA6/H0YOViA511F3V3/irIotSuZS4xJi2kSV8pzLdPQfOUPBJ4SAw9/3JLYZtnInMI3AhDxiXDPcpDICSpyoqCsxjwwPa/n8KXTZdaPmc0L56Dfeh6Uu9Vztx+uQFt3U3RwI75U7ev3y3a3Pa0/uDw1QRj6RR8+BjL5xwOAWXPAnxp9JlHNUwi1iWolVDycAaN1jn5UOvxrVdcMR8BfB0a84WWFb/XHJxVXIiipuJG1szevgcB9F1VJGO0OEua9ZuNCVs7EtScuM0E81wFuy2GDSwCqqxw+JmEr3KS8vyYXXJ1nHJ3xZ3OZqF9VwZQ3uNe5HfS0Xg+WwfUkmDJRS6ZOWSbcO6LuNANncEfXKoT1ZxPutNOxsl3IUtoZWVgPdTxOrfNTCKh8iA7ax1QvcbJnydJeRk= ceph-5558ab6c-9096-11ee-95a2-87774f69a715 2023-12-01T22:10:47.934 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:47.933+0000 7fdfbd3d8700 1 -- 172.21.15.29:0/1034097838 >> [v2:172.21.15.29:6800/3122114772,v1:172.21.15.29:6801/3122114772] conn(0x7fdfa402fdb0 msgr2=0x7fdfa4032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:10:47.934 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:47.933+0000 7fdfbd3d8700 1 --2- 172.21.15.29:0/1034097838 >> [v2:172.21.15.29:6800/3122114772,v1:172.21.15.29:6801/3122114772] conn(0x7fdfa402fdb0 0x7fdfa4032270 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7fdfac0096a0 tx=0x7fdfac006b40).stop 2023-12-01T22:10:47.934 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:47.933+0000 7fdfbd3d8700 1 -- 172.21.15.29:0/1034097838 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdfb8070400 msgr2=0x7fdfb810cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:10:47.934 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:47.933+0000 7fdfbd3d8700 1 --2- 172.21.15.29:0/1034097838 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdfb8070400 0x7fdfb810cd50 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7fdfa8002f40 tx=0x7fdfa8006ee0).stop 2023-12-01T22:10:47.935 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:47.933+0000 7fdfbd3d8700 1 -- 172.21.15.29:0/1034097838 shutdown_connections 2023-12-01T22:10:47.935 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:47.933+0000 7fdfbd3d8700 1 --2- 172.21.15.29:0/1034097838 >> [v2:172.21.15.29:6800/3122114772,v1:172.21.15.29:6801/3122114772] conn(0x7fdfa402fdb0 0x7fdfa4032270 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:10:47.935 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:47.933+0000 7fdfbd3d8700 1 --2- 172.21.15.29:0/1034097838 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdfb8070400 0x7fdfb810cd50 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:10:47.935 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:47.933+0000 7fdfbd3d8700 1 -- 172.21.15.29:0/1034097838 >> 172.21.15.29:0/1034097838 conn(0x7fdfb80f63b0 msgr2=0x7fdfb80f7f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:10:47.935 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:47.933+0000 7fdfbd3d8700 1 -- 172.21.15.29:0/1034097838 shutdown_connections 2023-12-01T22:10:47.935 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:47.933+0000 7fdfbd3d8700 1 -- 172.21.15.29:0/1034097838 wait complete. 2023-12-01T22:10:48.226 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:47 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cephadm 2023-12-01T22:10:46.903457+0000 mgr.smithi029.loxrfu (mgr.14116) 6 : cephadm [INF] Generating ssh key... 2023-12-01T22:10:48.226 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:47 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:10:46.987427+0000 mon.smithi029 (mon.0) 53 : audit [INF] from='mgr.14116 172.21.15.29:0/3573596141' entity='mgr.smithi029.loxrfu' 2023-12-01T22:10:48.226 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:47 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:10:46.989221+0000 mon.smithi029 (mon.0) 54 : audit [INF] from='mgr.14116 172.21.15.29:0/3573596141' entity='mgr.smithi029.loxrfu' 2023-12-01T22:10:48.273 INFO:teuthology.orchestra.run.smithi029.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2023-12-01T22:10:48.274 INFO:teuthology.orchestra.run.smithi029.stderr:Adding key to root@localhost authorized_keys... 2023-12-01T22:10:48.275 INFO:teuthology.orchestra.run.smithi029.stderr:Adding host smithi029... 2023-12-01T22:10:48.275 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi029 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/5558ab6c-9096-11ee-95a2-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpou9lnrt4:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpxe9edifa:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch host add smithi029 172.21.15.29 2023-12-01T22:10:48.672 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:48.671+0000 7f70ec236700 1 Processor -- start 2023-12-01T22:10:48.672 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:48.671+0000 7f70ec236700 1 -- start start 2023-12-01T22:10:48.673 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:48.672+0000 7f70ec236700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70e4072500 0x7f70e4070aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-01T22:10:48.673 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:48.672+0000 7f70ec236700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70e4072920 con 0x7f70e4072500 2023-12-01T22:10:48.674 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:48.672+0000 7f70e9fd2700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70e4072500 0x7f70e4070aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:10:48.674 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:48.672+0000 7f70e9fd2700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70e4072500 0x7f70e4070aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60668/0 (socket says 172.21.15.29:60668) 2023-12-01T22:10:48.674 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:48.672+0000 7f70e9fd2700 1 -- 172.21.15.29:0/2357441588 learned_addr learned my addr 172.21.15.29:0/2357441588 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-12-01T22:10:48.674 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:48.672+0000 7f70e9fd2700 1 -- 172.21.15.29:0/2357441588 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70e4070f90 con 0x7f70e4072500 2023-12-01T22:10:48.674 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:48.672+0000 7f70e9fd2700 1 --2- 172.21.15.29:0/2357441588 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70e4072500 0x7f70e4070aa0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f70e000b5f0 tx=0x7f70e0006ee0).ready entity=mon.0 client_cookie=7d8719b76b381e8c server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:10:48.674 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:48.672+0000 7f70e8fd0700 1 -- 172.21.15.29:0/2357441588 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f70e000d600 con 0x7f70e4072500 2023-12-01T22:10:48.674 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:48.673+0000 7f70e8fd0700 1 -- 172.21.15.29:0/2357441588 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f70e000dbd0 con 0x7f70e4072500 2023-12-01T22:10:48.674 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:48.673+0000 7f70e8fd0700 1 -- 172.21.15.29:0/2357441588 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f70e0011c00 con 0x7f70e4072500 2023-12-01T22:10:48.676 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:48.673+0000 7f70ec236700 1 -- 172.21.15.29:0/2357441588 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70e4072500 msgr2=0x7f70e4070aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:10:48.676 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:48.673+0000 7f70ec236700 1 --2- 172.21.15.29:0/2357441588 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70e4072500 0x7f70e4070aa0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f70e000b5f0 tx=0x7f70e0006ee0).stop 2023-12-01T22:10:48.676 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:48.673+0000 7f70ec236700 1 -- 172.21.15.29:0/2357441588 shutdown_connections 2023-12-01T22:10:48.676 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:48.673+0000 7f70ec236700 1 --2- 172.21.15.29:0/2357441588 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70e4072500 0x7f70e4070aa0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:10:48.676 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:48.673+0000 7f70ec236700 1 -- 172.21.15.29:0/2357441588 >> 172.21.15.29:0/2357441588 conn(0x7f70e40f74b0 msgr2=0x7f70e40f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:10:48.676 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:48.673+0000 7f70ec236700 1 -- 172.21.15.29:0/2357441588 shutdown_connections 2023-12-01T22:10:48.676 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:48.673+0000 7f70ec236700 1 -- 172.21.15.29:0/2357441588 wait complete. 2023-12-01T22:10:48.676 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:48.674+0000 7f70ec236700 1 Processor -- start 2023-12-01T22:10:48.676 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:48.674+0000 7f70ec236700 1 -- start start 2023-12-01T22:10:48.676 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:48.674+0000 7f70ec236700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70e4072500 0x7f70e410cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:10:48.677 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:48.674+0000 7f70ec236700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70e410d240 con 0x7f70e4072500 2023-12-01T22:10:48.677 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:48.674+0000 7f70e9fd2700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70e4072500 0x7f70e410cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:10:48.677 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:48.674+0000 7f70e9fd2700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70e4072500 0x7f70e410cd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60670/0 (socket says 172.21.15.29:60670) 2023-12-01T22:10:48.677 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:48.674+0000 7f70e9fd2700 1 -- 172.21.15.29:0/2840328333 learned_addr learned my addr 172.21.15.29:0/2840328333 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-12-01T22:10:48.677 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:48.675+0000 7f70e9fd2700 1 -- 172.21.15.29:0/2840328333 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70e000b200 con 0x7f70e4072500 2023-12-01T22:10:48.677 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:48.675+0000 7f70e9fd2700 1 --2- 172.21.15.29:0/2840328333 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70e4072500 0x7f70e410cd50 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f70e000b190 tx=0x7f70e0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:10:48.678 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:48.675+0000 7f70d27fc700 1 -- 172.21.15.29:0/2840328333 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f70e0011a20 con 0x7f70e4072500 2023-12-01T22:10:48.678 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:48.675+0000 7f70ec236700 1 -- 172.21.15.29:0/2840328333 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70e410d440 con 0x7f70e4072500 2023-12-01T22:10:48.678 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:48.675+0000 7f70d27fc700 1 -- 172.21.15.29:0/2840328333 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f70e001b880 con 0x7f70e4072500 2023-12-01T22:10:48.678 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:48.675+0000 7f70d27fc700 1 -- 172.21.15.29:0/2840328333 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f70e001a710 con 0x7f70e4072500 2023-12-01T22:10:48.679 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:48.675+0000 7f70ec236700 1 -- 172.21.15.29:0/2840328333 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70e410d850 con 0x7f70e4072500 2023-12-01T22:10:48.679 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:48.677+0000 7f70ec236700 1 -- 172.21.15.29:0/2840328333 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70e4045ad0 con 0x7f70e4072500 2023-12-01T22:10:48.679 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:48.677+0000 7f70d27fc700 1 -- 172.21.15.29:0/2840328333 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f70e000f070 con 0x7f70e4072500 2023-12-01T22:10:48.679 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:48.677+0000 7f70d27fc700 1 --2- 172.21.15.29:0/2840328333 >> [v2:172.21.15.29:6800/3122114772,v1:172.21.15.29:6801/3122114772] conn(0x7f70cc02fd50 0x7f70cc032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:10:48.679 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:48.677+0000 7f70d27fc700 1 -- 172.21.15.29:0/2840328333 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f70e0041bf0 con 0x7f70e4072500 2023-12-01T22:10:48.679 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:48.677+0000 7f70e97d1700 1 --2- 172.21.15.29:0/2840328333 >> [v2:172.21.15.29:6800/3122114772,v1:172.21.15.29:6801/3122114772] conn(0x7f70cc02fd50 0x7f70cc032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:10:48.679 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:48.678+0000 7f70e97d1700 1 --2- 172.21.15.29:0/2840328333 >> [v2:172.21.15.29:6800/3122114772,v1:172.21.15.29:6801/3122114772] conn(0x7f70cc02fd50 0x7f70cc032210 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f70d40096a0 tx=0x7f70d4006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:10:48.682 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:48.681+0000 7f70d27fc700 1 -- 172.21.15.29:0/2840328333 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f70e002d950 con 0x7f70e4072500 2023-12-01T22:10:48.872 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:48.870+0000 7f70ec236700 1 -- 172.21.15.29:0/2840328333 --> [v2:172.21.15.29:6800/3122114772,v1:172.21.15.29:6801/3122114772] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi029", "addr": "172.21.15.29", "target": ["mon-mgr", ""]}) v1 -- 0x7f70e410e050 con 0x7f70cc02fd50 2023-12-01T22:10:49.276 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:48 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:10:47.931295+0000 mgr.smithi029.loxrfu (mgr.14116) 7 : audit [DBG] from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2023-12-01T22:10:49.531 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: Added host 'smithi029' 2023-12-01T22:10:49.531 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:49.528+0000 7f70d27fc700 1 -- 172.21.15.29:0/2840328333 <== mgr.14116 v2:172.21.15.29:6800/3122114772 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+23 (secure 0 0 0) 0x7f70e410e050 con 0x7f70cc02fd50 2023-12-01T22:10:49.533 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:49.531+0000 7f70ec236700 1 -- 172.21.15.29:0/2840328333 >> [v2:172.21.15.29:6800/3122114772,v1:172.21.15.29:6801/3122114772] conn(0x7f70cc02fd50 msgr2=0x7f70cc032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:10:49.533 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:49.531+0000 7f70ec236700 1 --2- 172.21.15.29:0/2840328333 >> [v2:172.21.15.29:6800/3122114772,v1:172.21.15.29:6801/3122114772] conn(0x7f70cc02fd50 0x7f70cc032210 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f70d40096a0 tx=0x7f70d4006b40).stop 2023-12-01T22:10:49.533 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:49.531+0000 7f70ec236700 1 -- 172.21.15.29:0/2840328333 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70e4072500 msgr2=0x7f70e410cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:10:49.533 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:49.531+0000 7f70ec236700 1 --2- 172.21.15.29:0/2840328333 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70e4072500 0x7f70e410cd50 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f70e000b190 tx=0x7f70e0006ee0).stop 2023-12-01T22:10:49.533 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:49.531+0000 7f70ec236700 1 -- 172.21.15.29:0/2840328333 shutdown_connections 2023-12-01T22:10:49.534 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:49.531+0000 7f70ec236700 1 --2- 172.21.15.29:0/2840328333 >> [v2:172.21.15.29:6800/3122114772,v1:172.21.15.29:6801/3122114772] conn(0x7f70cc02fd50 0x7f70cc032210 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:10:49.534 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:49.531+0000 7f70ec236700 1 --2- 172.21.15.29:0/2840328333 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70e4072500 0x7f70e410cd50 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:10:49.534 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:49.531+0000 7f70ec236700 1 -- 172.21.15.29:0/2840328333 >> 172.21.15.29:0/2840328333 conn(0x7f70e40f74b0 msgr2=0x7f70e40f9030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:10:49.534 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:49.531+0000 7f70ec236700 1 -- 172.21.15.29:0/2840328333 shutdown_connections 2023-12-01T22:10:49.534 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:49.531+0000 7f70ec236700 1 -- 172.21.15.29:0/2840328333 wait complete. 2023-12-01T22:10:49.834 INFO:teuthology.orchestra.run.smithi029.stderr:Deploying mon service with default placement... 2023-12-01T22:10:49.835 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi029 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/5558ab6c-9096-11ee-95a2-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpou9lnrt4:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpxe9edifa:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply mon 2023-12-01T22:10:50.188 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:50.184+0000 7f064ef88700 1 Processor -- start 2023-12-01T22:10:50.188 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:50.184+0000 7f064ef88700 1 -- start start 2023-12-01T22:10:50.188 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:50.184+0000 7f064ef88700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0648070460 0x7f06480fe930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-01T22:10:50.188 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:50.184+0000 7f064ef88700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06480fee20 con 0x7f0648070460 2023-12-01T22:10:50.188 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:50.185+0000 7f064cd24700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0648070460 0x7f06480fe930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:10:50.188 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:50.185+0000 7f064cd24700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0648070460 0x7f06480fe930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60682/0 (socket says 172.21.15.29:60682) 2023-12-01T22:10:50.188 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:50.185+0000 7f064cd24700 1 -- 172.21.15.29:0/1824753229 learned_addr learned my addr 172.21.15.29:0/1824753229 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-12-01T22:10:50.188 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:50.185+0000 7f064cd24700 1 -- 172.21.15.29:0/1824753229 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06480fef60 con 0x7f0648070460 2023-12-01T22:10:50.188 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:50.186+0000 7f064cd24700 1 --2- 172.21.15.29:0/1824753229 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0648070460 0x7f06480fe930 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f063c00b5f0 tx=0x7f063c006ee0).ready entity=mon.0 client_cookie=3ba8fe00b8d7c3a server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:10:50.188 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:50.186+0000 7f06477fe700 1 -- 172.21.15.29:0/1824753229 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f063c00d600 con 0x7f0648070460 2023-12-01T22:10:50.189 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:50.186+0000 7f06477fe700 1 -- 172.21.15.29:0/1824753229 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f063c00dbd0 con 0x7f0648070460 2023-12-01T22:10:50.189 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:50.186+0000 7f06477fe700 1 -- 172.21.15.29:0/1824753229 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f063c011c00 con 0x7f0648070460 2023-12-01T22:10:50.191 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:50.186+0000 7f064ef88700 1 -- 172.21.15.29:0/1824753229 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0648070460 msgr2=0x7f06480fe930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:10:50.191 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:50.186+0000 7f064ef88700 1 --2- 172.21.15.29:0/1824753229 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0648070460 0x7f06480fe930 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f063c00b5f0 tx=0x7f063c006ee0).stop 2023-12-01T22:10:50.192 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:50.186+0000 7f064ef88700 1 -- 172.21.15.29:0/1824753229 shutdown_connections 2023-12-01T22:10:50.192 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:50.186+0000 7f064ef88700 1 --2- 172.21.15.29:0/1824753229 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0648070460 0x7f06480fe930 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:10:50.192 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:50.186+0000 7f064ef88700 1 -- 172.21.15.29:0/1824753229 >> 172.21.15.29:0/1824753229 conn(0x7f06480f63b0 msgr2=0x7f06480f87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:10:50.192 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:50.186+0000 7f064ef88700 1 -- 172.21.15.29:0/1824753229 shutdown_connections 2023-12-01T22:10:50.192 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:50.187+0000 7f064ef88700 1 -- 172.21.15.29:0/1824753229 wait complete. 2023-12-01T22:10:50.192 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:50.187+0000 7f064ef88700 1 Processor -- start 2023-12-01T22:10:50.192 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:50.187+0000 7f064ef88700 1 -- start start 2023-12-01T22:10:50.192 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:50.187+0000 7f064ef88700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0648070460 0x7f064810a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:10:50.192 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:50.188+0000 7f064ef88700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f064810aec0 con 0x7f0648070460 2023-12-01T22:10:50.192 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:50.188+0000 7f064cd24700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0648070460 0x7f064810a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:10:50.192 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:50.188+0000 7f064cd24700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0648070460 0x7f064810a9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60688/0 (socket says 172.21.15.29:60688) 2023-12-01T22:10:50.192 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:50.188+0000 7f064cd24700 1 -- 172.21.15.29:0/1678810009 learned_addr learned my addr 172.21.15.29:0/1678810009 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-12-01T22:10:50.192 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:50.188+0000 7f064cd24700 1 -- 172.21.15.29:0/1678810009 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f063c00b200 con 0x7f0648070460 2023-12-01T22:10:50.192 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:50.188+0000 7f064cd24700 1 --2- 172.21.15.29:0/1678810009 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0648070460 0x7f064810a9d0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f063c00b190 tx=0x7f063c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:10:50.193 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:50.188+0000 7f06457fa700 1 -- 172.21.15.29:0/1678810009 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f063c01b410 con 0x7f0648070460 2023-12-01T22:10:50.193 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:50.188+0000 7f06457fa700 1 -- 172.21.15.29:0/1678810009 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f063c01b9e0 con 0x7f0648070460 2023-12-01T22:10:50.193 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:50.188+0000 7f064ef88700 1 -- 172.21.15.29:0/1678810009 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f064810b0c0 con 0x7f0648070460 2023-12-01T22:10:50.193 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:50.188+0000 7f06457fa700 1 -- 172.21.15.29:0/1678810009 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f063c01a880 con 0x7f0648070460 2023-12-01T22:10:50.193 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:50.188+0000 7f064ef88700 1 -- 172.21.15.29:0/1678810009 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f064810b4d0 con 0x7f0648070460 2023-12-01T22:10:50.193 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:50.189+0000 7f06457fa700 1 -- 172.21.15.29:0/1678810009 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f063c00f070 con 0x7f0648070460 2023-12-01T22:10:50.193 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:50.189+0000 7f06457fa700 1 --2- 172.21.15.29:0/1678810009 >> [v2:172.21.15.29:6800/3122114772,v1:172.21.15.29:6801/3122114772] conn(0x7f062c02fdb0 0x7f062c032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:10:50.193 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:50.189+0000 7f06457fa700 1 -- 172.21.15.29:0/1678810009 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f063c0411b0 con 0x7f0648070460 2023-12-01T22:10:50.193 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:50.189+0000 7f0647fff700 1 --2- 172.21.15.29:0/1678810009 >> [v2:172.21.15.29:6800/3122114772,v1:172.21.15.29:6801/3122114772] conn(0x7f062c02fdb0 0x7f062c032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:10:50.193 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:50.190+0000 7f064ef88700 1 -- 172.21.15.29:0/1678810009 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0648059100 con 0x7f0648070460 2023-12-01T22:10:50.193 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:50.190+0000 7f0647fff700 1 --2- 172.21.15.29:0/1678810009 >> [v2:172.21.15.29:6800/3122114772,v1:172.21.15.29:6801/3122114772] conn(0x7f062c02fdb0 0x7f062c032270 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f06340096a0 tx=0x7f0634006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:10:50.196 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:50.194+0000 7f06457fa700 1 -- 172.21.15.29:0/1678810009 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f063c025460 con 0x7f0648070460 2023-12-01T22:10:50.393 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:50.391+0000 7f064ef88700 1 -- 172.21.15.29:0/1678810009 --> [v2:172.21.15.29:6800/3122114772,v1:172.21.15.29:6801/3122114772] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}) v1 -- 0x7f0648104900 con 0x7f062c02fdb0 2023-12-01T22:10:50.397 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:50.395+0000 7f06457fa700 1 -- 172.21.15.29:0/1678810009 <== mgr.14116 v2:172.21.15.29:6800/3122114772 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f0648104900 con 0x7f062c02fdb0 2023-12-01T22:10:50.397 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: Scheduled mon update... 2023-12-01T22:10:50.400 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:50.397+0000 7f064ef88700 1 -- 172.21.15.29:0/1678810009 >> [v2:172.21.15.29:6800/3122114772,v1:172.21.15.29:6801/3122114772] conn(0x7f062c02fdb0 msgr2=0x7f062c032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:10:50.400 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:50.397+0000 7f064ef88700 1 --2- 172.21.15.29:0/1678810009 >> [v2:172.21.15.29:6800/3122114772,v1:172.21.15.29:6801/3122114772] conn(0x7f062c02fdb0 0x7f062c032270 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f06340096a0 tx=0x7f0634006b40).stop 2023-12-01T22:10:50.400 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:50.397+0000 7f064ef88700 1 -- 172.21.15.29:0/1678810009 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0648070460 msgr2=0x7f064810a9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:10:50.400 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:50.397+0000 7f064ef88700 1 --2- 172.21.15.29:0/1678810009 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0648070460 0x7f064810a9d0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f063c00b190 tx=0x7f063c006ee0).stop 2023-12-01T22:10:50.400 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:50.397+0000 7f064ef88700 1 -- 172.21.15.29:0/1678810009 shutdown_connections 2023-12-01T22:10:50.400 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:50.397+0000 7f064ef88700 1 --2- 172.21.15.29:0/1678810009 >> [v2:172.21.15.29:6800/3122114772,v1:172.21.15.29:6801/3122114772] conn(0x7f062c02fdb0 0x7f062c032270 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:10:50.400 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:50.397+0000 7f064ef88700 1 --2- 172.21.15.29:0/1678810009 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0648070460 0x7f064810a9d0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:10:50.400 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:50.397+0000 7f064ef88700 1 -- 172.21.15.29:0/1678810009 >> 172.21.15.29:0/1678810009 conn(0x7f06480f63b0 msgr2=0x7f06480f7f10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:10:50.400 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:50.397+0000 7f064ef88700 1 -- 172.21.15.29:0/1678810009 shutdown_connections 2023-12-01T22:10:50.400 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:50.398+0000 7f064ef88700 1 -- 172.21.15.29:0/1678810009 wait complete. 2023-12-01T22:10:50.676 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:50 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:10:48.872353+0000 mgr.smithi029.loxrfu (mgr.14116) 8 : audit [DBG] from='client.14134 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi029", "addr": "172.21.15.29", "target": ["mon-mgr", ""]}]: dispatch 2023-12-01T22:10:50.676 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:50 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:10:49.528615+0000 mon.smithi029 (mon.0) 55 : audit [INF] from='mgr.14116 172.21.15.29:0/3573596141' entity='mgr.smithi029.loxrfu' 2023-12-01T22:10:50.676 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:50 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cephadm 2023-12-01T22:10:49.528970+0000 mgr.smithi029.loxrfu (mgr.14116) 9 : cephadm [INF] Added host smithi029 2023-12-01T22:10:50.676 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:50 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:10:49.529239+0000 mon.smithi029 (mon.0) 56 : audit [DBG] from='mgr.14116 172.21.15.29:0/3573596141' entity='mgr.smithi029.loxrfu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-01T22:10:50.676 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:50 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:10:50.396168+0000 mon.smithi029 (mon.0) 57 : audit [INF] from='mgr.14116 172.21.15.29:0/3573596141' entity='mgr.smithi029.loxrfu' 2023-12-01T22:10:50.740 INFO:teuthology.orchestra.run.smithi029.stderr:Deploying mgr service with default placement... 2023-12-01T22:10:50.740 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi029 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/5558ab6c-9096-11ee-95a2-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpou9lnrt4:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpxe9edifa:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply mgr 2023-12-01T22:10:51.118 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:51.115+0000 7fd7bd1ca700 1 Processor -- start 2023-12-01T22:10:51.118 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:51.115+0000 7fd7bd1ca700 1 -- start start 2023-12-01T22:10:51.118 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:51.115+0000 7fd7bd1ca700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7b80703f0 0x7fd7b80fe920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-01T22:10:51.118 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:51.115+0000 7fd7bd1ca700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7b80fee10 con 0x7fd7b80703f0 2023-12-01T22:10:51.118 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:51.115+0000 7fd7b6d9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7b80703f0 0x7fd7b80fe920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:10:51.118 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:51.116+0000 7fd7b6d9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7b80703f0 0x7fd7b80fe920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60700/0 (socket says 172.21.15.29:60700) 2023-12-01T22:10:51.119 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:51.116+0000 7fd7b6d9d700 1 -- 172.21.15.29:0/918854638 learned_addr learned my addr 172.21.15.29:0/918854638 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-12-01T22:10:51.119 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:51.116+0000 7fd7b6d9d700 1 -- 172.21.15.29:0/918854638 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7b80fef50 con 0x7fd7b80703f0 2023-12-01T22:10:51.119 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:51.116+0000 7fd7b6d9d700 1 --2- 172.21.15.29:0/918854638 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7b80703f0 0x7fd7b80fe920 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7fd7a000b5f0 tx=0x7fd7a0006ee0).ready entity=mon.0 client_cookie=524e80f776b14911 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:10:51.120 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:51.116+0000 7fd7b5d9b700 1 -- 172.21.15.29:0/918854638 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd7a000d600 con 0x7fd7b80703f0 2023-12-01T22:10:51.120 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:51.116+0000 7fd7b5d9b700 1 -- 172.21.15.29:0/918854638 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fd7a000dbd0 con 0x7fd7b80703f0 2023-12-01T22:10:51.120 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:51.116+0000 7fd7b5d9b700 1 -- 172.21.15.29:0/918854638 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd7a0011c00 con 0x7fd7b80703f0 2023-12-01T22:10:51.120 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:51.117+0000 7fd7bd1ca700 1 -- 172.21.15.29:0/918854638 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7b80703f0 msgr2=0x7fd7b80fe920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:10:51.120 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:51.117+0000 7fd7bd1ca700 1 --2- 172.21.15.29:0/918854638 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7b80703f0 0x7fd7b80fe920 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7fd7a000b5f0 tx=0x7fd7a0006ee0).stop 2023-12-01T22:10:51.120 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:51.117+0000 7fd7bd1ca700 1 -- 172.21.15.29:0/918854638 shutdown_connections 2023-12-01T22:10:51.120 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:51.117+0000 7fd7bd1ca700 1 --2- 172.21.15.29:0/918854638 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7b80703f0 0x7fd7b80fe920 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:10:51.120 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:51.117+0000 7fd7bd1ca700 1 -- 172.21.15.29:0/918854638 >> 172.21.15.29:0/918854638 conn(0x7fd7b80f63c0 msgr2=0x7fd7b80f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:10:51.120 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:51.117+0000 7fd7bd1ca700 1 -- 172.21.15.29:0/918854638 shutdown_connections 2023-12-01T22:10:51.120 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:51.117+0000 7fd7bd1ca700 1 -- 172.21.15.29:0/918854638 wait complete. 2023-12-01T22:10:51.120 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:51.118+0000 7fd7bd1ca700 1 Processor -- start 2023-12-01T22:10:51.120 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:51.118+0000 7fd7bd1ca700 1 -- start start 2023-12-01T22:10:51.121 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:51.118+0000 7fd7bd1ca700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7b80703f0 0x7fd7b80fc3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:10:51.121 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:51.118+0000 7fd7bd1ca700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7b80fa820 con 0x7fd7b80703f0 2023-12-01T22:10:51.121 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:51.118+0000 7fd7b6d9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7b80703f0 0x7fd7b80fc3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:10:51.121 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:51.118+0000 7fd7b6d9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7b80703f0 0x7fd7b80fc3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60716/0 (socket says 172.21.15.29:60716) 2023-12-01T22:10:51.122 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:51.118+0000 7fd7b6d9d700 1 -- 172.21.15.29:0/39436413 learned_addr learned my addr 172.21.15.29:0/39436413 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-12-01T22:10:51.122 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:51.118+0000 7fd7b6d9d700 1 -- 172.21.15.29:0/39436413 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7a000b200 con 0x7fd7b80703f0 2023-12-01T22:10:51.122 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:51.118+0000 7fd7b6d9d700 1 --2- 172.21.15.29:0/39436413 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7b80703f0 0x7fd7b80fc3a0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7fd7a000e040 tx=0x7fd7a0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:10:51.122 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:51.119+0000 7fd7af7fe700 1 -- 172.21.15.29:0/39436413 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd7a000d9f0 con 0x7fd7b80703f0 2023-12-01T22:10:51.122 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:51.119+0000 7fd7af7fe700 1 -- 172.21.15.29:0/39436413 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fd7a001a9f0 con 0x7fd7b80703f0 2023-12-01T22:10:51.122 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:51.119+0000 7fd7bd1ca700 1 -- 172.21.15.29:0/39436413 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7b80fa9c0 con 0x7fd7b80703f0 2023-12-01T22:10:51.122 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:51.119+0000 7fd7af7fe700 1 -- 172.21.15.29:0/39436413 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd7a001b8c0 con 0x7fd7b80703f0 2023-12-01T22:10:51.122 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:51.119+0000 7fd7bd1ca700 1 -- 172.21.15.29:0/39436413 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7b80fadd0 con 0x7fd7b80703f0 2023-12-01T22:10:51.122 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:51.120+0000 7fd7af7fe700 1 -- 172.21.15.29:0/39436413 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fd7a000f070 con 0x7fd7b80703f0 2023-12-01T22:10:51.123 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:51.120+0000 7fd7bd1ca700 1 -- 172.21.15.29:0/39436413 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7b8045ad0 con 0x7fd7b80703f0 2023-12-01T22:10:51.123 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:51.120+0000 7fd7af7fe700 1 --2- 172.21.15.29:0/39436413 >> [v2:172.21.15.29:6800/3122114772,v1:172.21.15.29:6801/3122114772] conn(0x7fd79802fd50 0x7fd798032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:10:51.123 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:51.120+0000 7fd7af7fe700 1 -- 172.21.15.29:0/39436413 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fd7a00412f0 con 0x7fd7b80703f0 2023-12-01T22:10:51.123 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:51.120+0000 7fd7b659c700 1 --2- 172.21.15.29:0/39436413 >> [v2:172.21.15.29:6800/3122114772,v1:172.21.15.29:6801/3122114772] conn(0x7fd79802fd50 0x7fd798032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:10:51.123 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:51.120+0000 7fd7b659c700 1 --2- 172.21.15.29:0/39436413 >> [v2:172.21.15.29:6800/3122114772,v1:172.21.15.29:6801/3122114772] conn(0x7fd79802fd50 0x7fd798032210 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7fd7a80096a0 tx=0x7fd7a8006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:10:51.126 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:51.124+0000 7fd7af7fe700 1 -- 172.21.15.29:0/39436413 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd7a0022af0 con 0x7fd7b80703f0 2023-12-01T22:10:51.321 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:51.318+0000 7fd7bd1ca700 1 -- 172.21.15.29:0/39436413 --> [v2:172.21.15.29:6800/3122114772,v1:172.21.15.29:6801/3122114772] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7fd7b80fb010 con 0x7fd79802fd50 2023-12-01T22:10:51.324 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:51.323+0000 7fd7af7fe700 1 -- 172.21.15.29:0/39436413 <== mgr.14116 v2:172.21.15.29:6800/3122114772 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fd7b80fb010 con 0x7fd79802fd50 2023-12-01T22:10:51.324 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: Scheduled mgr update... 2023-12-01T22:10:51.327 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:51.324+0000 7fd7bd1ca700 1 -- 172.21.15.29:0/39436413 >> [v2:172.21.15.29:6800/3122114772,v1:172.21.15.29:6801/3122114772] conn(0x7fd79802fd50 msgr2=0x7fd798032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:10:51.327 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:51.324+0000 7fd7bd1ca700 1 --2- 172.21.15.29:0/39436413 >> [v2:172.21.15.29:6800/3122114772,v1:172.21.15.29:6801/3122114772] conn(0x7fd79802fd50 0x7fd798032210 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7fd7a80096a0 tx=0x7fd7a8006b40).stop 2023-12-01T22:10:51.327 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:51.324+0000 7fd7bd1ca700 1 -- 172.21.15.29:0/39436413 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7b80703f0 msgr2=0x7fd7b80fc3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:10:51.327 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:51.324+0000 7fd7bd1ca700 1 --2- 172.21.15.29:0/39436413 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7b80703f0 0x7fd7b80fc3a0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7fd7a000e040 tx=0x7fd7a0006ee0).stop 2023-12-01T22:10:51.327 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:51.325+0000 7fd7bd1ca700 1 -- 172.21.15.29:0/39436413 shutdown_connections 2023-12-01T22:10:51.327 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:51.325+0000 7fd7bd1ca700 1 --2- 172.21.15.29:0/39436413 >> [v2:172.21.15.29:6800/3122114772,v1:172.21.15.29:6801/3122114772] conn(0x7fd79802fd50 0x7fd798032210 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:10:51.327 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:51.325+0000 7fd7bd1ca700 1 --2- 172.21.15.29:0/39436413 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7b80703f0 0x7fd7b80fc3a0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:10:51.328 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:51.325+0000 7fd7bd1ca700 1 -- 172.21.15.29:0/39436413 >> 172.21.15.29:0/39436413 conn(0x7fd7b80f63c0 msgr2=0x7fd7b80f7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:10:51.328 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:51.325+0000 7fd7bd1ca700 1 -- 172.21.15.29:0/39436413 shutdown_connections 2023-12-01T22:10:51.328 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:51.325+0000 7fd7bd1ca700 1 -- 172.21.15.29:0/39436413 wait complete. 2023-12-01T22:10:51.680 INFO:teuthology.orchestra.run.smithi029.stderr:Deploying crash service with default placement... 2023-12-01T22:10:51.681 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi029 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/5558ab6c-9096-11ee-95a2-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpou9lnrt4:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpxe9edifa:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply crash 2023-12-01T22:10:51.683 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:51 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:10:50.393040+0000 mgr.smithi029.loxrfu (mgr.14116) 10 : audit [DBG] from='client.14136 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2023-12-01T22:10:51.684 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:51 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cephadm 2023-12-01T22:10:50.393870+0000 mgr.smithi029.loxrfu (mgr.14116) 11 : cephadm [INF] Saving service mon spec with placement count:5 2023-12-01T22:10:51.684 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:51 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:10:51.323718+0000 mon.smithi029 (mon.0) 58 : audit [INF] from='mgr.14116 172.21.15.29:0/3573596141' entity='mgr.smithi029.loxrfu' 2023-12-01T22:10:52.036 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:52.032+0000 7fb22495a700 1 Processor -- start 2023-12-01T22:10:52.036 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:52.032+0000 7fb22495a700 1 -- start start 2023-12-01T22:10:52.036 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:52.032+0000 7fb22495a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2200fab80 0x7fb2200fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-01T22:10:52.036 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:52.032+0000 7fb22495a700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2200fb490 con 0x7fb2200fab80 2023-12-01T22:10:52.036 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:52.033+0000 7fb21e59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2200fab80 0x7fb2200fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:10:52.036 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:52.033+0000 7fb21e59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2200fab80 0x7fb2200fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60730/0 (socket says 172.21.15.29:60730) 2023-12-01T22:10:52.037 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:52.033+0000 7fb21e59c700 1 -- 172.21.15.29:0/1157177044 learned_addr learned my addr 172.21.15.29:0/1157177044 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-12-01T22:10:52.037 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:52.033+0000 7fb21e59c700 1 -- 172.21.15.29:0/1157177044 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2200fb5d0 con 0x7fb2200fab80 2023-12-01T22:10:52.037 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:52.033+0000 7fb21e59c700 1 --2- 172.21.15.29:0/1157177044 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2200fab80 0x7fb2200fafa0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7fb21000b5f0 tx=0x7fb210006ee0).ready entity=mon.0 client_cookie=b20f2162b677d1ce server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:10:52.037 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:52.033+0000 7fb21dd9b700 1 -- 172.21.15.29:0/1157177044 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb21000d600 con 0x7fb2200fab80 2023-12-01T22:10:52.037 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:52.033+0000 7fb21dd9b700 1 -- 172.21.15.29:0/1157177044 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fb21000dbd0 con 0x7fb2200fab80 2023-12-01T22:10:52.037 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:52.033+0000 7fb21dd9b700 1 -- 172.21.15.29:0/1157177044 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb210011c00 con 0x7fb2200fab80 2023-12-01T22:10:52.037 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:52.034+0000 7fb22495a700 1 -- 172.21.15.29:0/1157177044 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2200fab80 msgr2=0x7fb2200fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:10:52.037 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:52.034+0000 7fb22495a700 1 --2- 172.21.15.29:0/1157177044 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2200fab80 0x7fb2200fafa0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7fb21000b5f0 tx=0x7fb210006ee0).stop 2023-12-01T22:10:52.037 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:52.034+0000 7fb22495a700 1 -- 172.21.15.29:0/1157177044 shutdown_connections 2023-12-01T22:10:52.037 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:52.034+0000 7fb22495a700 1 --2- 172.21.15.29:0/1157177044 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2200fab80 0x7fb2200fafa0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:10:52.037 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:52.034+0000 7fb22495a700 1 -- 172.21.15.29:0/1157177044 >> 172.21.15.29:0/1157177044 conn(0x7fb2200f63c0 msgr2=0x7fb2200f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:10:52.037 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:52.034+0000 7fb22495a700 1 -- 172.21.15.29:0/1157177044 shutdown_connections 2023-12-01T22:10:52.037 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:52.034+0000 7fb22495a700 1 -- 172.21.15.29:0/1157177044 wait complete. 2023-12-01T22:10:52.039 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:52.035+0000 7fb22495a700 1 Processor -- start 2023-12-01T22:10:52.040 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:52.035+0000 7fb22495a700 1 -- start start 2023-12-01T22:10:52.040 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:52.035+0000 7fb22495a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2200fab80 0x7fb22010aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:10:52.040 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:52.035+0000 7fb22495a700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb22010af10 con 0x7fb2200fab80 2023-12-01T22:10:52.040 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:52.035+0000 7fb21e59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2200fab80 0x7fb22010aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:10:52.040 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:52.035+0000 7fb21e59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2200fab80 0x7fb22010aa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60734/0 (socket says 172.21.15.29:60734) 2023-12-01T22:10:52.040 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:52.035+0000 7fb21e59c700 1 -- 172.21.15.29:0/1839658196 learned_addr learned my addr 172.21.15.29:0/1839658196 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-12-01T22:10:52.040 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:52.036+0000 7fb21e59c700 1 -- 172.21.15.29:0/1839658196 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb21000b200 con 0x7fb2200fab80 2023-12-01T22:10:52.040 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:52.036+0000 7fb21e59c700 1 --2- 172.21.15.29:0/1839658196 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2200fab80 0x7fb22010aa20 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7fb21000e040 tx=0x7fb210006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:10:52.040 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:52.036+0000 7fb2177fe700 1 -- 172.21.15.29:0/1839658196 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb21001b410 con 0x7fb2200fab80 2023-12-01T22:10:52.040 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:52.036+0000 7fb2177fe700 1 -- 172.21.15.29:0/1839658196 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fb21001b9e0 con 0x7fb2200fab80 2023-12-01T22:10:52.040 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:52.036+0000 7fb2177fe700 1 -- 172.21.15.29:0/1839658196 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb21001a820 con 0x7fb2200fab80 2023-12-01T22:10:52.040 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:52.036+0000 7fb22495a700 1 -- 172.21.15.29:0/1839658196 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb22010b110 con 0x7fb2200fab80 2023-12-01T22:10:52.040 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:52.036+0000 7fb22495a700 1 -- 172.21.15.29:0/1839658196 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb22010b520 con 0x7fb2200fab80 2023-12-01T22:10:52.040 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:52.037+0000 7fb2177fe700 1 -- 172.21.15.29:0/1839658196 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fb21000f070 con 0x7fb2200fab80 2023-12-01T22:10:52.041 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:52.037+0000 7fb22495a700 1 -- 172.21.15.29:0/1839658196 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb220045ad0 con 0x7fb2200fab80 2023-12-01T22:10:52.041 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:52.037+0000 7fb2177fe700 1 --2- 172.21.15.29:0/1839658196 >> [v2:172.21.15.29:6800/3122114772,v1:172.21.15.29:6801/3122114772] conn(0x7fb20002fe00 0x7fb2000322c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:10:52.041 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:52.037+0000 7fb2177fe700 1 -- 172.21.15.29:0/1839658196 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fb210041430 con 0x7fb2200fab80 2023-12-01T22:10:52.041 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:52.038+0000 7fb215d9b700 1 --2- 172.21.15.29:0/1839658196 >> [v2:172.21.15.29:6800/3122114772,v1:172.21.15.29:6801/3122114772] conn(0x7fb20002fe00 0x7fb2000322c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:10:52.041 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:52.038+0000 7fb215d9b700 1 --2- 172.21.15.29:0/1839658196 >> [v2:172.21.15.29:6800/3122114772,v1:172.21.15.29:6801/3122114772] conn(0x7fb20002fe00 0x7fb2000322c0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7fb2080096a0 tx=0x7fb208006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:10:52.043 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:52.042+0000 7fb2177fe700 1 -- 172.21.15.29:0/1839658196 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb210022d20 con 0x7fb2200fab80 2023-12-01T22:10:52.240 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:52.238+0000 7fb22495a700 1 -- 172.21.15.29:0/1839658196 --> [v2:172.21.15.29:6800/3122114772,v1:172.21.15.29:6801/3122114772] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}) v1 -- 0x7fb22010be00 con 0x7fb20002fe00 2023-12-01T22:10:52.244 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:52.243+0000 7fb2177fe700 1 -- 172.21.15.29:0/1839658196 <== mgr.14116 v2:172.21.15.29:6800/3122114772 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7fb22010be00 con 0x7fb20002fe00 2023-12-01T22:10:52.244 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: Scheduled crash update... 2023-12-01T22:10:52.247 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:52.244+0000 7fb22495a700 1 -- 172.21.15.29:0/1839658196 >> [v2:172.21.15.29:6800/3122114772,v1:172.21.15.29:6801/3122114772] conn(0x7fb20002fe00 msgr2=0x7fb2000322c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:10:52.247 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:52.244+0000 7fb22495a700 1 --2- 172.21.15.29:0/1839658196 >> [v2:172.21.15.29:6800/3122114772,v1:172.21.15.29:6801/3122114772] conn(0x7fb20002fe00 0x7fb2000322c0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7fb2080096a0 tx=0x7fb208006b40).stop 2023-12-01T22:10:52.247 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:52.244+0000 7fb22495a700 1 -- 172.21.15.29:0/1839658196 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2200fab80 msgr2=0x7fb22010aa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:10:52.247 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:52.244+0000 7fb22495a700 1 --2- 172.21.15.29:0/1839658196 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2200fab80 0x7fb22010aa20 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7fb21000e040 tx=0x7fb210006ee0).stop 2023-12-01T22:10:52.247 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:52.244+0000 7fb22495a700 1 -- 172.21.15.29:0/1839658196 shutdown_connections 2023-12-01T22:10:52.247 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:52.244+0000 7fb22495a700 1 --2- 172.21.15.29:0/1839658196 >> [v2:172.21.15.29:6800/3122114772,v1:172.21.15.29:6801/3122114772] conn(0x7fb20002fe00 0x7fb2000322c0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:10:52.247 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:52.244+0000 7fb22495a700 1 --2- 172.21.15.29:0/1839658196 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2200fab80 0x7fb22010aa20 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:10:52.247 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:52.244+0000 7fb22495a700 1 -- 172.21.15.29:0/1839658196 >> 172.21.15.29:0/1839658196 conn(0x7fb2200f63c0 msgr2=0x7fb220070090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:10:52.247 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:52.244+0000 7fb22495a700 1 -- 172.21.15.29:0/1839658196 shutdown_connections 2023-12-01T22:10:52.247 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:52.245+0000 7fb22495a700 1 -- 172.21.15.29:0/1839658196 wait complete. 2023-12-01T22:10:52.588 INFO:teuthology.orchestra.run.smithi029.stderr:Enabling mgr prometheus module... 2023-12-01T22:10:52.588 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi029 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/5558ab6c-9096-11ee-95a2-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpou9lnrt4:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpxe9edifa:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable prometheus 2023-12-01T22:10:52.971 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:52.969+0000 7f11f7338700 1 Processor -- start 2023-12-01T22:10:52.971 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:52.969+0000 7f11f7338700 1 -- start start 2023-12-01T22:10:52.971 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:52.969+0000 7f11f7338700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f11f0070440 0x7f11f0100d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-01T22:10:52.971 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:52.969+0000 7f11f7338700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11f00fc740 con 0x7f11f0070440 2023-12-01T22:10:52.972 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:52.969+0000 7f11f50d4700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f11f0070440 0x7f11f0100d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:10:52.972 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:52.969+0000 7f11f50d4700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f11f0070440 0x7f11f0100d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60744/0 (socket says 172.21.15.29:60744) 2023-12-01T22:10:52.973 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:52.969+0000 7f11f50d4700 1 -- 172.21.15.29:0/1885571990 learned_addr learned my addr 172.21.15.29:0/1885571990 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-12-01T22:10:52.973 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:52.970+0000 7f11f50d4700 1 -- 172.21.15.29:0/1885571990 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11f0101240 con 0x7f11f0070440 2023-12-01T22:10:52.973 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:52.970+0000 7f11f50d4700 1 --2- 172.21.15.29:0/1885571990 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f11f0070440 0x7f11f0100d50 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f11ec00b5f0 tx=0x7f11ec006ee0).ready entity=mon.0 client_cookie=4eab4bffe5694f4b server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:10:52.973 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:52.970+0000 7f11dffff700 1 -- 172.21.15.29:0/1885571990 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f11ec00d600 con 0x7f11f0070440 2023-12-01T22:10:52.973 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:52.970+0000 7f11dffff700 1 -- 172.21.15.29:0/1885571990 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f11ec00dbd0 con 0x7f11f0070440 2023-12-01T22:10:52.973 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:52.970+0000 7f11dffff700 1 -- 172.21.15.29:0/1885571990 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f11ec011c00 con 0x7f11f0070440 2023-12-01T22:10:52.973 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:52.971+0000 7f11f7338700 1 -- 172.21.15.29:0/1885571990 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f11f0070440 msgr2=0x7f11f0100d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:10:52.974 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:52.971+0000 7f11f7338700 1 --2- 172.21.15.29:0/1885571990 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f11f0070440 0x7f11f0100d50 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f11ec00b5f0 tx=0x7f11ec006ee0).stop 2023-12-01T22:10:52.974 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:52.971+0000 7f11f7338700 1 -- 172.21.15.29:0/1885571990 shutdown_connections 2023-12-01T22:10:52.974 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:52.971+0000 7f11f7338700 1 --2- 172.21.15.29:0/1885571990 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f11f0070440 0x7f11f0100d50 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:10:52.974 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:52.971+0000 7f11f7338700 1 -- 172.21.15.29:0/1885571990 >> 172.21.15.29:0/1885571990 conn(0x7f11f00f63b0 msgr2=0x7f11f00f87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:10:52.974 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:52.971+0000 7f11f7338700 1 -- 172.21.15.29:0/1885571990 shutdown_connections 2023-12-01T22:10:52.974 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:52.971+0000 7f11f7338700 1 -- 172.21.15.29:0/1885571990 wait complete. 2023-12-01T22:10:52.976 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:52.972+0000 7f11f7338700 1 Processor -- start 2023-12-01T22:10:52.976 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:52.972+0000 7f11f7338700 1 -- start start 2023-12-01T22:10:52.976 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:52.972+0000 7f11f7338700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f11f0070440 0x7f11f010cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:10:52.976 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:52.972+0000 7f11f7338700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11f010d0e0 con 0x7f11f0070440 2023-12-01T22:10:52.976 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:52.972+0000 7f11f50d4700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f11f0070440 0x7f11f010cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:10:52.976 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:52.972+0000 7f11f50d4700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f11f0070440 0x7f11f010cbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60754/0 (socket says 172.21.15.29:60754) 2023-12-01T22:10:52.976 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:52.972+0000 7f11f50d4700 1 -- 172.21.15.29:0/3204578272 learned_addr learned my addr 172.21.15.29:0/3204578272 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-12-01T22:10:52.977 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:52.973+0000 7f11f50d4700 1 -- 172.21.15.29:0/3204578272 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11ec00b200 con 0x7f11f0070440 2023-12-01T22:10:52.977 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:52.973+0000 7f11f50d4700 1 --2- 172.21.15.29:0/3204578272 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f11f0070440 0x7f11f010cbf0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f11ec00b190 tx=0x7f11ec006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:10:52.977 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:52.973+0000 7f11ddffb700 1 -- 172.21.15.29:0/3204578272 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f11ec01b410 con 0x7f11f0070440 2023-12-01T22:10:52.977 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:52.973+0000 7f11ddffb700 1 -- 172.21.15.29:0/3204578272 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f11ec01b9e0 con 0x7f11f0070440 2023-12-01T22:10:52.977 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:52.973+0000 7f11f7338700 1 -- 172.21.15.29:0/3204578272 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11f010d2e0 con 0x7f11f0070440 2023-12-01T22:10:52.977 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:52.973+0000 7f11ddffb700 1 -- 172.21.15.29:0/3204578272 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f11ec01a880 con 0x7f11f0070440 2023-12-01T22:10:52.977 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:52.973+0000 7f11f7338700 1 -- 172.21.15.29:0/3204578272 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11f010d6f0 con 0x7f11f0070440 2023-12-01T22:10:52.977 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:52.974+0000 7f11ddffb700 1 -- 172.21.15.29:0/3204578272 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f11ec00f070 con 0x7f11f0070440 2023-12-01T22:10:52.977 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:52.974+0000 7f11f7338700 1 -- 172.21.15.29:0/3204578272 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11f0045ad0 con 0x7f11f0070440 2023-12-01T22:10:52.977 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:52.974+0000 7f11ddffb700 1 --2- 172.21.15.29:0/3204578272 >> [v2:172.21.15.29:6800/3122114772,v1:172.21.15.29:6801/3122114772] conn(0x7f11d802fdb0 0x7f11d8032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:10:52.977 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:52.974+0000 7f11ddffb700 1 -- 172.21.15.29:0/3204578272 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f11ec0411e0 con 0x7f11f0070440 2023-12-01T22:10:52.977 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:52.974+0000 7f11f48d3700 1 --2- 172.21.15.29:0/3204578272 >> [v2:172.21.15.29:6800/3122114772,v1:172.21.15.29:6801/3122114772] conn(0x7f11d802fdb0 0x7f11d8032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:10:52.978 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:52.975+0000 7f11f48d3700 1 --2- 172.21.15.29:0/3204578272 >> [v2:172.21.15.29:6800/3122114772,v1:172.21.15.29:6801/3122114772] conn(0x7f11d802fdb0 0x7f11d8032270 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f11e00096a0 tx=0x7f11e0006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:10:52.980 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:52.979+0000 7f11ddffb700 1 -- 172.21.15.29:0/3204578272 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f11ec0224f0 con 0x7f11f0070440 2023-12-01T22:10:53.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:52 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:10:51.320684+0000 mgr.smithi029.loxrfu (mgr.14116) 12 : audit [DBG] from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2023-12-01T22:10:53.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:52 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cephadm 2023-12-01T22:10:51.321281+0000 mgr.smithi029.loxrfu (mgr.14116) 13 : cephadm [INF] Saving service mgr spec with placement count:2 2023-12-01T22:10:53.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:52 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:10:52.243654+0000 mon.smithi029 (mon.0) 59 : audit [INF] from='mgr.14116 172.21.15.29:0/3573596141' entity='mgr.smithi029.loxrfu' 2023-12-01T22:10:53.187 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:53.185+0000 7f11f7338700 1 -- 172.21.15.29:0/3204578272 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "prometheus"} v 0) v1 -- 0x7f11f0101da0 con 0x7f11f0070440 2023-12-01T22:10:54.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:53 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:10:52.240503+0000 mgr.smithi029.loxrfu (mgr.14116) 14 : audit [DBG] from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2023-12-01T22:10:54.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:53 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cephadm 2023-12-01T22:10:52.241235+0000 mgr.smithi029.loxrfu (mgr.14116) 15 : cephadm [INF] Saving service crash spec with placement * 2023-12-01T22:10:54.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:53 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:10:53.186567+0000 mon.smithi029 (mon.0) 60 : audit [INF] from='client.? 172.21.15.29:0/3204578272' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2023-12-01T22:10:54.180 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:54.178+0000 7f11ddffb700 1 -- 172.21.15.29:0/3204578272 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 9) v1 ==== 36134+0+0 (secure 0 0 0) 0x7f11ec025460 con 0x7f11f0070440 2023-12-01T22:10:54.182 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:54.180+0000 7f11ddffb700 1 -- 172.21.15.29:0/3204578272 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "prometheus"}]=0 v9) v1 ==== 89+0+0 (secure 0 0 0) 0x7f11ec019770 con 0x7f11f0070440 2023-12-01T22:10:54.185 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:54.183+0000 7f11f7338700 1 -- 172.21.15.29:0/3204578272 >> [v2:172.21.15.29:6800/3122114772,v1:172.21.15.29:6801/3122114772] conn(0x7f11d802fdb0 msgr2=0x7f11d8032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:10:54.185 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:54.183+0000 7f11f7338700 1 --2- 172.21.15.29:0/3204578272 >> [v2:172.21.15.29:6800/3122114772,v1:172.21.15.29:6801/3122114772] conn(0x7f11d802fdb0 0x7f11d8032270 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f11e00096a0 tx=0x7f11e0006b40).stop 2023-12-01T22:10:54.185 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:54.183+0000 7f11f7338700 1 -- 172.21.15.29:0/3204578272 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f11f0070440 msgr2=0x7f11f010cbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:10:54.185 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:54.183+0000 7f11f7338700 1 --2- 172.21.15.29:0/3204578272 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f11f0070440 0x7f11f010cbf0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f11ec00b190 tx=0x7f11ec006ee0).stop 2023-12-01T22:10:54.185 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:54.183+0000 7f11f7338700 1 -- 172.21.15.29:0/3204578272 shutdown_connections 2023-12-01T22:10:54.185 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:54.183+0000 7f11f7338700 1 --2- 172.21.15.29:0/3204578272 >> [v2:172.21.15.29:6800/3122114772,v1:172.21.15.29:6801/3122114772] conn(0x7f11d802fdb0 0x7f11d8032270 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:10:54.186 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:54.183+0000 7f11f7338700 1 --2- 172.21.15.29:0/3204578272 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f11f0070440 0x7f11f010cbf0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:10:54.186 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:54.183+0000 7f11f7338700 1 -- 172.21.15.29:0/3204578272 >> 172.21.15.29:0/3204578272 conn(0x7f11f00f63b0 msgr2=0x7f11f00f7e20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:10:54.186 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:54.184+0000 7f11f7338700 1 -- 172.21.15.29:0/3204578272 shutdown_connections 2023-12-01T22:10:54.186 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:54.184+0000 7f11f7338700 1 -- 172.21.15.29:0/3204578272 wait complete. 2023-12-01T22:10:54.555 INFO:teuthology.orchestra.run.smithi029.stderr:Deploying prometheus service with default placement... 2023-12-01T22:10:54.555 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi029 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/5558ab6c-9096-11ee-95a2-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpou9lnrt4:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpxe9edifa:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply prometheus 2023-12-01T22:10:54.777 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:54 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:10:53.176563+0000 mgr.smithi029.loxrfu (mgr.14116) 16 : cluster [DBG] pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:10:54.777 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:54 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:10:54.181293+0000 mon.smithi029 (mon.0) 61 : audit [INF] from='client.? 172.21.15.29:0/3204578272' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2023-12-01T22:10:54.777 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:10:54 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:10:54.181352+0000 mon.smithi029 (mon.0) 62 : cluster [DBG] mgrmap e9: smithi029.loxrfu(active, since 21s) 2023-12-01T22:10:54.913 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:54.911+0000 7f9cdefad700 1 Processor -- start 2023-12-01T22:10:54.913 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:54.911+0000 7f9cdefad700 1 -- start start 2023-12-01T22:10:54.913 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:54.911+0000 7f9cdefad700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cd80703f0 0x7f9cd8100b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-01T22:10:54.914 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:54.911+0000 7f9cdefad700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cd80fc560 con 0x7f9cd80703f0 2023-12-01T22:10:54.914 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:54.911+0000 7f9cddfab700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cd80703f0 0x7f9cd8100b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:10:54.915 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:54.911+0000 7f9cddfab700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cd80703f0 0x7f9cd8100b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38042/0 (socket says 172.21.15.29:38042) 2023-12-01T22:10:54.915 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:54.911+0000 7f9cddfab700 1 -- 172.21.15.29:0/3105619311 learned_addr learned my addr 172.21.15.29:0/3105619311 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-12-01T22:10:54.915 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:54.912+0000 7f9cddfab700 1 -- 172.21.15.29:0/3105619311 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cd8101060 con 0x7f9cd80703f0 2023-12-01T22:10:54.915 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:54.912+0000 7f9cddfab700 1 --2- 172.21.15.29:0/3105619311 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cd80703f0 0x7f9cd8100b70 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f9cd400b5f0 tx=0x7f9cd4006ee0).ready entity=mon.0 client_cookie=e896c3bc61c1cab4 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:10:54.915 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:54.912+0000 7f9cdcfa9700 1 -- 172.21.15.29:0/3105619311 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9cd400d600 con 0x7f9cd80703f0 2023-12-01T22:10:54.915 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:54.912+0000 7f9cdcfa9700 1 -- 172.21.15.29:0/3105619311 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f9cd400dbd0 con 0x7f9cd80703f0 2023-12-01T22:10:54.915 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:54.912+0000 7f9cdcfa9700 1 -- 172.21.15.29:0/3105619311 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9cd4011c00 con 0x7f9cd80703f0 2023-12-01T22:10:54.919 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:54.913+0000 7f9cdefad700 1 -- 172.21.15.29:0/3105619311 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cd80703f0 msgr2=0x7f9cd8100b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:10:54.919 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:54.913+0000 7f9cdefad700 1 --2- 172.21.15.29:0/3105619311 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cd80703f0 0x7f9cd8100b70 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f9cd400b5f0 tx=0x7f9cd4006ee0).stop 2023-12-01T22:10:54.919 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:54.913+0000 7f9cdefad700 1 -- 172.21.15.29:0/3105619311 shutdown_connections 2023-12-01T22:10:54.919 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:54.913+0000 7f9cdefad700 1 --2- 172.21.15.29:0/3105619311 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cd80703f0 0x7f9cd8100b70 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:10:54.919 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:54.913+0000 7f9cdefad700 1 -- 172.21.15.29:0/3105619311 >> 172.21.15.29:0/3105619311 conn(0x7f9cd80f6210 msgr2=0x7f9cd80f8630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:10:54.919 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:54.913+0000 7f9cdefad700 1 -- 172.21.15.29:0/3105619311 shutdown_connections 2023-12-01T22:10:54.919 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:54.913+0000 7f9cdefad700 1 -- 172.21.15.29:0/3105619311 wait complete. 2023-12-01T22:10:54.919 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:54.913+0000 7f9cdefad700 1 Processor -- start 2023-12-01T22:10:54.919 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:54.913+0000 7f9cdefad700 1 -- start start 2023-12-01T22:10:54.920 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:54.914+0000 7f9cdefad700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cd80703f0 0x7f9cd810ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:10:54.920 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:54.914+0000 7f9cdefad700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cd810cf70 con 0x7f9cd80703f0 2023-12-01T22:10:54.920 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:54.914+0000 7f9cddfab700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cd80703f0 0x7f9cd810ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:10:54.920 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:54.914+0000 7f9cddfab700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cd80703f0 0x7f9cd810ca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38046/0 (socket says 172.21.15.29:38046) 2023-12-01T22:10:54.920 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:54.914+0000 7f9cddfab700 1 -- 172.21.15.29:0/98749328 learned_addr learned my addr 172.21.15.29:0/98749328 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-12-01T22:10:54.920 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:54.914+0000 7f9cddfab700 1 -- 172.21.15.29:0/98749328 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cd400b200 con 0x7f9cd80703f0 2023-12-01T22:10:54.920 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:54.914+0000 7f9cddfab700 1 --2- 172.21.15.29:0/98749328 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cd80703f0 0x7f9cd810ca80 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f9cd400e040 tx=0x7f9cd4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:10:54.920 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:54.914+0000 7f9cc67fc700 1 -- 172.21.15.29:0/98749328 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9cd400d9f0 con 0x7f9cd80703f0 2023-12-01T22:10:54.920 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:54.914+0000 7f9cc67fc700 1 -- 172.21.15.29:0/98749328 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f9cd401a9f0 con 0x7f9cd80703f0 2023-12-01T22:10:54.920 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:54.915+0000 7f9cdefad700 1 -- 172.21.15.29:0/98749328 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cd810d170 con 0x7f9cd80703f0 2023-12-01T22:10:54.920 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:54.915+0000 7f9cc67fc700 1 -- 172.21.15.29:0/98749328 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9cd401b8c0 con 0x7f9cd80703f0 2023-12-01T22:10:54.920 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:54.915+0000 7f9cdefad700 1 -- 172.21.15.29:0/98749328 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cd810d580 con 0x7f9cd80703f0 2023-12-01T22:10:54.920 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:54.915+0000 7f9cc67fc700 1 -- 172.21.15.29:0/98749328 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 9) v1 ==== 36134+0+0 (secure 0 0 0) 0x7f9cd400f070 con 0x7f9cd80703f0 2023-12-01T22:10:54.920 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:54.915+0000 7f9cc67fc700 1 --2- 172.21.15.29:0/98749328 >> [v2:172.21.15.29:6800/3122114772,v1:172.21.15.29:6801/3122114772] conn(0x7f9cc002fdb0 0x7f9cc0032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:10:54.921 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:54.916+0000 7f9cc67fc700 1 -- 172.21.15.29:0/98749328 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f9cd4042140 con 0x7f9cd80703f0 2023-12-01T22:10:54.921 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:54.916+0000 7f9cdd7aa700 1 -- 172.21.15.29:0/98749328 >> [v2:172.21.15.29:6800/3122114772,v1:172.21.15.29:6801/3122114772] conn(0x7f9cc002fdb0 msgr2=0x7f9cc0032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/3122114772 2023-12-01T22:10:54.921 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:54.916+0000 7f9cdd7aa700 1 --2- 172.21.15.29:0/98749328 >> [v2:172.21.15.29:6800/3122114772,v1:172.21.15.29:6801/3122114772] conn(0x7f9cc002fdb0 0x7f9cc0032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-12-01T22:10:54.921 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:54.916+0000 7f9cdefad700 1 -- 172.21.15.29:0/98749328 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cd8059100 con 0x7f9cd80703f0 2023-12-01T22:10:54.922 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:54.920+0000 7f9cc67fc700 1 -- 172.21.15.29:0/98749328 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9cd4016070 con 0x7f9cd80703f0 2023-12-01T22:10:55.117 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:55.115+0000 7f9cdefad700 1 -- 172.21.15.29:0/98749328 --> [v2:172.21.15.29:6800/3122114772,v1:172.21.15.29:6801/3122114772] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7f9cd81068b0 con 0x7f9cc002fdb0 2023-12-01T22:10:55.118 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:55.117+0000 7f9cdd7aa700 1 -- 172.21.15.29:0/98749328 >> [v2:172.21.15.29:6800/3122114772,v1:172.21.15.29:6801/3122114772] conn(0x7f9cc002fdb0 msgr2=0x7f9cc0032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/3122114772 2023-12-01T22:10:55.118 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:55.117+0000 7f9cdd7aa700 1 --2- 172.21.15.29:0/98749328 >> [v2:172.21.15.29:6800/3122114772,v1:172.21.15.29:6801/3122114772] conn(0x7f9cc002fdb0 0x7f9cc0032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-12-01T22:10:55.520 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:55.518+0000 7f9cdd7aa700 1 -- 172.21.15.29:0/98749328 >> [v2:172.21.15.29:6800/3122114772,v1:172.21.15.29:6801/3122114772] conn(0x7f9cc002fdb0 msgr2=0x7f9cc0032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/3122114772 2023-12-01T22:10:55.520 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:55.518+0000 7f9cdd7aa700 1 --2- 172.21.15.29:0/98749328 >> [v2:172.21.15.29:6800/3122114772,v1:172.21.15.29:6801/3122114772] conn(0x7f9cc002fdb0 0x7f9cc0032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-12-01T22:10:56.321 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:56.319+0000 7f9cdd7aa700 1 -- 172.21.15.29:0/98749328 >> [v2:172.21.15.29:6800/3122114772,v1:172.21.15.29:6801/3122114772] conn(0x7f9cc002fdb0 msgr2=0x7f9cc0032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/3122114772 2023-12-01T22:10:56.321 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:56.319+0000 7f9cdd7aa700 1 --2- 172.21.15.29:0/98749328 >> [v2:172.21.15.29:6800/3122114772,v1:172.21.15.29:6801/3122114772] conn(0x7f9cc002fdb0 0x7f9cc0032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-12-01T22:10:57.923 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:57.920+0000 7f9cdd7aa700 1 -- 172.21.15.29:0/98749328 >> [v2:172.21.15.29:6800/3122114772,v1:172.21.15.29:6801/3122114772] conn(0x7f9cc002fdb0 msgr2=0x7f9cc0032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/3122114772 2023-12-01T22:10:57.923 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:57.920+0000 7f9cdd7aa700 1 --2- 172.21.15.29:0/98749328 >> [v2:172.21.15.29:6800/3122114772,v1:172.21.15.29:6801/3122114772] conn(0x7f9cc002fdb0 0x7f9cc0032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 3.200000 2023-12-01T22:10:59.873 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:59.871+0000 7f9cc67fc700 1 -- 172.21.15.29:0/98749328 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 10) v1 ==== 35984+0+0 (secure 0 0 0) 0x7f9cd40257e0 con 0x7f9cd80703f0 2023-12-01T22:10:59.873 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:59.871+0000 7f9cc67fc700 1 -- 172.21.15.29:0/98749328 >> [v2:172.21.15.29:6800/3122114772,v1:172.21.15.29:6801/3122114772] conn(0x7f9cc002fdb0 msgr2=0x7f9cc0032270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-01T22:10:59.873 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:10:59.871+0000 7f9cc67fc700 1 --2- 172.21.15.29:0/98749328 >> [v2:172.21.15.29:6800/3122114772,v1:172.21.15.29:6801/3122114772] conn(0x7f9cc002fdb0 0x7f9cc0032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-12-01T22:11:00.875 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:00.872+0000 7f9cc67fc700 1 -- 172.21.15.29:0/98749328 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mgrmap(e 11) v1 ==== 36054+0+0 (secure 0 0 0) 0x7f9cd4024ca0 con 0x7f9cd80703f0 2023-12-01T22:11:00.876 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:00.873+0000 7f9cc67fc700 1 --2- 172.21.15.29:0/98749328 >> [v2:172.21.15.29:6800/1129362254,v1:172.21.15.29:6801/1129362254] conn(0x7f9cc0033c20 0x7f9cc0036010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:11:00.876 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:00.873+0000 7f9cc67fc700 1 -- 172.21.15.29:0/98749328 --> [v2:172.21.15.29:6800/1129362254,v1:172.21.15.29:6801/1129362254] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7f9cd81068b0 con 0x7f9cc0033c20 2023-12-01T22:11:00.876 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:00.873+0000 7f9cdd7aa700 1 --2- 172.21.15.29:0/98749328 >> [v2:172.21.15.29:6800/1129362254,v1:172.21.15.29:6801/1129362254] conn(0x7f9cc0033c20 0x7f9cc0036010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:11:00.876 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:00.874+0000 7f9cdd7aa700 1 --2- 172.21.15.29:0/98749328 >> [v2:172.21.15.29:6800/1129362254,v1:172.21.15.29:6801/1129362254] conn(0x7f9cc0033c20 0x7f9cc0036010 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f9cc80036d0 tx=0x7f9cc8006cb0).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:11:00.881 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:00.879+0000 7f9cc67fc700 1 -- 172.21.15.29:0/98749328 <== mgr.14144 v2:172.21.15.29:6800/1129362254 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7f9cd81068b0 con 0x7f9cc0033c20 2023-12-01T22:11:00.881 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: Scheduled prometheus update... 2023-12-01T22:11:00.884 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:00.882+0000 7f9cdefad700 1 -- 172.21.15.29:0/98749328 >> [v2:172.21.15.29:6800/1129362254,v1:172.21.15.29:6801/1129362254] conn(0x7f9cc0033c20 msgr2=0x7f9cc0036010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:11:00.884 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:00.882+0000 7f9cdefad700 1 --2- 172.21.15.29:0/98749328 >> [v2:172.21.15.29:6800/1129362254,v1:172.21.15.29:6801/1129362254] conn(0x7f9cc0033c20 0x7f9cc0036010 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f9cc80036d0 tx=0x7f9cc8006cb0).stop 2023-12-01T22:11:00.885 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:00.882+0000 7f9cdefad700 1 -- 172.21.15.29:0/98749328 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cd80703f0 msgr2=0x7f9cd810ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:11:00.885 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:00.882+0000 7f9cdefad700 1 --2- 172.21.15.29:0/98749328 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cd80703f0 0x7f9cd810ca80 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f9cd400e040 tx=0x7f9cd4006ee0).stop 2023-12-01T22:11:00.885 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:00.882+0000 7f9cdefad700 1 -- 172.21.15.29:0/98749328 shutdown_connections 2023-12-01T22:11:00.886 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:00.882+0000 7f9cdefad700 1 --2- 172.21.15.29:0/98749328 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cd80703f0 0x7f9cd810ca80 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:11:00.886 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:00.882+0000 7f9cdefad700 1 --2- 172.21.15.29:0/98749328 >> [v2:172.21.15.29:6800/1129362254,v1:172.21.15.29:6801/1129362254] conn(0x7f9cc0033c20 0x7f9cc0036010 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:11:00.886 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:00.882+0000 7f9cdefad700 1 --2- 172.21.15.29:0/98749328 >> [v2:172.21.15.29:6800/3122114772,v1:172.21.15.29:6801/3122114772] conn(0x7f9cc002fdb0 0x7f9cc0032270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-12-01T22:11:00.886 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:00.882+0000 7f9cdefad700 1 -- 172.21.15.29:0/98749328 >> 172.21.15.29:0/98749328 conn(0x7f9cd80f6210 msgr2=0x7f9cd80f7cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:11:00.886 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:00.883+0000 7f9cdefad700 1 -- 172.21.15.29:0/98749328 shutdown_connections 2023-12-01T22:11:00.886 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:00.883+0000 7f9cdefad700 1 -- 172.21.15.29:0/98749328 wait complete. 2023-12-01T22:11:01.080 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:11:00 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:10:59.820126+0000 mon.smithi029 (mon.0) 63 : cluster [INF] Active manager daemon smithi029.loxrfu restarted 2023-12-01T22:11:01.080 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:11:00 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:10:59.820555+0000 mon.smithi029 (mon.0) 64 : cluster [INF] Activating manager daemon smithi029.loxrfu 2023-12-01T22:11:01.080 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:11:00 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:10:59.822363+0000 mon.smithi029 (mon.0) 65 : cluster [DBG] osdmap e3: 0 total, 0 up, 0 in 2023-12-01T22:11:01.080 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:11:00 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:10:59.873279+0000 mon.smithi029 (mon.0) 66 : cluster [DBG] mgrmap e10: smithi029.loxrfu(active, starting, since 0.0528237s) 2023-12-01T22:11:01.080 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:11:00 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:10:59.874369+0000 mon.smithi029 (mon.0) 67 : audit [DBG] from='mgr.14144 172.21.15.29:0/2246039843' entity='mgr.smithi029.loxrfu' cmd=[{"prefix": "mon metadata", "id": "smithi029"}]: dispatch 2023-12-01T22:11:01.080 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:11:00 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:10:59.874970+0000 mon.smithi029 (mon.0) 68 : audit [DBG] from='mgr.14144 172.21.15.29:0/2246039843' entity='mgr.smithi029.loxrfu' cmd=[{"prefix": "mgr metadata", "who": "smithi029.loxrfu", "id": "smithi029.loxrfu"}]: dispatch 2023-12-01T22:11:01.080 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:11:00 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:10:59.875859+0000 mon.smithi029 (mon.0) 69 : audit [DBG] from='mgr.14144 172.21.15.29:0/2246039843' entity='mgr.smithi029.loxrfu' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-12-01T22:11:01.080 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:11:00 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:10:59.876146+0000 mon.smithi029 (mon.0) 70 : audit [DBG] from='mgr.14144 172.21.15.29:0/2246039843' entity='mgr.smithi029.loxrfu' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-12-01T22:11:01.080 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:11:00 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:10:59.876407+0000 mon.smithi029 (mon.0) 71 : audit [DBG] from='mgr.14144 172.21.15.29:0/2246039843' entity='mgr.smithi029.loxrfu' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-12-01T22:11:01.080 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:11:00 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:10:59.890649+0000 mon.smithi029 (mon.0) 72 : cluster [INF] Manager daemon smithi029.loxrfu is now available 2023-12-01T22:11:01.081 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:11:00 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:11:00.004627+0000 mon.smithi029 (mon.0) 73 : audit [DBG] from='mgr.14144 172.21.15.29:0/2246039843' entity='mgr.smithi029.loxrfu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-01T22:11:01.081 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:11:00 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:11:00.006387+0000 mon.smithi029 (mon.0) 74 : audit [DBG] from='mgr.14144 172.21.15.29:0/2246039843' entity='mgr.smithi029.loxrfu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-01T22:11:01.081 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:11:00 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:11:00.024094+0000 mon.smithi029 (mon.0) 75 : audit [INF] from='mgr.14144 172.21.15.29:0/2246039843' entity='mgr.smithi029.loxrfu' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi029.loxrfu/mirror_snapshot_schedule"}]: dispatch 2023-12-01T22:11:01.081 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:11:00 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:11:00.034270+0000 mon.smithi029 (mon.0) 76 : audit [INF] from='mgr.14144 172.21.15.29:0/2246039843' entity='mgr.smithi029.loxrfu' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi029.loxrfu/trash_purge_schedule"}]: dispatch 2023-12-01T22:11:01.254 INFO:teuthology.orchestra.run.smithi029.stderr:Deploying grafana service with default placement... 2023-12-01T22:11:01.255 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi029 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/5558ab6c-9096-11ee-95a2-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpou9lnrt4:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpxe9edifa:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply grafana 2023-12-01T22:11:01.621 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:01.617+0000 7f127980d700 1 Processor -- start 2023-12-01T22:11:01.621 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:01.617+0000 7f127980d700 1 -- start start 2023-12-01T22:11:01.621 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:01.617+0000 7f127980d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1274070420 0x7f12740fe970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-01T22:11:01.621 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:01.617+0000 7f127980d700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12740fee60 con 0x7f1274070420 2023-12-01T22:11:01.621 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:01.617+0000 7f1272ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1274070420 0x7f12740fe970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:11:01.622 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:01.617+0000 7f1272ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1274070420 0x7f12740fe970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38104/0 (socket says 172.21.15.29:38104) 2023-12-01T22:11:01.622 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:01.617+0000 7f1272ffd700 1 -- 172.21.15.29:0/3065499628 learned_addr learned my addr 172.21.15.29:0/3065499628 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-12-01T22:11:01.622 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:01.618+0000 7f1272ffd700 1 -- 172.21.15.29:0/3065499628 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12740fefa0 con 0x7f1274070420 2023-12-01T22:11:01.622 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:01.618+0000 7f1272ffd700 1 --2- 172.21.15.29:0/3065499628 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1274070420 0x7f12740fe970 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f126400b5f0 tx=0x7f1264006ee0).ready entity=mon.0 client_cookie=e2e1ca949ff24749 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:11:01.622 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:01.618+0000 7f1271ffb700 1 -- 172.21.15.29:0/3065499628 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f126400d600 con 0x7f1274070420 2023-12-01T22:11:01.622 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:01.618+0000 7f1271ffb700 1 -- 172.21.15.29:0/3065499628 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f126400dbd0 con 0x7f1274070420 2023-12-01T22:11:01.622 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:01.618+0000 7f1271ffb700 1 -- 172.21.15.29:0/3065499628 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1264011c00 con 0x7f1274070420 2023-12-01T22:11:01.622 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:01.619+0000 7f127980d700 1 -- 172.21.15.29:0/3065499628 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1274070420 msgr2=0x7f12740fe970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:11:01.622 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:01.619+0000 7f127980d700 1 --2- 172.21.15.29:0/3065499628 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1274070420 0x7f12740fe970 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f126400b5f0 tx=0x7f1264006ee0).stop 2023-12-01T22:11:01.622 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:01.619+0000 7f127980d700 1 -- 172.21.15.29:0/3065499628 shutdown_connections 2023-12-01T22:11:01.622 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:01.619+0000 7f127980d700 1 --2- 172.21.15.29:0/3065499628 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1274070420 0x7f12740fe970 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:11:01.622 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:01.619+0000 7f127980d700 1 -- 172.21.15.29:0/3065499628 >> 172.21.15.29:0/3065499628 conn(0x7f12740f63b0 msgr2=0x7f12740f87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:11:01.623 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:01.619+0000 7f127980d700 1 -- 172.21.15.29:0/3065499628 shutdown_connections 2023-12-01T22:11:01.623 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:01.619+0000 7f127980d700 1 -- 172.21.15.29:0/3065499628 wait complete. 2023-12-01T22:11:01.623 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:01.620+0000 7f127980d700 1 Processor -- start 2023-12-01T22:11:01.625 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:01.620+0000 7f127980d700 1 -- start start 2023-12-01T22:11:01.625 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:01.620+0000 7f127980d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1274070420 0x7f12740fc3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:11:01.626 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:01.620+0000 7f127980d700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12740fa870 con 0x7f1274070420 2023-12-01T22:11:01.626 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:01.620+0000 7f1272ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1274070420 0x7f12740fc3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:11:01.626 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:01.620+0000 7f1272ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1274070420 0x7f12740fc3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38110/0 (socket says 172.21.15.29:38110) 2023-12-01T22:11:01.626 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:01.620+0000 7f1272ffd700 1 -- 172.21.15.29:0/3848367924 learned_addr learned my addr 172.21.15.29:0/3848367924 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-12-01T22:11:01.626 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:01.620+0000 7f1272ffd700 1 -- 172.21.15.29:0/3848367924 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f126400b200 con 0x7f1274070420 2023-12-01T22:11:01.626 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:01.620+0000 7f1272ffd700 1 --2- 172.21.15.29:0/3848367924 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1274070420 0x7f12740fc3f0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f126400e040 tx=0x7f1264006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:11:01.626 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:01.621+0000 7f125b7fe700 1 -- 172.21.15.29:0/3848367924 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f126400d9f0 con 0x7f1274070420 2023-12-01T22:11:01.626 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:01.621+0000 7f125b7fe700 1 -- 172.21.15.29:0/3848367924 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f126401a9f0 con 0x7f1274070420 2023-12-01T22:11:01.626 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:01.621+0000 7f127980d700 1 -- 172.21.15.29:0/3848367924 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12740faa10 con 0x7f1274070420 2023-12-01T22:11:01.626 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:01.621+0000 7f125b7fe700 1 -- 172.21.15.29:0/3848367924 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f126401b890 con 0x7f1274070420 2023-12-01T22:11:01.626 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:01.621+0000 7f127980d700 1 -- 172.21.15.29:0/3848367924 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12740fae20 con 0x7f1274070420 2023-12-01T22:11:01.627 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:01.622+0000 7f125b7fe700 1 -- 172.21.15.29:0/3848367924 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 11) v1 ==== 36054+0+0 (secure 0 0 0) 0x7f126400f070 con 0x7f1274070420 2023-12-01T22:11:01.627 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:01.622+0000 7f125b7fe700 1 --2- 172.21.15.29:0/3848367924 >> [v2:172.21.15.29:6800/1129362254,v1:172.21.15.29:6801/1129362254] conn(0x7f126002fd70 0x7f1260032230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:11:01.627 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:01.622+0000 7f125b7fe700 1 -- 172.21.15.29:0/3848367924 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f1264041350 con 0x7f1274070420 2023-12-01T22:11:01.627 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:01.622+0000 7f127980d700 1 -- 172.21.15.29:0/3848367924 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1274059100 con 0x7f1274070420 2023-12-01T22:11:01.627 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:01.622+0000 7f12727fc700 1 --2- 172.21.15.29:0/3848367924 >> [v2:172.21.15.29:6800/1129362254,v1:172.21.15.29:6801/1129362254] conn(0x7f126002fd70 0x7f1260032230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:11:01.627 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:01.622+0000 7f12727fc700 1 --2- 172.21.15.29:0/3848367924 >> [v2:172.21.15.29:6800/1129362254,v1:172.21.15.29:6801/1129362254] conn(0x7f126002fd70 0x7f1260032230 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f12680096a0 tx=0x7f1268006b40).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:11:01.628 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:01.627+0000 7f125b7fe700 1 -- 172.21.15.29:0/3848367924 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f12640224f0 con 0x7f1274070420 2023-12-01T22:11:01.824 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:01.821+0000 7f127980d700 1 -- 172.21.15.29:0/3848367924 --> [v2:172.21.15.29:6800/1129362254,v1:172.21.15.29:6801/1129362254] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}) v1 -- 0x7f12740ffbc0 con 0x7f126002fd70 2023-12-01T22:11:01.833 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:01.831+0000 7f125b7fe700 1 -- 172.21.15.29:0/3848367924 <== mgr.14144 v2:172.21.15.29:6800/1129362254 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f12740ffbc0 con 0x7f126002fd70 2023-12-01T22:11:01.833 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: Scheduled grafana update... 2023-12-01T22:11:01.836 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:01.833+0000 7f127980d700 1 -- 172.21.15.29:0/3848367924 >> [v2:172.21.15.29:6800/1129362254,v1:172.21.15.29:6801/1129362254] conn(0x7f126002fd70 msgr2=0x7f1260032230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:11:01.836 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:01.833+0000 7f127980d700 1 --2- 172.21.15.29:0/3848367924 >> [v2:172.21.15.29:6800/1129362254,v1:172.21.15.29:6801/1129362254] conn(0x7f126002fd70 0x7f1260032230 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f12680096a0 tx=0x7f1268006b40).stop 2023-12-01T22:11:01.836 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:01.833+0000 7f127980d700 1 -- 172.21.15.29:0/3848367924 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1274070420 msgr2=0x7f12740fc3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:11:01.836 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:01.833+0000 7f127980d700 1 --2- 172.21.15.29:0/3848367924 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1274070420 0x7f12740fc3f0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f126400e040 tx=0x7f1264006ee0).stop 2023-12-01T22:11:01.836 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:01.833+0000 7f127980d700 1 -- 172.21.15.29:0/3848367924 shutdown_connections 2023-12-01T22:11:01.836 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:01.833+0000 7f127980d700 1 --2- 172.21.15.29:0/3848367924 >> [v2:172.21.15.29:6800/1129362254,v1:172.21.15.29:6801/1129362254] conn(0x7f126002fd70 0x7f1260032230 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:11:01.836 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:01.833+0000 7f127980d700 1 --2- 172.21.15.29:0/3848367924 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1274070420 0x7f12740fc3f0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:11:01.836 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:01.833+0000 7f127980d700 1 -- 172.21.15.29:0/3848367924 >> 172.21.15.29:0/3848367924 conn(0x7f12740f63b0 msgr2=0x7f12740f7eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:11:01.836 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:01.833+0000 7f127980d700 1 -- 172.21.15.29:0/3848367924 shutdown_connections 2023-12-01T22:11:01.836 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:01.833+0000 7f127980d700 1 -- 172.21.15.29:0/3848367924 wait complete. 2023-12-01T22:11:02.277 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:11:01 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:11:00.876564+0000 mon.smithi029 (mon.0) 77 : cluster [DBG] mgrmap e11: smithi029.loxrfu(active, since 1.05611s) 2023-12-01T22:11:02.277 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:11:01 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cephadm 2023-12-01T22:11:00.877864+0000 mgr.smithi029.loxrfu (mgr.14144) 2 : cephadm [INF] Saving service prometheus spec with placement count:1 2023-12-01T22:11:02.277 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:11:01 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:11:00.880265+0000 mon.smithi029 (mon.0) 78 : audit [INF] from='mgr.14144 172.21.15.29:0/2246039843' entity='mgr.smithi029.loxrfu' 2023-12-01T22:11:02.277 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:11:01 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:11:01.826636+0000 mon.smithi029 (mon.0) 79 : audit [INF] from='mgr.14144 172.21.15.29:0/2246039843' entity='mgr.smithi029.loxrfu' 2023-12-01T22:11:02.513 INFO:teuthology.orchestra.run.smithi029.stderr:Deploying node-exporter service with default placement... 2023-12-01T22:11:02.513 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi029 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/5558ab6c-9096-11ee-95a2-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpou9lnrt4:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpxe9edifa:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply node-exporter 2023-12-01T22:11:02.915 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:02.911+0000 7ff049b39700 1 Processor -- start 2023-12-01T22:11:02.915 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:02.911+0000 7ff049b39700 1 -- start start 2023-12-01T22:11:02.915 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:02.912+0000 7ff049b39700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0440fa9d0 0x7ff0440fadf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-01T22:11:02.915 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:02.912+0000 7ff049b39700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0440fb2e0 con 0x7ff0440fa9d0 2023-12-01T22:11:02.915 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:02.912+0000 7ff048b37700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0440fa9d0 0x7ff0440fadf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:11:02.915 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:02.912+0000 7ff048b37700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0440fa9d0 0x7ff0440fadf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38114/0 (socket says 172.21.15.29:38114) 2023-12-01T22:11:02.915 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:02.912+0000 7ff048b37700 1 -- 172.21.15.29:0/2284037090 learned_addr learned my addr 172.21.15.29:0/2284037090 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-12-01T22:11:02.915 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:02.912+0000 7ff048b37700 1 -- 172.21.15.29:0/2284037090 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0440fb420 con 0x7ff0440fa9d0 2023-12-01T22:11:02.915 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:02.913+0000 7ff048b37700 1 --2- 172.21.15.29:0/2284037090 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0440fa9d0 0x7ff0440fadf0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7ff04000bc00 tx=0x7ff04000b570).ready entity=mon.0 client_cookie=6b92c35ab2657e3f server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:11:02.915 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:02.913+0000 7ff03b7fe700 1 -- 172.21.15.29:0/2284037090 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff04000dd70 con 0x7ff0440fa9d0 2023-12-01T22:11:02.916 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:02.913+0000 7ff03b7fe700 1 -- 172.21.15.29:0/2284037090 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7ff0400108e0 con 0x7ff0440fa9d0 2023-12-01T22:11:02.916 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:02.913+0000 7ff03b7fe700 1 -- 172.21.15.29:0/2284037090 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff04000f860 con 0x7ff0440fa9d0 2023-12-01T22:11:02.918 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:02.913+0000 7ff049b39700 1 -- 172.21.15.29:0/2284037090 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0440fa9d0 msgr2=0x7ff0440fadf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:11:02.918 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:02.913+0000 7ff049b39700 1 --2- 172.21.15.29:0/2284037090 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0440fa9d0 0x7ff0440fadf0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7ff04000bc00 tx=0x7ff04000b570).stop 2023-12-01T22:11:02.918 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:02.914+0000 7ff049b39700 1 -- 172.21.15.29:0/2284037090 shutdown_connections 2023-12-01T22:11:02.918 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:02.914+0000 7ff049b39700 1 --2- 172.21.15.29:0/2284037090 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0440fa9d0 0x7ff0440fadf0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:11:02.918 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:02.914+0000 7ff049b39700 1 -- 172.21.15.29:0/2284037090 >> 172.21.15.29:0/2284037090 conn(0x7ff0440f6210 msgr2=0x7ff0440f8630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:11:02.919 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:02.914+0000 7ff049b39700 1 -- 172.21.15.29:0/2284037090 shutdown_connections 2023-12-01T22:11:02.919 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:02.914+0000 7ff049b39700 1 -- 172.21.15.29:0/2284037090 wait complete. 2023-12-01T22:11:02.919 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:02.914+0000 7ff049b39700 1 Processor -- start 2023-12-01T22:11:02.919 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:02.915+0000 7ff049b39700 1 -- start start 2023-12-01T22:11:02.919 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:02.915+0000 7ff049b39700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0440fa9d0 0x7ff044108570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:11:02.919 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:02.915+0000 7ff049b39700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff044108a60 con 0x7ff0440fa9d0 2023-12-01T22:11:02.919 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:02.915+0000 7ff048b37700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0440fa9d0 0x7ff044108570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:11:02.919 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:02.915+0000 7ff048b37700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0440fa9d0 0x7ff044108570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38126/0 (socket says 172.21.15.29:38126) 2023-12-01T22:11:02.919 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:02.915+0000 7ff048b37700 1 -- 172.21.15.29:0/1634716462 learned_addr learned my addr 172.21.15.29:0/1634716462 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-12-01T22:11:02.919 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:02.916+0000 7ff048b37700 1 -- 172.21.15.29:0/1634716462 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff04000b810 con 0x7ff0440fa9d0 2023-12-01T22:11:02.919 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:02.916+0000 7ff048b37700 1 --2- 172.21.15.29:0/1634716462 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0440fa9d0 0x7ff044108570 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7ff040010fb0 tx=0x7ff04000b570).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:11:02.919 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:02.916+0000 7ff0397fa700 1 -- 172.21.15.29:0/1634716462 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff04000fd30 con 0x7ff0440fa9d0 2023-12-01T22:11:02.919 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:02.916+0000 7ff0397fa700 1 -- 172.21.15.29:0/1634716462 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7ff04000fe90 con 0x7ff0440fa9d0 2023-12-01T22:11:02.919 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:02.916+0000 7ff049b39700 1 -- 172.21.15.29:0/1634716462 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff044108c60 con 0x7ff0440fa9d0 2023-12-01T22:11:02.920 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:02.916+0000 7ff0397fa700 1 -- 172.21.15.29:0/1634716462 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff040023600 con 0x7ff0440fa9d0 2023-12-01T22:11:02.920 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:02.916+0000 7ff049b39700 1 -- 172.21.15.29:0/1634716462 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff044109070 con 0x7ff0440fa9d0 2023-12-01T22:11:02.920 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:02.917+0000 7ff0397fa700 1 -- 172.21.15.29:0/1634716462 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7ff04002a060 con 0x7ff0440fa9d0 2023-12-01T22:11:02.920 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:02.917+0000 7ff0397fa700 1 --2- 172.21.15.29:0/1634716462 >> [v2:172.21.15.29:6800/1129362254,v1:172.21.15.29:6801/1129362254] conn(0x7ff03002feb0 0x7ff030032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:11:02.920 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:02.917+0000 7ff0397fa700 1 -- 172.21.15.29:0/1634716462 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7ff040045320 con 0x7ff0440fa9d0 2023-12-01T22:11:02.921 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:02.917+0000 7ff03bfff700 1 --2- 172.21.15.29:0/1634716462 >> [v2:172.21.15.29:6800/1129362254,v1:172.21.15.29:6801/1129362254] conn(0x7ff03002feb0 0x7ff030032370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:11:02.921 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:02.917+0000 7ff049b39700 1 -- 172.21.15.29:0/1634716462 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff044045ad0 con 0x7ff0440fa9d0 2023-12-01T22:11:02.921 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:02.918+0000 7ff03bfff700 1 --2- 172.21.15.29:0/1634716462 >> [v2:172.21.15.29:6800/1129362254,v1:172.21.15.29:6801/1129362254] conn(0x7ff03002feb0 0x7ff030032370 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7ff0340096a0 tx=0x7ff034006b40).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:11:02.923 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:02.922+0000 7ff0397fa700 1 -- 172.21.15.29:0/1634716462 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff040014070 con 0x7ff0440fa9d0 2023-12-01T22:11:03.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:11:02 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:11:01.823636+0000 mgr.smithi029.loxrfu (mgr.14144) 3 : audit [DBG] from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2023-12-01T22:11:03.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:11:02 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cephadm 2023-12-01T22:11:01.824228+0000 mgr.smithi029.loxrfu (mgr.14144) 4 : cephadm [INF] Saving service grafana spec with placement count:1 2023-12-01T22:11:03.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:11:02 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:11:02.829578+0000 mon.smithi029 (mon.0) 80 : cluster [DBG] mgrmap e12: smithi029.loxrfu(active, since 3s) 2023-12-01T22:11:03.117 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:03.115+0000 7ff049b39700 1 -- 172.21.15.29:0/1634716462 --> [v2:172.21.15.29:6800/1129362254,v1:172.21.15.29:6801/1129362254] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7ff044102420 con 0x7ff03002feb0 2023-12-01T22:11:03.121 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:03.119+0000 7ff0397fa700 1 -- 172.21.15.29:0/1634716462 <== mgr.14144 v2:172.21.15.29:6800/1129362254 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7ff044102420 con 0x7ff03002feb0 2023-12-01T22:11:03.121 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: Scheduled node-exporter update... 2023-12-01T22:11:03.123 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:03.121+0000 7ff049b39700 1 -- 172.21.15.29:0/1634716462 >> [v2:172.21.15.29:6800/1129362254,v1:172.21.15.29:6801/1129362254] conn(0x7ff03002feb0 msgr2=0x7ff030032370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:11:03.124 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:03.121+0000 7ff049b39700 1 --2- 172.21.15.29:0/1634716462 >> [v2:172.21.15.29:6800/1129362254,v1:172.21.15.29:6801/1129362254] conn(0x7ff03002feb0 0x7ff030032370 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7ff0340096a0 tx=0x7ff034006b40).stop 2023-12-01T22:11:03.124 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:03.121+0000 7ff049b39700 1 -- 172.21.15.29:0/1634716462 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0440fa9d0 msgr2=0x7ff044108570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:11:03.124 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:03.121+0000 7ff049b39700 1 --2- 172.21.15.29:0/1634716462 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0440fa9d0 0x7ff044108570 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7ff040010fb0 tx=0x7ff04000b570).stop 2023-12-01T22:11:03.124 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:03.121+0000 7ff049b39700 1 -- 172.21.15.29:0/1634716462 shutdown_connections 2023-12-01T22:11:03.124 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:03.121+0000 7ff049b39700 1 --2- 172.21.15.29:0/1634716462 >> [v2:172.21.15.29:6800/1129362254,v1:172.21.15.29:6801/1129362254] conn(0x7ff03002feb0 0x7ff030032370 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:11:03.124 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:03.121+0000 7ff049b39700 1 --2- 172.21.15.29:0/1634716462 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0440fa9d0 0x7ff044108570 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:11:03.124 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:03.121+0000 7ff049b39700 1 -- 172.21.15.29:0/1634716462 >> 172.21.15.29:0/1634716462 conn(0x7ff0440f6210 msgr2=0x7ff0440ff760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:11:03.124 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:03.121+0000 7ff049b39700 1 -- 172.21.15.29:0/1634716462 shutdown_connections 2023-12-01T22:11:03.124 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:03.121+0000 7ff049b39700 1 -- 172.21.15.29:0/1634716462 wait complete. 2023-12-01T22:11:03.488 INFO:teuthology.orchestra.run.smithi029.stderr:Deploying alertmanager service with default placement... 2023-12-01T22:11:03.488 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi029 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/5558ab6c-9096-11ee-95a2-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpou9lnrt4:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpxe9edifa:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply alertmanager 2023-12-01T22:11:03.862 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:03.859+0000 7fcf7e6c5700 1 Processor -- start 2023-12-01T22:11:03.862 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:03.859+0000 7fcf7e6c5700 1 -- start start 2023-12-01T22:11:03.862 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:03.859+0000 7fcf7e6c5700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf780703b0 0x7fcf78100d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-01T22:11:03.862 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:03.859+0000 7fcf7e6c5700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf780fc750 con 0x7fcf780703b0 2023-12-01T22:11:03.862 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:03.859+0000 7fcf77fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf780703b0 0x7fcf78100d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:11:03.862 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:03.859+0000 7fcf77fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf780703b0 0x7fcf78100d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38136/0 (socket says 172.21.15.29:38136) 2023-12-01T22:11:03.862 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:03.859+0000 7fcf77fff700 1 -- 172.21.15.29:0/373949962 learned_addr learned my addr 172.21.15.29:0/373949962 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-12-01T22:11:03.862 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:03.859+0000 7fcf77fff700 1 -- 172.21.15.29:0/373949962 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf78101250 con 0x7fcf780703b0 2023-12-01T22:11:03.862 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:03.860+0000 7fcf77fff700 1 --2- 172.21.15.29:0/373949962 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf780703b0 0x7fcf78100d60 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7fcf6800b5f0 tx=0x7fcf68006ee0).ready entity=mon.0 client_cookie=d6a4daff67bb7904 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:11:03.865 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:03.860+0000 7fcf76ffd700 1 -- 172.21.15.29:0/373949962 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcf6800d600 con 0x7fcf780703b0 2023-12-01T22:11:03.865 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:03.860+0000 7fcf76ffd700 1 -- 172.21.15.29:0/373949962 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fcf6800dbd0 con 0x7fcf780703b0 2023-12-01T22:11:03.865 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:03.860+0000 7fcf76ffd700 1 -- 172.21.15.29:0/373949962 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcf68011c00 con 0x7fcf780703b0 2023-12-01T22:11:03.865 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:03.860+0000 7fcf7e6c5700 1 -- 172.21.15.29:0/373949962 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf780703b0 msgr2=0x7fcf78100d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:11:03.865 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:03.860+0000 7fcf7e6c5700 1 --2- 172.21.15.29:0/373949962 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf780703b0 0x7fcf78100d60 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7fcf6800b5f0 tx=0x7fcf68006ee0).stop 2023-12-01T22:11:03.865 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:03.860+0000 7fcf7e6c5700 1 -- 172.21.15.29:0/373949962 shutdown_connections 2023-12-01T22:11:03.865 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:03.860+0000 7fcf7e6c5700 1 --2- 172.21.15.29:0/373949962 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf780703b0 0x7fcf78100d60 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:11:03.865 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:03.860+0000 7fcf7e6c5700 1 -- 172.21.15.29:0/373949962 >> 172.21.15.29:0/373949962 conn(0x7fcf780f63c0 msgr2=0x7fcf780f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:11:03.865 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:03.860+0000 7fcf7e6c5700 1 -- 172.21.15.29:0/373949962 shutdown_connections 2023-12-01T22:11:03.865 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:03.860+0000 7fcf7e6c5700 1 -- 172.21.15.29:0/373949962 wait complete. 2023-12-01T22:11:03.865 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:03.861+0000 7fcf7e6c5700 1 Processor -- start 2023-12-01T22:11:03.865 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:03.861+0000 7fcf7e6c5700 1 -- start start 2023-12-01T22:11:03.866 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:03.862+0000 7fcf7e6c5700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf780703b0 0x7fcf7810cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:11:03.866 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:03.862+0000 7fcf7e6c5700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf7810d100 con 0x7fcf780703b0 2023-12-01T22:11:03.866 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:03.862+0000 7fcf77fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf780703b0 0x7fcf7810cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:11:03.866 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:03.862+0000 7fcf77fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf780703b0 0x7fcf7810cc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38138/0 (socket says 172.21.15.29:38138) 2023-12-01T22:11:03.866 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:03.862+0000 7fcf77fff700 1 -- 172.21.15.29:0/2122140131 learned_addr learned my addr 172.21.15.29:0/2122140131 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-12-01T22:11:03.866 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:03.862+0000 7fcf77fff700 1 -- 172.21.15.29:0/2122140131 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf6800b200 con 0x7fcf780703b0 2023-12-01T22:11:03.866 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:03.862+0000 7fcf77fff700 1 --2- 172.21.15.29:0/2122140131 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf780703b0 0x7fcf7810cc10 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7fcf6800b190 tx=0x7fcf68006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:11:03.866 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:03.862+0000 7fcf74ff9700 1 -- 172.21.15.29:0/2122140131 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcf6801b410 con 0x7fcf780703b0 2023-12-01T22:11:03.866 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:03.862+0000 7fcf74ff9700 1 -- 172.21.15.29:0/2122140131 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fcf6801b9e0 con 0x7fcf780703b0 2023-12-01T22:11:03.866 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:03.862+0000 7fcf7e6c5700 1 -- 172.21.15.29:0/2122140131 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf7810d300 con 0x7fcf780703b0 2023-12-01T22:11:03.866 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:03.862+0000 7fcf7e6c5700 1 -- 172.21.15.29:0/2122140131 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf7810d710 con 0x7fcf780703b0 2023-12-01T22:11:03.867 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:03.864+0000 7fcf74ff9700 1 -- 172.21.15.29:0/2122140131 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcf6801a880 con 0x7fcf780703b0 2023-12-01T22:11:03.867 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:03.864+0000 7fcf74ff9700 1 -- 172.21.15.29:0/2122140131 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7fcf6800f070 con 0x7fcf780703b0 2023-12-01T22:11:03.867 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:03.864+0000 7fcf74ff9700 1 --2- 172.21.15.29:0/2122140131 >> [v2:172.21.15.29:6800/1129362254,v1:172.21.15.29:6801/1129362254] conn(0x7fcf6402ff50 0x7fcf64032410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:11:03.868 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:03.864+0000 7fcf74ff9700 1 -- 172.21.15.29:0/2122140131 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7fcf68042350 con 0x7fcf780703b0 2023-12-01T22:11:03.868 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:03.864+0000 7fcf777fe700 1 --2- 172.21.15.29:0/2122140131 >> [v2:172.21.15.29:6800/1129362254,v1:172.21.15.29:6801/1129362254] conn(0x7fcf6402ff50 0x7fcf64032410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:11:03.868 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:03.864+0000 7fcf7e6c5700 1 -- 172.21.15.29:0/2122140131 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf58004fa0 con 0x7fcf780703b0 2023-12-01T22:11:03.868 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:03.865+0000 7fcf777fe700 1 --2- 172.21.15.29:0/2122140131 >> [v2:172.21.15.29:6800/1129362254,v1:172.21.15.29:6801/1129362254] conn(0x7fcf6402ff50 0x7fcf64032410 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fcf6c0096a0 tx=0x7fcf6c006b40).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:11:03.870 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:03.869+0000 7fcf74ff9700 1 -- 172.21.15.29:0/2122140131 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcf68016070 con 0x7fcf780703b0 2023-12-01T22:11:04.066 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:04.064+0000 7fcf7e6c5700 1 -- 172.21.15.29:0/2122140131 --> [v2:172.21.15.29:6800/1129362254,v1:172.21.15.29:6801/1129362254] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}) v1 -- 0x7fcf58000bc0 con 0x7fcf6402ff50 2023-12-01T22:11:04.069 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:04.068+0000 7fcf74ff9700 1 -- 172.21.15.29:0/2122140131 <== mgr.14144 v2:172.21.15.29:6800/1129362254 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7fcf58000bc0 con 0x7fcf6402ff50 2023-12-01T22:11:04.070 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: Scheduled alertmanager update... 2023-12-01T22:11:04.071 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:04.070+0000 7fcf7e6c5700 1 -- 172.21.15.29:0/2122140131 >> [v2:172.21.15.29:6800/1129362254,v1:172.21.15.29:6801/1129362254] conn(0x7fcf6402ff50 msgr2=0x7fcf64032410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:11:04.072 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:04.070+0000 7fcf7e6c5700 1 --2- 172.21.15.29:0/2122140131 >> [v2:172.21.15.29:6800/1129362254,v1:172.21.15.29:6801/1129362254] conn(0x7fcf6402ff50 0x7fcf64032410 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fcf6c0096a0 tx=0x7fcf6c006b40).stop 2023-12-01T22:11:04.072 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:04.070+0000 7fcf7e6c5700 1 -- 172.21.15.29:0/2122140131 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf780703b0 msgr2=0x7fcf7810cc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:11:04.072 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:04.070+0000 7fcf7e6c5700 1 --2- 172.21.15.29:0/2122140131 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf780703b0 0x7fcf7810cc10 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7fcf6800b190 tx=0x7fcf68006ee0).stop 2023-12-01T22:11:04.073 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:04.070+0000 7fcf7e6c5700 1 -- 172.21.15.29:0/2122140131 shutdown_connections 2023-12-01T22:11:04.073 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:04.070+0000 7fcf7e6c5700 1 --2- 172.21.15.29:0/2122140131 >> [v2:172.21.15.29:6800/1129362254,v1:172.21.15.29:6801/1129362254] conn(0x7fcf6402ff50 0x7fcf64032410 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:11:04.073 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:04.070+0000 7fcf7e6c5700 1 --2- 172.21.15.29:0/2122140131 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf780703b0 0x7fcf7810cc10 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:11:04.073 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:04.070+0000 7fcf7e6c5700 1 -- 172.21.15.29:0/2122140131 >> 172.21.15.29:0/2122140131 conn(0x7fcf780f63c0 msgr2=0x7fcf780f7ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:11:04.073 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:04.070+0000 7fcf7e6c5700 1 -- 172.21.15.29:0/2122140131 shutdown_connections 2023-12-01T22:11:04.073 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:04.070+0000 7fcf7e6c5700 1 -- 172.21.15.29:0/2122140131 wait complete. 2023-12-01T22:11:04.391 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:11:04 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:11:03.117151+0000 mgr.smithi029.loxrfu (mgr.14144) 5 : audit [DBG] from='client.14154 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2023-12-01T22:11:04.391 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:11:04 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cephadm 2023-12-01T22:11:03.117883+0000 mgr.smithi029.loxrfu (mgr.14144) 6 : cephadm [INF] Saving service node-exporter spec with placement * 2023-12-01T22:11:04.391 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:11:04 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:11:03.120296+0000 mon.smithi029 (mon.0) 81 : audit [INF] from='mgr.14144 172.21.15.29:0/2246039843' entity='mgr.smithi029.loxrfu' 2023-12-01T22:11:04.391 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:11:04 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:11:04.068864+0000 mon.smithi029 (mon.0) 82 : audit [INF] from='mgr.14144 172.21.15.29:0/2246039843' entity='mgr.smithi029.loxrfu' 2023-12-01T22:11:04.421 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi029 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/5558ab6c-9096-11ee-95a2-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpou9lnrt4:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpxe9edifa:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mgr mgr/cephadm/container_init True --force 2023-12-01T22:11:04.797 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:04.793+0000 7ffa268ea700 1 Processor -- start 2023-12-01T22:11:04.797 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:04.793+0000 7ffa268ea700 1 -- start start 2023-12-01T22:11:04.797 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:04.794+0000 7ffa268ea700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa200fd2a0 0x7ffa200fd6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-01T22:11:04.797 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:04.794+0000 7ffa268ea700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa200fdbb0 con 0x7ffa200fd2a0 2023-12-01T22:11:04.797 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:04.794+0000 7ffa1ffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa200fd2a0 0x7ffa200fd6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:11:04.797 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:04.794+0000 7ffa1ffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa200fd2a0 0x7ffa200fd6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57120/0 (socket says 172.21.15.29:57120) 2023-12-01T22:11:04.797 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:04.794+0000 7ffa1ffff700 1 -- 172.21.15.29:0/2301079892 learned_addr learned my addr 172.21.15.29:0/2301079892 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-12-01T22:11:04.799 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:04.794+0000 7ffa1ffff700 1 -- 172.21.15.29:0/2301079892 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa200fdcf0 con 0x7ffa200fd2a0 2023-12-01T22:11:04.799 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:04.794+0000 7ffa1ffff700 1 --2- 172.21.15.29:0/2301079892 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa200fd2a0 0x7ffa200fd6c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7ffa1000b5f0 tx=0x7ffa10006ee0).ready entity=mon.0 client_cookie=59d67c43b9981a68 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:11:04.799 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:04.795+0000 7ffa1effd700 1 -- 172.21.15.29:0/2301079892 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffa1000d600 con 0x7ffa200fd2a0 2023-12-01T22:11:04.799 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:04.795+0000 7ffa1effd700 1 -- 172.21.15.29:0/2301079892 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7ffa1000dbd0 con 0x7ffa200fd2a0 2023-12-01T22:11:04.799 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:04.795+0000 7ffa1effd700 1 -- 172.21.15.29:0/2301079892 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffa10011c00 con 0x7ffa200fd2a0 2023-12-01T22:11:04.799 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:04.795+0000 7ffa268ea700 1 -- 172.21.15.29:0/2301079892 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa200fd2a0 msgr2=0x7ffa200fd6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:11:04.799 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:04.795+0000 7ffa268ea700 1 --2- 172.21.15.29:0/2301079892 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa200fd2a0 0x7ffa200fd6c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7ffa1000b5f0 tx=0x7ffa10006ee0).stop 2023-12-01T22:11:04.799 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:04.795+0000 7ffa268ea700 1 -- 172.21.15.29:0/2301079892 shutdown_connections 2023-12-01T22:11:04.800 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:04.795+0000 7ffa268ea700 1 --2- 172.21.15.29:0/2301079892 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa200fd2a0 0x7ffa200fd6c0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:11:04.800 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:04.795+0000 7ffa268ea700 1 -- 172.21.15.29:0/2301079892 >> 172.21.15.29:0/2301079892 conn(0x7ffa200f5b40 msgr2=0x7ffa200f7f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:11:04.800 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:04.795+0000 7ffa268ea700 1 -- 172.21.15.29:0/2301079892 shutdown_connections 2023-12-01T22:11:04.800 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:04.795+0000 7ffa268ea700 1 -- 172.21.15.29:0/2301079892 wait complete. 2023-12-01T22:11:04.800 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:04.796+0000 7ffa268ea700 1 Processor -- start 2023-12-01T22:11:04.800 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:04.796+0000 7ffa268ea700 1 -- start start 2023-12-01T22:11:04.800 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:04.796+0000 7ffa268ea700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa200fd2a0 0x7ffa2010cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:11:04.801 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:04.796+0000 7ffa268ea700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa2010d0f0 con 0x7ffa200fd2a0 2023-12-01T22:11:04.801 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:04.796+0000 7ffa1ffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa200fd2a0 0x7ffa2010cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:11:04.801 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:04.797+0000 7ffa1ffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa200fd2a0 0x7ffa2010cc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57132/0 (socket says 172.21.15.29:57132) 2023-12-01T22:11:04.801 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:04.797+0000 7ffa1ffff700 1 -- 172.21.15.29:0/1888741339 learned_addr learned my addr 172.21.15.29:0/1888741339 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-12-01T22:11:04.801 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:04.797+0000 7ffa1ffff700 1 -- 172.21.15.29:0/1888741339 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa1000b200 con 0x7ffa200fd2a0 2023-12-01T22:11:04.801 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:04.797+0000 7ffa1ffff700 1 --2- 172.21.15.29:0/1888741339 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa200fd2a0 0x7ffa2010cc00 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7ffa10002f40 tx=0x7ffa10006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:11:04.801 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:04.797+0000 7ffa1cff9700 1 -- 172.21.15.29:0/1888741339 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffa1001b410 con 0x7ffa200fd2a0 2023-12-01T22:11:04.801 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:04.797+0000 7ffa1cff9700 1 -- 172.21.15.29:0/1888741339 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7ffa1001b9e0 con 0x7ffa200fd2a0 2023-12-01T22:11:04.801 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:04.797+0000 7ffa268ea700 1 -- 172.21.15.29:0/1888741339 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa2010d2f0 con 0x7ffa200fd2a0 2023-12-01T22:11:04.802 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:04.797+0000 7ffa1cff9700 1 -- 172.21.15.29:0/1888741339 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffa1001a8b0 con 0x7ffa200fd2a0 2023-12-01T22:11:04.802 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:04.797+0000 7ffa268ea700 1 -- 172.21.15.29:0/1888741339 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa2010d700 con 0x7ffa200fd2a0 2023-12-01T22:11:04.802 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:04.798+0000 7ffa1cff9700 1 -- 172.21.15.29:0/1888741339 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7ffa1000f070 con 0x7ffa200fd2a0 2023-12-01T22:11:04.802 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:04.798+0000 7ffa1cff9700 1 --2- 172.21.15.29:0/1888741339 >> [v2:172.21.15.29:6800/1129362254,v1:172.21.15.29:6801/1129362254] conn(0x7ffa0002fe60 0x7ffa00032320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:11:04.802 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:04.798+0000 7ffa1cff9700 1 -- 172.21.15.29:0/1888741339 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7ffa10041490 con 0x7ffa200fd2a0 2023-12-01T22:11:04.802 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:04.798+0000 7ffa268ea700 1 -- 172.21.15.29:0/1888741339 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa20045ad0 con 0x7ffa200fd2a0 2023-12-01T22:11:04.802 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:04.798+0000 7ffa1f7fe700 1 --2- 172.21.15.29:0/1888741339 >> [v2:172.21.15.29:6800/1129362254,v1:172.21.15.29:6801/1129362254] conn(0x7ffa0002fe60 0x7ffa00032320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:11:04.802 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:04.799+0000 7ffa1f7fe700 1 --2- 172.21.15.29:0/1888741339 >> [v2:172.21.15.29:6800/1129362254,v1:172.21.15.29:6801/1129362254] conn(0x7ffa0002fe60 0x7ffa00032320 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7ffa080096a0 tx=0x7ffa08006b40).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:11:04.804 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:04.803+0000 7ffa1cff9700 1 -- 172.21.15.29:0/1888741339 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ffa10016070 con 0x7ffa200fd2a0 2023-12-01T22:11:04.974 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:04.972+0000 7ffa268ea700 1 -- 172.21.15.29:0/1888741339 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7ffa200fa7c0 con 0x7ffa200fd2a0 2023-12-01T22:11:04.976 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:04.975+0000 7ffa1cff9700 1 -- 172.21.15.29:0/1888741339 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v8)=0 v8) v1 ==== 142+0+0 (secure 0 0 0) 0x7ffa10022e80 con 0x7ffa200fd2a0 2023-12-01T22:11:04.979 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:04.976+0000 7ffa268ea700 1 -- 172.21.15.29:0/1888741339 >> [v2:172.21.15.29:6800/1129362254,v1:172.21.15.29:6801/1129362254] conn(0x7ffa0002fe60 msgr2=0x7ffa00032320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:11:04.979 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:04.976+0000 7ffa268ea700 1 --2- 172.21.15.29:0/1888741339 >> [v2:172.21.15.29:6800/1129362254,v1:172.21.15.29:6801/1129362254] conn(0x7ffa0002fe60 0x7ffa00032320 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7ffa080096a0 tx=0x7ffa08006b40).stop 2023-12-01T22:11:04.979 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:04.977+0000 7ffa268ea700 1 -- 172.21.15.29:0/1888741339 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa200fd2a0 msgr2=0x7ffa2010cc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:11:04.979 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:04.977+0000 7ffa268ea700 1 --2- 172.21.15.29:0/1888741339 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa200fd2a0 0x7ffa2010cc00 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7ffa10002f40 tx=0x7ffa10006ee0).stop 2023-12-01T22:11:04.979 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:04.977+0000 7ffa268ea700 1 -- 172.21.15.29:0/1888741339 shutdown_connections 2023-12-01T22:11:04.979 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:04.977+0000 7ffa268ea700 1 --2- 172.21.15.29:0/1888741339 >> [v2:172.21.15.29:6800/1129362254,v1:172.21.15.29:6801/1129362254] conn(0x7ffa0002fe60 0x7ffa00032320 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:11:04.980 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:04.977+0000 7ffa268ea700 1 --2- 172.21.15.29:0/1888741339 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa200fd2a0 0x7ffa2010cc00 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:11:04.980 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:04.977+0000 7ffa268ea700 1 -- 172.21.15.29:0/1888741339 >> 172.21.15.29:0/1888741339 conn(0x7ffa200f5b40 msgr2=0x7ffa200faf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:11:04.980 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:04.978+0000 7ffa268ea700 1 -- 172.21.15.29:0/1888741339 shutdown_connections 2023-12-01T22:11:04.980 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:04.978+0000 7ffa268ea700 1 -- 172.21.15.29:0/1888741339 wait complete. 2023-12-01T22:11:05.257 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:11:05 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:11:04.065977+0000 mgr.smithi029.loxrfu (mgr.14144) 7 : audit [DBG] from='client.14156 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2023-12-01T22:11:05.257 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:11:05 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cephadm 2023-12-01T22:11:04.066611+0000 mgr.smithi029.loxrfu (mgr.14144) 8 : cephadm [INF] Saving service alertmanager spec with placement count:1 2023-12-01T22:11:05.257 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:11:05 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:11:04.976146+0000 mon.smithi029 (mon.0) 83 : audit [INF] from='client.? 172.21.15.29:0/1888741339' entity='client.admin' 2023-12-01T22:11:05.329 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi029 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/5558ab6c-9096-11ee-95a2-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpou9lnrt4:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpxe9edifa:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mgr mgr/dashboard/ssl_server_port 8443 2023-12-01T22:11:05.705 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:05.702+0000 7fdb50ad4700 1 Processor -- start 2023-12-01T22:11:05.705 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:05.702+0000 7fdb50ad4700 1 -- start start 2023-12-01T22:11:05.705 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:05.702+0000 7fdb50ad4700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb4c0fc130 0x7fdb4c0fc550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-01T22:11:05.705 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:05.702+0000 7fdb50ad4700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb4c0fca40 con 0x7fdb4c0fc130 2023-12-01T22:11:05.705 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:05.703+0000 7fdb4b7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb4c0fc130 0x7fdb4c0fc550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:11:05.705 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:05.703+0000 7fdb4b7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb4c0fc130 0x7fdb4c0fc550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57140/0 (socket says 172.21.15.29:57140) 2023-12-01T22:11:05.705 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:05.703+0000 7fdb4b7fe700 1 -- 172.21.15.29:0/2542926199 learned_addr learned my addr 172.21.15.29:0/2542926199 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-12-01T22:11:05.706 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:05.703+0000 7fdb4b7fe700 1 -- 172.21.15.29:0/2542926199 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb4c0fcb80 con 0x7fdb4c0fc130 2023-12-01T22:11:05.707 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:05.703+0000 7fdb4b7fe700 1 --2- 172.21.15.29:0/2542926199 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb4c0fc130 0x7fdb4c0fc550 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7fdb3c00b5f0 tx=0x7fdb3c006ee0).ready entity=mon.0 client_cookie=8906dbe01e8186d9 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:11:05.707 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:05.703+0000 7fdb4a7fc700 1 -- 172.21.15.29:0/2542926199 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdb3c00d600 con 0x7fdb4c0fc130 2023-12-01T22:11:05.707 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:05.703+0000 7fdb4a7fc700 1 -- 172.21.15.29:0/2542926199 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fdb3c00dbd0 con 0x7fdb4c0fc130 2023-12-01T22:11:05.707 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:05.703+0000 7fdb4a7fc700 1 -- 172.21.15.29:0/2542926199 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdb3c011c00 con 0x7fdb4c0fc130 2023-12-01T22:11:05.707 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:05.704+0000 7fdb50ad4700 1 -- 172.21.15.29:0/2542926199 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb4c0fc130 msgr2=0x7fdb4c0fc550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:11:05.707 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:05.704+0000 7fdb50ad4700 1 --2- 172.21.15.29:0/2542926199 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb4c0fc130 0x7fdb4c0fc550 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7fdb3c00b5f0 tx=0x7fdb3c006ee0).stop 2023-12-01T22:11:05.707 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:05.704+0000 7fdb50ad4700 1 -- 172.21.15.29:0/2542926199 shutdown_connections 2023-12-01T22:11:05.708 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:05.704+0000 7fdb50ad4700 1 --2- 172.21.15.29:0/2542926199 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb4c0fc130 0x7fdb4c0fc550 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:11:05.708 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:05.704+0000 7fdb50ad4700 1 -- 172.21.15.29:0/2542926199 >> 172.21.15.29:0/2542926199 conn(0x7fdb4c0f59a0 msgr2=0x7fdb4c0f7dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:11:05.708 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:05.704+0000 7fdb50ad4700 1 -- 172.21.15.29:0/2542926199 shutdown_connections 2023-12-01T22:11:05.708 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:05.704+0000 7fdb50ad4700 1 -- 172.21.15.29:0/2542926199 wait complete. 2023-12-01T22:11:05.708 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:05.705+0000 7fdb50ad4700 1 Processor -- start 2023-12-01T22:11:05.708 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:05.705+0000 7fdb50ad4700 1 -- start start 2023-12-01T22:11:05.709 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:05.705+0000 7fdb50ad4700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb4c0fc130 0x7fdb4c10ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:11:05.709 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:05.705+0000 7fdb50ad4700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb4c10cf60 con 0x7fdb4c0fc130 2023-12-01T22:11:05.709 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:05.705+0000 7fdb4b7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb4c0fc130 0x7fdb4c10ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:11:05.709 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:05.705+0000 7fdb4b7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb4c0fc130 0x7fdb4c10ca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57148/0 (socket says 172.21.15.29:57148) 2023-12-01T22:11:05.709 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:05.705+0000 7fdb4b7fe700 1 -- 172.21.15.29:0/1388928469 learned_addr learned my addr 172.21.15.29:0/1388928469 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-12-01T22:11:05.709 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:05.705+0000 7fdb4b7fe700 1 -- 172.21.15.29:0/1388928469 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb3c00b200 con 0x7fdb4c0fc130 2023-12-01T22:11:05.709 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:05.706+0000 7fdb4b7fe700 1 --2- 172.21.15.29:0/1388928469 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb4c0fc130 0x7fdb4c10ca70 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7fdb3c00e040 tx=0x7fdb3c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:11:05.709 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:05.706+0000 7fdb33fff700 1 -- 172.21.15.29:0/1388928469 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdb3c01b410 con 0x7fdb4c0fc130 2023-12-01T22:11:05.709 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:05.706+0000 7fdb33fff700 1 -- 172.21.15.29:0/1388928469 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fdb3c01b9e0 con 0x7fdb4c0fc130 2023-12-01T22:11:05.710 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:05.706+0000 7fdb33fff700 1 -- 172.21.15.29:0/1388928469 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdb3c01a770 con 0x7fdb4c0fc130 2023-12-01T22:11:05.710 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:05.706+0000 7fdb50ad4700 1 -- 172.21.15.29:0/1388928469 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb4c10d160 con 0x7fdb4c0fc130 2023-12-01T22:11:05.710 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:05.706+0000 7fdb50ad4700 1 -- 172.21.15.29:0/1388928469 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb4c10d570 con 0x7fdb4c0fc130 2023-12-01T22:11:05.710 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:05.707+0000 7fdb33fff700 1 -- 172.21.15.29:0/1388928469 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7fdb3c00f070 con 0x7fdb4c0fc130 2023-12-01T22:11:05.710 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:05.707+0000 7fdb33fff700 1 --2- 172.21.15.29:0/1388928469 >> [v2:172.21.15.29:6800/1129362254,v1:172.21.15.29:6801/1129362254] conn(0x7fdb3802feb0 0x7fdb38032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:11:05.710 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:05.707+0000 7fdb33fff700 1 -- 172.21.15.29:0/1388928469 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7fdb3c041110 con 0x7fdb4c0fc130 2023-12-01T22:11:05.710 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:05.707+0000 7fdb50ad4700 1 -- 172.21.15.29:0/1388928469 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb4c045ad0 con 0x7fdb4c0fc130 2023-12-01T22:11:05.710 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:05.707+0000 7fdb4affd700 1 --2- 172.21.15.29:0/1388928469 >> [v2:172.21.15.29:6800/1129362254,v1:172.21.15.29:6801/1129362254] conn(0x7fdb3802feb0 0x7fdb38032370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:11:05.710 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:05.708+0000 7fdb4affd700 1 --2- 172.21.15.29:0/1388928469 >> [v2:172.21.15.29:6800/1129362254,v1:172.21.15.29:6801/1129362254] conn(0x7fdb3802feb0 0x7fdb38032370 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fdb400096a0 tx=0x7fdb40006b40).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:11:05.713 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:05.712+0000 7fdb33fff700 1 -- 172.21.15.29:0/1388928469 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdb3c016070 con 0x7fdb4c0fc130 2023-12-01T22:11:05.878 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:05.876+0000 7fdb50ad4700 1 -- 172.21.15.29:0/1388928469 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7fdb4c106a60 con 0x7fdb4c0fc130 2023-12-01T22:11:05.880 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:05.879+0000 7fdb33fff700 1 -- 172.21.15.29:0/1388928469 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v9)=0 v9) v1 ==== 130+0+0 (secure 0 0 0) 0x7fdb3c010360 con 0x7fdb4c0fc130 2023-12-01T22:11:05.883 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:05.881+0000 7fdb50ad4700 1 -- 172.21.15.29:0/1388928469 >> [v2:172.21.15.29:6800/1129362254,v1:172.21.15.29:6801/1129362254] conn(0x7fdb3802feb0 msgr2=0x7fdb38032370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:11:05.884 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:05.881+0000 7fdb50ad4700 1 --2- 172.21.15.29:0/1388928469 >> [v2:172.21.15.29:6800/1129362254,v1:172.21.15.29:6801/1129362254] conn(0x7fdb3802feb0 0x7fdb38032370 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fdb400096a0 tx=0x7fdb40006b40).stop 2023-12-01T22:11:05.884 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:05.881+0000 7fdb50ad4700 1 -- 172.21.15.29:0/1388928469 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb4c0fc130 msgr2=0x7fdb4c10ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:11:05.884 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:05.881+0000 7fdb50ad4700 1 --2- 172.21.15.29:0/1388928469 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb4c0fc130 0x7fdb4c10ca70 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7fdb3c00e040 tx=0x7fdb3c006ee0).stop 2023-12-01T22:11:05.884 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:05.881+0000 7fdb50ad4700 1 -- 172.21.15.29:0/1388928469 shutdown_connections 2023-12-01T22:11:05.884 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:05.881+0000 7fdb50ad4700 1 --2- 172.21.15.29:0/1388928469 >> [v2:172.21.15.29:6800/1129362254,v1:172.21.15.29:6801/1129362254] conn(0x7fdb3802feb0 0x7fdb38032370 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:11:05.884 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:05.881+0000 7fdb50ad4700 1 --2- 172.21.15.29:0/1388928469 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb4c0fc130 0x7fdb4c10ca70 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:11:05.884 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:05.881+0000 7fdb50ad4700 1 -- 172.21.15.29:0/1388928469 >> 172.21.15.29:0/1388928469 conn(0x7fdb4c0f59a0 msgr2=0x7fdb4c0f74c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:11:05.884 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:05.881+0000 7fdb50ad4700 1 -- 172.21.15.29:0/1388928469 shutdown_connections 2023-12-01T22:11:05.884 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:05.881+0000 7fdb50ad4700 1 -- 172.21.15.29:0/1388928469 wait complete. 2023-12-01T22:11:06.221 INFO:teuthology.orchestra.run.smithi029.stderr:Enabling the dashboard module... 2023-12-01T22:11:06.221 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi029 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/5558ab6c-9096-11ee-95a2-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpou9lnrt4:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpxe9edifa:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable dashboard 2023-12-01T22:11:06.589 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:06.580+0000 7fc7068ce700 1 Processor -- start 2023-12-01T22:11:06.589 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:06.580+0000 7fc7068ce700 1 -- start start 2023-12-01T22:11:06.589 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:06.580+0000 7fc7068ce700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc700070390 0x7fc700100d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-01T22:11:06.589 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:06.580+0000 7fc7068ce700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7000fc770 con 0x7fc700070390 2023-12-01T22:11:06.589 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:06.580+0000 7fc6fffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc700070390 0x7fc700100d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:11:06.590 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:06.580+0000 7fc6fffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc700070390 0x7fc700100d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57160/0 (socket says 172.21.15.29:57160) 2023-12-01T22:11:06.590 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:06.580+0000 7fc6fffff700 1 -- 172.21.15.29:0/1445519751 learned_addr learned my addr 172.21.15.29:0/1445519751 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-12-01T22:11:06.590 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:06.580+0000 7fc6fffff700 1 -- 172.21.15.29:0/1445519751 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc700101270 con 0x7fc700070390 2023-12-01T22:11:06.590 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:06.581+0000 7fc6fffff700 1 --2- 172.21.15.29:0/1445519751 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc700070390 0x7fc700100d80 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7fc6f000b5f0 tx=0x7fc6f0006ee0).ready entity=mon.0 client_cookie=39b93559f5d73886 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:11:06.590 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:06.581+0000 7fc6feffd700 1 -- 172.21.15.29:0/1445519751 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc6f000d600 con 0x7fc700070390 2023-12-01T22:11:06.590 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:06.581+0000 7fc6feffd700 1 -- 172.21.15.29:0/1445519751 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fc6f000dbd0 con 0x7fc700070390 2023-12-01T22:11:06.590 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:06.581+0000 7fc6feffd700 1 -- 172.21.15.29:0/1445519751 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc6f0011c00 con 0x7fc700070390 2023-12-01T22:11:06.590 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:06.581+0000 7fc7068ce700 1 -- 172.21.15.29:0/1445519751 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc700070390 msgr2=0x7fc700100d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:11:06.590 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:06.581+0000 7fc7068ce700 1 --2- 172.21.15.29:0/1445519751 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc700070390 0x7fc700100d80 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7fc6f000b5f0 tx=0x7fc6f0006ee0).stop 2023-12-01T22:11:06.590 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:06.581+0000 7fc7068ce700 1 -- 172.21.15.29:0/1445519751 shutdown_connections 2023-12-01T22:11:06.590 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:06.581+0000 7fc7068ce700 1 --2- 172.21.15.29:0/1445519751 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc700070390 0x7fc700100d80 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:11:06.591 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:06.581+0000 7fc7068ce700 1 -- 172.21.15.29:0/1445519751 >> 172.21.15.29:0/1445519751 conn(0x7fc7000f63c0 msgr2=0x7fc7000f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:11:06.591 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:06.581+0000 7fc7068ce700 1 -- 172.21.15.29:0/1445519751 shutdown_connections 2023-12-01T22:11:06.591 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:06.581+0000 7fc7068ce700 1 -- 172.21.15.29:0/1445519751 wait complete. 2023-12-01T22:11:06.591 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:06.582+0000 7fc7068ce700 1 Processor -- start 2023-12-01T22:11:06.591 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:06.582+0000 7fc7068ce700 1 -- start start 2023-12-01T22:11:06.591 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:06.582+0000 7fc7068ce700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc700070390 0x7fc70010cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:11:06.591 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:06.582+0000 7fc7068ce700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc70010d0f0 con 0x7fc700070390 2023-12-01T22:11:06.591 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:06.583+0000 7fc6fffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc700070390 0x7fc70010cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:11:06.591 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:06.583+0000 7fc6fffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc700070390 0x7fc70010cc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57176/0 (socket says 172.21.15.29:57176) 2023-12-01T22:11:06.591 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:06.583+0000 7fc6fffff700 1 -- 172.21.15.29:0/4041522697 learned_addr learned my addr 172.21.15.29:0/4041522697 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-12-01T22:11:06.591 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:06.583+0000 7fc6fffff700 1 -- 172.21.15.29:0/4041522697 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6f000b200 con 0x7fc700070390 2023-12-01T22:11:06.591 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:06.583+0000 7fc6fffff700 1 --2- 172.21.15.29:0/4041522697 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc700070390 0x7fc70010cc00 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7fc6f000b190 tx=0x7fc6f0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:11:06.591 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:06.583+0000 7fc6fcff9700 1 -- 172.21.15.29:0/4041522697 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc6f001b410 con 0x7fc700070390 2023-12-01T22:11:06.591 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:06.583+0000 7fc7068ce700 1 -- 172.21.15.29:0/4041522697 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc70010d2f0 con 0x7fc700070390 2023-12-01T22:11:06.592 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:06.583+0000 7fc6fcff9700 1 -- 172.21.15.29:0/4041522697 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fc6f001b9e0 con 0x7fc700070390 2023-12-01T22:11:06.592 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:06.583+0000 7fc6fcff9700 1 -- 172.21.15.29:0/4041522697 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc6f001a880 con 0x7fc700070390 2023-12-01T22:11:06.592 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:06.583+0000 7fc7068ce700 1 -- 172.21.15.29:0/4041522697 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc70010d700 con 0x7fc700070390 2023-12-01T22:11:06.592 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:06.584+0000 7fc6fcff9700 1 -- 172.21.15.29:0/4041522697 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7fc6f000f070 con 0x7fc700070390 2023-12-01T22:11:06.592 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:06.584+0000 7fc6fcff9700 1 --2- 172.21.15.29:0/4041522697 >> [v2:172.21.15.29:6800/1129362254,v1:172.21.15.29:6801/1129362254] conn(0x7fc6ec02fe60 0x7fc6ec032320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:11:06.592 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:06.584+0000 7fc6fcff9700 1 -- 172.21.15.29:0/4041522697 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7fc6f00413e0 con 0x7fc700070390 2023-12-01T22:11:06.592 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:06.585+0000 7fc6ff7fe700 1 --2- 172.21.15.29:0/4041522697 >> [v2:172.21.15.29:6800/1129362254,v1:172.21.15.29:6801/1129362254] conn(0x7fc6ec02fe60 0x7fc6ec032320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:11:06.592 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:06.585+0000 7fc7068ce700 1 -- 172.21.15.29:0/4041522697 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc700059100 con 0x7fc700070390 2023-12-01T22:11:06.592 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:06.585+0000 7fc6ff7fe700 1 --2- 172.21.15.29:0/4041522697 >> [v2:172.21.15.29:6800/1129362254,v1:172.21.15.29:6801/1129362254] conn(0x7fc6ec02fe60 0x7fc6ec032320 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7fc6f40096a0 tx=0x7fc6f4006b40).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:11:06.592 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:06.589+0000 7fc6fcff9700 1 -- 172.21.15.29:0/4041522697 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc6f0016080 con 0x7fc700070390 2023-12-01T22:11:06.799 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:06.797+0000 7fc7068ce700 1 -- 172.21.15.29:0/4041522697 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7fc700101d70 con 0x7fc700070390 2023-12-01T22:11:06.883 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:06.881+0000 7fc6fcff9700 1 -- 172.21.15.29:0/4041522697 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 13) v1 ==== 36208+0+0 (secure 0 0 0) 0x7fc6f001bb50 con 0x7fc700070390 2023-12-01T22:11:06.884 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:06.882+0000 7fc6fcff9700 1 -- 172.21.15.29:0/4041522697 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v13) v1 ==== 88+0+0 (secure 0 0 0) 0x7fc6f0024830 con 0x7fc700070390 2023-12-01T22:11:06.888 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:06.885+0000 7fc7068ce700 1 -- 172.21.15.29:0/4041522697 >> [v2:172.21.15.29:6800/1129362254,v1:172.21.15.29:6801/1129362254] conn(0x7fc6ec02fe60 msgr2=0x7fc6ec032320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:11:06.888 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:06.885+0000 7fc7068ce700 1 --2- 172.21.15.29:0/4041522697 >> [v2:172.21.15.29:6800/1129362254,v1:172.21.15.29:6801/1129362254] conn(0x7fc6ec02fe60 0x7fc6ec032320 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7fc6f40096a0 tx=0x7fc6f4006b40).stop 2023-12-01T22:11:06.888 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:06.885+0000 7fc7068ce700 1 -- 172.21.15.29:0/4041522697 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc700070390 msgr2=0x7fc70010cc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:11:06.888 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:06.885+0000 7fc7068ce700 1 --2- 172.21.15.29:0/4041522697 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc700070390 0x7fc70010cc00 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7fc6f000b190 tx=0x7fc6f0006ee0).stop 2023-12-01T22:11:06.888 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:06.885+0000 7fc7068ce700 1 -- 172.21.15.29:0/4041522697 shutdown_connections 2023-12-01T22:11:06.888 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:06.885+0000 7fc7068ce700 1 --2- 172.21.15.29:0/4041522697 >> [v2:172.21.15.29:6800/1129362254,v1:172.21.15.29:6801/1129362254] conn(0x7fc6ec02fe60 0x7fc6ec032320 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:11:06.888 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:06.885+0000 7fc7068ce700 1 --2- 172.21.15.29:0/4041522697 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc700070390 0x7fc70010cc00 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:11:06.888 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:06.885+0000 7fc7068ce700 1 -- 172.21.15.29:0/4041522697 >> 172.21.15.29:0/4041522697 conn(0x7fc7000f63c0 msgr2=0x7fc7000f7ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:11:06.888 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:06.885+0000 7fc7068ce700 1 -- 172.21.15.29:0/4041522697 shutdown_connections 2023-12-01T22:11:06.888 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:06.885+0000 7fc7068ce700 1 -- 172.21.15.29:0/4041522697 wait complete. 2023-12-01T22:11:07.276 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:11:06 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:11:05.880285+0000 mon.smithi029 (mon.0) 84 : audit [INF] from='client.? 172.21.15.29:0/1388928469' entity='client.admin' 2023-12-01T22:11:07.277 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:11:06 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:11:06.798516+0000 mon.smithi029 (mon.0) 85 : audit [INF] from='client.? 172.21.15.29:0/4041522697' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2023-12-01T22:11:07.339 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi029 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/5558ab6c-9096-11ee-95a2-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpou9lnrt4:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpxe9edifa:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr stat 2023-12-01T22:11:08.140 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:08.138+0000 7fafebffa700 1 Processor -- start 2023-12-01T22:11:08.141 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:08.138+0000 7fafebffa700 1 -- start start 2023-12-01T22:11:08.141 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:08.138+0000 7fafebffa700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fafe40fd2a0 0x7fafe40fd6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-01T22:11:08.141 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:08.138+0000 7fafebffa700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafe40fdbb0 con 0x7fafe40fd2a0 2023-12-01T22:11:08.141 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:08.138+0000 7fafe9d96700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fafe40fd2a0 0x7fafe40fd6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:11:08.141 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:08.138+0000 7fafe9d96700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fafe40fd2a0 0x7fafe40fd6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57208/0 (socket says 172.21.15.29:57208) 2023-12-01T22:11:08.141 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:08.138+0000 7fafe9d96700 1 -- 172.21.15.29:0/2655916916 learned_addr learned my addr 172.21.15.29:0/2655916916 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-12-01T22:11:08.143 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:08.139+0000 7fafe9d96700 1 -- 172.21.15.29:0/2655916916 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafe40fdcf0 con 0x7fafe40fd2a0 2023-12-01T22:11:08.143 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:08.139+0000 7fafe9d96700 1 --2- 172.21.15.29:0/2655916916 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fafe40fd2a0 0x7fafe40fd6c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7fafe000b5f0 tx=0x7fafe0006ee0).ready entity=mon.0 client_cookie=fa6cd59f14c9eff8 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:11:08.143 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:08.139+0000 7fafe8d94700 1 -- 172.21.15.29:0/2655916916 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fafe000d600 con 0x7fafe40fd2a0 2023-12-01T22:11:08.143 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:08.139+0000 7fafe8d94700 1 -- 172.21.15.29:0/2655916916 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fafe000dbd0 con 0x7fafe40fd2a0 2023-12-01T22:11:08.143 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:08.139+0000 7fafe8d94700 1 -- 172.21.15.29:0/2655916916 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fafe0011c00 con 0x7fafe40fd2a0 2023-12-01T22:11:08.143 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:08.140+0000 7fafebffa700 1 -- 172.21.15.29:0/2655916916 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fafe40fd2a0 msgr2=0x7fafe40fd6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:11:08.143 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:08.140+0000 7fafebffa700 1 --2- 172.21.15.29:0/2655916916 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fafe40fd2a0 0x7fafe40fd6c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7fafe000b5f0 tx=0x7fafe0006ee0).stop 2023-12-01T22:11:08.143 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:08.140+0000 7fafebffa700 1 -- 172.21.15.29:0/2655916916 shutdown_connections 2023-12-01T22:11:08.143 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:08.140+0000 7fafebffa700 1 --2- 172.21.15.29:0/2655916916 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fafe40fd2a0 0x7fafe40fd6c0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:11:08.143 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:08.140+0000 7fafebffa700 1 -- 172.21.15.29:0/2655916916 >> 172.21.15.29:0/2655916916 conn(0x7fafe40f5b40 msgr2=0x7fafe40f7f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:11:08.144 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:08.140+0000 7fafebffa700 1 -- 172.21.15.29:0/2655916916 shutdown_connections 2023-12-01T22:11:08.144 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:08.140+0000 7fafebffa700 1 -- 172.21.15.29:0/2655916916 wait complete. 2023-12-01T22:11:08.144 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:08.141+0000 7fafebffa700 1 Processor -- start 2023-12-01T22:11:08.144 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:08.141+0000 7fafebffa700 1 -- start start 2023-12-01T22:11:08.146 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:08.141+0000 7fafebffa700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fafe40fd2a0 0x7fafe410a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:11:08.146 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:08.141+0000 7fafebffa700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafe410aee0 con 0x7fafe40fd2a0 2023-12-01T22:11:08.146 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:08.142+0000 7fafe9d96700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fafe40fd2a0 0x7fafe410a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:11:08.146 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:08.142+0000 7fafe9d96700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fafe40fd2a0 0x7fafe410a9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57212/0 (socket says 172.21.15.29:57212) 2023-12-01T22:11:08.146 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:08.142+0000 7fafe9d96700 1 -- 172.21.15.29:0/2083392293 learned_addr learned my addr 172.21.15.29:0/2083392293 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-12-01T22:11:08.146 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:08.142+0000 7fafe9d96700 1 -- 172.21.15.29:0/2083392293 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafe000b200 con 0x7fafe40fd2a0 2023-12-01T22:11:08.146 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:08.142+0000 7fafe9d96700 1 --2- 172.21.15.29:0/2083392293 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fafe40fd2a0 0x7fafe410a9f0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7fafe0002f40 tx=0x7fafe0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:11:08.146 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:08.142+0000 7fafd27fc700 1 -- 172.21.15.29:0/2083392293 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fafe001b410 con 0x7fafe40fd2a0 2023-12-01T22:11:08.146 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:08.142+0000 7fafd27fc700 1 -- 172.21.15.29:0/2083392293 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fafe001b9e0 con 0x7fafe40fd2a0 2023-12-01T22:11:08.146 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:08.142+0000 7fafebffa700 1 -- 172.21.15.29:0/2083392293 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafe410b0e0 con 0x7fafe40fd2a0 2023-12-01T22:11:08.146 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:08.142+0000 7fafd27fc700 1 -- 172.21.15.29:0/2083392293 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fafe001a8b0 con 0x7fafe40fd2a0 2023-12-01T22:11:08.147 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:08.143+0000 7fafebffa700 1 -- 172.21.15.29:0/2083392293 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafe410b4f0 con 0x7fafe40fd2a0 2023-12-01T22:11:08.147 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:08.144+0000 7fafd27fc700 1 -- 172.21.15.29:0/2083392293 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 36208+0+0 (secure 0 0 0) 0x7fafe000f070 con 0x7fafe40fd2a0 2023-12-01T22:11:08.147 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:08.144+0000 7fafd27fc700 1 --2- 172.21.15.29:0/2083392293 >> [v2:172.21.15.29:6800/1129362254,v1:172.21.15.29:6801/1129362254] conn(0x7fafcc02ff00 0x7fafcc0323c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:11:08.147 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:08.144+0000 7fafd27fc700 1 -- 172.21.15.29:0/2083392293 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7fafe0041370 con 0x7fafe40fd2a0 2023-12-01T22:11:08.147 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:08.144+0000 7fafebffa700 1 -- 172.21.15.29:0/2083392293 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafe4045ad0 con 0x7fafe40fd2a0 2023-12-01T22:11:08.147 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:08.144+0000 7fafe9595700 1 -- 172.21.15.29:0/2083392293 >> [v2:172.21.15.29:6800/1129362254,v1:172.21.15.29:6801/1129362254] conn(0x7fafcc02ff00 msgr2=0x7fafcc0323c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/1129362254 2023-12-01T22:11:08.147 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:08.144+0000 7fafe9595700 1 --2- 172.21.15.29:0/2083392293 >> [v2:172.21.15.29:6800/1129362254,v1:172.21.15.29:6801/1129362254] conn(0x7fafcc02ff00 0x7fafcc0323c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-12-01T22:11:08.151 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:08.149+0000 7fafd27fc700 1 -- 172.21.15.29:0/2083392293 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fafe0016070 con 0x7fafe40fd2a0 2023-12-01T22:11:08.229 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:11:07 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:11:06.883449+0000 mon.smithi029 (mon.0) 86 : audit [INF] from='client.? 172.21.15.29:0/4041522697' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2023-12-01T22:11:08.229 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:11:07 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:11:06.883527+0000 mon.smithi029 (mon.0) 87 : cluster [DBG] mgrmap e13: smithi029.loxrfu(active, since 7s) 2023-12-01T22:11:08.347 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:08.345+0000 7fafe9595700 1 -- 172.21.15.29:0/2083392293 >> [v2:172.21.15.29:6800/1129362254,v1:172.21.15.29:6801/1129362254] conn(0x7fafcc02ff00 msgr2=0x7fafcc0323c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/1129362254 2023-12-01T22:11:08.347 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:08.345+0000 7fafe9595700 1 --2- 172.21.15.29:0/2083392293 >> [v2:172.21.15.29:6800/1129362254,v1:172.21.15.29:6801/1129362254] conn(0x7fafcc02ff00 0x7fafcc0323c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-12-01T22:11:08.410 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:08.407+0000 7fafebffa700 1 -- 172.21.15.29:0/2083392293 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fafe41044c0 con 0x7fafe40fd2a0 2023-12-01T22:11:08.410 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:08.408+0000 7fafd27fc700 1 -- 172.21.15.29:0/2083392293 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v13) v1 ==== 56+0+104 (secure 0 0 0) 0x7fafe0025460 con 0x7fafe40fd2a0 2023-12-01T22:11:08.410 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: { 2023-12-01T22:11:08.410 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "epoch": 13, 2023-12-01T22:11:08.410 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "available": true, 2023-12-01T22:11:08.410 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "active_name": "smithi029.loxrfu", 2023-12-01T22:11:08.410 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "num_standby": 0 2023-12-01T22:11:08.410 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: } 2023-12-01T22:11:08.412 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:08.410+0000 7fafebffa700 1 -- 172.21.15.29:0/2083392293 >> [v2:172.21.15.29:6800/1129362254,v1:172.21.15.29:6801/1129362254] conn(0x7fafcc02ff00 msgr2=0x7fafcc0323c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-01T22:11:08.412 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:08.410+0000 7fafebffa700 1 --2- 172.21.15.29:0/2083392293 >> [v2:172.21.15.29:6800/1129362254,v1:172.21.15.29:6801/1129362254] conn(0x7fafcc02ff00 0x7fafcc0323c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-12-01T22:11:08.412 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:08.410+0000 7fafebffa700 1 -- 172.21.15.29:0/2083392293 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fafe40fd2a0 msgr2=0x7fafe410a9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:11:08.412 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:08.410+0000 7fafebffa700 1 --2- 172.21.15.29:0/2083392293 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fafe40fd2a0 0x7fafe410a9f0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7fafe0002f40 tx=0x7fafe0006ee0).stop 2023-12-01T22:11:08.413 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:08.410+0000 7fafebffa700 1 -- 172.21.15.29:0/2083392293 shutdown_connections 2023-12-01T22:11:08.413 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:08.410+0000 7fafebffa700 1 --2- 172.21.15.29:0/2083392293 >> [v2:172.21.15.29:6800/1129362254,v1:172.21.15.29:6801/1129362254] conn(0x7fafcc02ff00 0x7fafcc0323c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-12-01T22:11:08.413 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:08.410+0000 7fafebffa700 1 --2- 172.21.15.29:0/2083392293 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fafe40fd2a0 0x7fafe410a9f0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:11:08.413 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:08.410+0000 7fafebffa700 1 -- 172.21.15.29:0/2083392293 >> 172.21.15.29:0/2083392293 conn(0x7fafe40f5b40 msgr2=0x7fafe40ff270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:11:08.413 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:08.410+0000 7fafebffa700 1 -- 172.21.15.29:0/2083392293 shutdown_connections 2023-12-01T22:11:08.413 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:08.410+0000 7fafebffa700 1 -- 172.21.15.29:0/2083392293 wait complete. 2023-12-01T22:11:08.829 INFO:teuthology.orchestra.run.smithi029.stderr:Waiting for the mgr to restart... 2023-12-01T22:11:08.829 INFO:teuthology.orchestra.run.smithi029.stderr:Waiting for mgr epoch 13... 2023-12-01T22:11:08.829 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi029 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/5558ab6c-9096-11ee-95a2-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpou9lnrt4:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpxe9edifa:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 tell mgr mgr_status 2023-12-01T22:11:09.228 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:09.226+0000 7f78aded6700 1 Processor -- start 2023-12-01T22:11:09.228 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:09.226+0000 7f78aded6700 1 -- start start 2023-12-01T22:11:09.228 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:09.226+0000 7f78aded6700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78a8072490 0x7f78a8070a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-01T22:11:09.229 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:09.226+0000 7f78aded6700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78a80728b0 con 0x7f78a8072490 2023-12-01T22:11:09.229 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:09.226+0000 7f78a77fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78a8072490 0x7f78a8070a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:11:09.229 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:09.226+0000 7f78a77fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78a8072490 0x7f78a8070a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57224/0 (socket says 172.21.15.29:57224) 2023-12-01T22:11:09.229 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:09.226+0000 7f78a77fe700 1 -- 172.21.15.29:0/4229796512 learned_addr learned my addr 172.21.15.29:0/4229796512 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-12-01T22:11:09.230 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:09.226+0000 7f78a77fe700 1 -- 172.21.15.29:0/4229796512 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78a8070f20 con 0x7f78a8072490 2023-12-01T22:11:09.230 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:09.227+0000 7f78a77fe700 1 --2- 172.21.15.29:0/4229796512 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78a8072490 0x7f78a8070a30 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f789800b5f0 tx=0x7f7898006ee0).ready entity=mon.0 client_cookie=4278de5a90b9e071 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:11:09.230 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:09.227+0000 7f78a67fc700 1 -- 172.21.15.29:0/4229796512 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f789800d600 con 0x7f78a8072490 2023-12-01T22:11:09.230 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:09.227+0000 7f78a67fc700 1 -- 172.21.15.29:0/4229796512 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f789800dbd0 con 0x7f78a8072490 2023-12-01T22:11:09.230 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:09.227+0000 7f78a67fc700 1 -- 172.21.15.29:0/4229796512 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7898011c00 con 0x7f78a8072490 2023-12-01T22:11:09.231 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:09.228+0000 7f78aded6700 1 -- 172.21.15.29:0/4229796512 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78a8072490 msgr2=0x7f78a8070a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:11:09.231 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:09.228+0000 7f78aded6700 1 --2- 172.21.15.29:0/4229796512 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78a8072490 0x7f78a8070a30 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f789800b5f0 tx=0x7f7898006ee0).stop 2023-12-01T22:11:09.232 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:09.228+0000 7f78aded6700 1 -- 172.21.15.29:0/4229796512 shutdown_connections 2023-12-01T22:11:09.232 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:09.228+0000 7f78aded6700 1 --2- 172.21.15.29:0/4229796512 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78a8072490 0x7f78a8070a30 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:11:09.232 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:09.228+0000 7f78aded6700 1 -- 172.21.15.29:0/4229796512 >> 172.21.15.29:0/4229796512 conn(0x7f78a80f74c0 msgr2=0x7f78a80f98e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:11:09.232 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:09.228+0000 7f78aded6700 1 -- 172.21.15.29:0/4229796512 shutdown_connections 2023-12-01T22:11:09.232 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:09.228+0000 7f78aded6700 1 -- 172.21.15.29:0/4229796512 wait complete. 2023-12-01T22:11:09.232 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:09.229+0000 7f78aded6700 1 Processor -- start 2023-12-01T22:11:09.232 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:09.229+0000 7f78aded6700 1 -- start start 2023-12-01T22:11:09.232 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:09.229+0000 7f78aded6700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78a8072490 0x7f78a810cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:11:09.232 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:09.229+0000 7f78aded6700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78a810d1d0 con 0x7f78a8072490 2023-12-01T22:11:09.232 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:09.230+0000 7f78a77fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78a8072490 0x7f78a810cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:11:09.232 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:09.230+0000 7f78a77fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78a8072490 0x7f78a810cce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57226/0 (socket says 172.21.15.29:57226) 2023-12-01T22:11:09.232 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:09.230+0000 7f78a77fe700 1 -- 172.21.15.29:0/2581111840 learned_addr learned my addr 172.21.15.29:0/2581111840 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-12-01T22:11:09.234 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:09.230+0000 7f78a77fe700 1 -- 172.21.15.29:0/2581111840 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f789800b200 con 0x7f78a8072490 2023-12-01T22:11:09.234 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:09.230+0000 7f78a77fe700 1 --2- 172.21.15.29:0/2581111840 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78a8072490 0x7f78a810cce0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f7898011820 tx=0x7f7898006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:11:09.234 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:09.230+0000 7f788ffff700 1 -- 172.21.15.29:0/2581111840 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f789801b410 con 0x7f78a8072490 2023-12-01T22:11:09.234 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:09.230+0000 7f78aded6700 1 -- 172.21.15.29:0/2581111840 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78a810d3d0 con 0x7f78a8072490 2023-12-01T22:11:09.234 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:09.230+0000 7f788ffff700 1 -- 172.21.15.29:0/2581111840 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f789801b9e0 con 0x7f78a8072490 2023-12-01T22:11:09.234 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:09.230+0000 7f788ffff700 1 -- 172.21.15.29:0/2581111840 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f789801a8b0 con 0x7f78a8072490 2023-12-01T22:11:09.234 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:09.230+0000 7f78aded6700 1 -- 172.21.15.29:0/2581111840 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78a810d7e0 con 0x7f78a8072490 2023-12-01T22:11:09.234 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:09.231+0000 7f788ffff700 1 -- 172.21.15.29:0/2581111840 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f789800f070 con 0x7f78a8072490 2023-12-01T22:11:09.234 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:09.231+0000 7f788ffff700 1 --2- 172.21.15.29:0/2581111840 >> [v2:172.21.15.29:6800/1129362254,v1:172.21.15.29:6801/1129362254] conn(0x7f789402feb0 0x7f7894032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:11:09.234 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:09.231+0000 7f788ffff700 1 -- 172.21.15.29:0/2581111840 --> [v2:172.21.15.29:6800/1129362254,v1:172.21.15.29:6801/1129362254] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7894032a30 con 0x7f789402feb0 2023-12-01T22:11:09.234 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:09.231+0000 7f788ffff700 1 -- 172.21.15.29:0/2581111840 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f78980414e0 con 0x7f78a8072490 2023-12-01T22:11:09.234 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:09.231+0000 7f78a6ffd700 1 -- 172.21.15.29:0/2581111840 >> [v2:172.21.15.29:6800/1129362254,v1:172.21.15.29:6801/1129362254] conn(0x7f789402feb0 msgr2=0x7f7894032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/1129362254 2023-12-01T22:11:09.234 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:09.232+0000 7f78a6ffd700 1 --2- 172.21.15.29:0/2581111840 >> [v2:172.21.15.29:6800/1129362254,v1:172.21.15.29:6801/1129362254] conn(0x7f789402feb0 0x7f7894032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-12-01T22:11:09.276 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:11:08 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:11:08.409037+0000 mon.smithi029 (mon.0) 88 : audit [DBG] from='client.? 172.21.15.29:0/2083392293' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-12-01T22:11:09.435 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:09.433+0000 7f78a6ffd700 1 -- 172.21.15.29:0/2581111840 >> [v2:172.21.15.29:6800/1129362254,v1:172.21.15.29:6801/1129362254] conn(0x7f789402feb0 msgr2=0x7f7894032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/1129362254 2023-12-01T22:11:09.435 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:09.433+0000 7f78a6ffd700 1 --2- 172.21.15.29:0/2581111840 >> [v2:172.21.15.29:6800/1129362254,v1:172.21.15.29:6801/1129362254] conn(0x7f789402feb0 0x7f7894032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-12-01T22:11:09.836 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:09.834+0000 7f78a6ffd700 1 -- 172.21.15.29:0/2581111840 >> [v2:172.21.15.29:6800/1129362254,v1:172.21.15.29:6801/1129362254] conn(0x7f789402feb0 msgr2=0x7f7894032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/1129362254 2023-12-01T22:11:09.836 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:09.834+0000 7f78a6ffd700 1 --2- 172.21.15.29:0/2581111840 >> [v2:172.21.15.29:6800/1129362254,v1:172.21.15.29:6801/1129362254] conn(0x7f789402feb0 0x7f7894032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-12-01T22:11:10.637 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:10.635+0000 7f78a6ffd700 1 -- 172.21.15.29:0/2581111840 >> [v2:172.21.15.29:6800/1129362254,v1:172.21.15.29:6801/1129362254] conn(0x7f789402feb0 msgr2=0x7f7894032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/1129362254 2023-12-01T22:11:10.637 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:10.635+0000 7f78a6ffd700 1 --2- 172.21.15.29:0/2581111840 >> [v2:172.21.15.29:6800/1129362254,v1:172.21.15.29:6801/1129362254] conn(0x7f789402feb0 0x7f7894032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-12-01T22:11:12.238 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:12.236+0000 7f78a6ffd700 1 -- 172.21.15.29:0/2581111840 >> [v2:172.21.15.29:6800/1129362254,v1:172.21.15.29:6801/1129362254] conn(0x7f789402feb0 msgr2=0x7f7894032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/1129362254 2023-12-01T22:11:12.239 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:12.236+0000 7f78a6ffd700 1 --2- 172.21.15.29:0/2581111840 >> [v2:172.21.15.29:6800/1129362254,v1:172.21.15.29:6801/1129362254] conn(0x7f789402feb0 0x7f7894032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 3.200000 2023-12-01T22:11:12.614 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:12.613+0000 7f788ffff700 1 -- 172.21.15.29:0/2581111840 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mgrmap(e 14) v1 ==== 35997+0+0 (secure 0 0 0) 0x7f789801ad90 con 0x7f78a8072490 2023-12-01T22:11:12.615 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:12.613+0000 7f788ffff700 1 -- 172.21.15.29:0/2581111840 >> [v2:172.21.15.29:6800/1129362254,v1:172.21.15.29:6801/1129362254] conn(0x7f789402feb0 msgr2=0x7f7894032370 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-01T22:11:12.615 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:12.613+0000 7f788ffff700 1 --2- 172.21.15.29:0/2581111840 >> [v2:172.21.15.29:6800/1129362254,v1:172.21.15.29:6801/1129362254] conn(0x7f789402feb0 0x7f7894032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-12-01T22:11:13.618 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:13.614+0000 7f788ffff700 1 -- 172.21.15.29:0/2581111840 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 15) v1 ==== 36067+0+0 (secure 0 0 0) 0x7f7898042600 con 0x7f78a8072490 2023-12-01T22:11:13.618 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:13.614+0000 7f788ffff700 1 --2- 172.21.15.29:0/2581111840 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f7894033b40 0x7f7894035f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:11:13.618 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:13.615+0000 7f788ffff700 1 -- 172.21.15.29:0/2581111840 --> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7894032a30 con 0x7f7894033b40 2023-12-01T22:11:13.618 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:13.615+0000 7f78a6ffd700 1 --2- 172.21.15.29:0/2581111840 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f7894033b40 0x7f7894035f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:11:13.618 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:13.616+0000 7f78a6ffd700 1 --2- 172.21.15.29:0/2581111840 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f7894033b40 0x7f7894035f30 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7f789c006fd0 tx=0x7f789c006cf0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:11:13.619 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:13.616+0000 7f788ffff700 1 -- 172.21.15.29:0/2581111840 <== mgr.14164 v2:172.21.15.29:6800/3698425734 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (secure 0 0 0) 0x7f7894032a30 con 0x7f7894033b40 2023-12-01T22:11:13.625 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:13.623+0000 7f78aded6700 1 -- 172.21.15.29:0/2581111840 --> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f78a8059100 con 0x7f7894033b40 2023-12-01T22:11:13.625 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:13.623+0000 7f788ffff700 1 -- 172.21.15.29:0/2581111840 <== mgr.14164 v2:172.21.15.29:6800/3698425734 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f78a8059100 con 0x7f7894033b40 2023-12-01T22:11:13.625 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: { 2023-12-01T22:11:13.625 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "mgrmap_epoch": 15, 2023-12-01T22:11:13.627 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "initialized": true 2023-12-01T22:11:13.627 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: } 2023-12-01T22:11:13.627 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:13.623+0000 7f78aded6700 1 -- 172.21.15.29:0/2581111840 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f7894033b40 msgr2=0x7f7894035f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:11:13.627 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:13.623+0000 7f78aded6700 1 --2- 172.21.15.29:0/2581111840 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f7894033b40 0x7f7894035f30 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7f789c006fd0 tx=0x7f789c006cf0).stop 2023-12-01T22:11:13.627 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:13.624+0000 7f78aded6700 1 -- 172.21.15.29:0/2581111840 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78a8072490 msgr2=0x7f78a810cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:11:13.627 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:13.624+0000 7f78aded6700 1 --2- 172.21.15.29:0/2581111840 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78a8072490 0x7f78a810cce0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f7898011820 tx=0x7f7898006ee0).stop 2023-12-01T22:11:13.627 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:13.624+0000 7f78aded6700 1 -- 172.21.15.29:0/2581111840 shutdown_connections 2023-12-01T22:11:13.627 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:13.624+0000 7f78aded6700 1 --2- 172.21.15.29:0/2581111840 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f7894033b40 0x7f7894035f30 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:11:13.627 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:13.624+0000 7f78aded6700 1 --2- 172.21.15.29:0/2581111840 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78a8072490 0x7f78a810cce0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:11:13.627 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:13.624+0000 7f78aded6700 1 --2- 172.21.15.29:0/2581111840 >> [v2:172.21.15.29:6800/1129362254,v1:172.21.15.29:6801/1129362254] conn(0x7f789402feb0 0x7f7894032370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-12-01T22:11:13.628 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:13.624+0000 7f78aded6700 1 -- 172.21.15.29:0/2581111840 >> 172.21.15.29:0/2581111840 conn(0x7f78a80f74c0 msgr2=0x7f78a80f9040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:11:13.628 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:13.624+0000 7f78aded6700 1 -- 172.21.15.29:0/2581111840 shutdown_connections 2023-12-01T22:11:13.628 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:13.624+0000 7f78aded6700 1 -- 172.21.15.29:0/2581111840 wait complete. 2023-12-01T22:11:13.929 INFO:teuthology.orchestra.run.smithi029.stderr:mgr epoch 13 is available 2023-12-01T22:11:13.929 INFO:teuthology.orchestra.run.smithi029.stderr:Generating a dashboard self-signed certificate... 2023-12-01T22:11:13.930 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi029 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/5558ab6c-9096-11ee-95a2-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpou9lnrt4:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpxe9edifa:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 dashboard create-self-signed-cert 2023-12-01T22:11:13.933 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:11:13 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:11:12.562130+0000 mon.smithi029 (mon.0) 89 : cluster [INF] Active manager daemon smithi029.loxrfu restarted 2023-12-01T22:11:13.933 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:11:13 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:11:12.562542+0000 mon.smithi029 (mon.0) 90 : cluster [INF] Activating manager daemon smithi029.loxrfu 2023-12-01T22:11:13.933 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:11:13 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:11:12.564215+0000 mon.smithi029 (mon.0) 91 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2023-12-01T22:11:13.934 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:11:13 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:11:12.614934+0000 mon.smithi029 (mon.0) 92 : cluster [DBG] mgrmap e14: smithi029.loxrfu(active, starting, since 0.0524818s) 2023-12-01T22:11:13.934 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:11:13 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:11:12.616319+0000 mon.smithi029 (mon.0) 93 : audit [DBG] from='mgr.14164 172.21.15.29:0/3195927111' entity='mgr.smithi029.loxrfu' cmd=[{"prefix": "mon metadata", "id": "smithi029"}]: dispatch 2023-12-01T22:11:13.934 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:11:13 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:11:12.616908+0000 mon.smithi029 (mon.0) 94 : audit [DBG] from='mgr.14164 172.21.15.29:0/3195927111' entity='mgr.smithi029.loxrfu' cmd=[{"prefix": "mgr metadata", "who": "smithi029.loxrfu", "id": "smithi029.loxrfu"}]: dispatch 2023-12-01T22:11:13.934 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:11:13 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:11:12.617965+0000 mon.smithi029 (mon.0) 95 : audit [DBG] from='mgr.14164 172.21.15.29:0/3195927111' entity='mgr.smithi029.loxrfu' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-12-01T22:11:13.934 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:11:13 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:11:12.618146+0000 mon.smithi029 (mon.0) 96 : audit [DBG] from='mgr.14164 172.21.15.29:0/3195927111' entity='mgr.smithi029.loxrfu' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-12-01T22:11:13.934 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:11:13 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:11:12.618289+0000 mon.smithi029 (mon.0) 97 : audit [DBG] from='mgr.14164 172.21.15.29:0/3195927111' entity='mgr.smithi029.loxrfu' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-12-01T22:11:13.935 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:11:13 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:11:12.631725+0000 mon.smithi029 (mon.0) 98 : cluster [INF] Manager daemon smithi029.loxrfu is now available 2023-12-01T22:11:13.935 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:11:13 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:11:12.745598+0000 mon.smithi029 (mon.0) 99 : audit [DBG] from='mgr.14164 172.21.15.29:0/3195927111' entity='mgr.smithi029.loxrfu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-01T22:11:13.935 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:11:13 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:11:12.748149+0000 mon.smithi029 (mon.0) 100 : audit [DBG] from='mgr.14164 172.21.15.29:0/3195927111' entity='mgr.smithi029.loxrfu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-01T22:11:13.935 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:11:13 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:11:12.773788+0000 mon.smithi029 (mon.0) 101 : audit [INF] from='mgr.14164 172.21.15.29:0/3195927111' entity='mgr.smithi029.loxrfu' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi029.loxrfu/mirror_snapshot_schedule"}]: dispatch 2023-12-01T22:11:13.935 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:11:13 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:11:12.777605+0000 mon.smithi029 (mon.0) 102 : audit [INF] from='mgr.14164 172.21.15.29:0/3195927111' entity='mgr.smithi029.loxrfu' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi029.loxrfu/trash_purge_schedule"}]: dispatch 2023-12-01T22:11:14.314 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:14.309+0000 7f9e67e32700 1 Processor -- start 2023-12-01T22:11:14.314 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:14.310+0000 7f9e67e32700 1 -- start start 2023-12-01T22:11:14.314 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:14.310+0000 7f9e67e32700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e600703d0 0x7f9e60100d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-01T22:11:14.315 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:14.310+0000 7f9e67e32700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e600fc730 con 0x7f9e600703d0 2023-12-01T22:11:14.315 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:14.310+0000 7f9e65bce700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e600703d0 0x7f9e60100d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:11:14.315 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:14.310+0000 7f9e65bce700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e600703d0 0x7f9e60100d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57290/0 (socket says 172.21.15.29:57290) 2023-12-01T22:11:14.315 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:14.310+0000 7f9e65bce700 1 -- 172.21.15.29:0/1448800867 learned_addr learned my addr 172.21.15.29:0/1448800867 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-12-01T22:11:14.319 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:14.311+0000 7f9e65bce700 1 -- 172.21.15.29:0/1448800867 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e60101230 con 0x7f9e600703d0 2023-12-01T22:11:14.319 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:14.311+0000 7f9e65bce700 1 --2- 172.21.15.29:0/1448800867 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e600703d0 0x7f9e60100d40 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f9e5c00b5f0 tx=0x7f9e5c006ee0).ready entity=mon.0 client_cookie=8b999816f33c1663 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:11:14.319 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:14.311+0000 7f9e64bcc700 1 -- 172.21.15.29:0/1448800867 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9e5c00d600 con 0x7f9e600703d0 2023-12-01T22:11:14.319 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:14.311+0000 7f9e64bcc700 1 -- 172.21.15.29:0/1448800867 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f9e5c00dbd0 con 0x7f9e600703d0 2023-12-01T22:11:14.320 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:14.311+0000 7f9e64bcc700 1 -- 172.21.15.29:0/1448800867 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9e5c011c00 con 0x7f9e600703d0 2023-12-01T22:11:14.320 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:14.312+0000 7f9e67e32700 1 -- 172.21.15.29:0/1448800867 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e600703d0 msgr2=0x7f9e60100d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:11:14.320 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:14.312+0000 7f9e67e32700 1 --2- 172.21.15.29:0/1448800867 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e600703d0 0x7f9e60100d40 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f9e5c00b5f0 tx=0x7f9e5c006ee0).stop 2023-12-01T22:11:14.320 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:14.312+0000 7f9e67e32700 1 -- 172.21.15.29:0/1448800867 shutdown_connections 2023-12-01T22:11:14.320 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:14.312+0000 7f9e67e32700 1 --2- 172.21.15.29:0/1448800867 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e600703d0 0x7f9e60100d40 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:11:14.320 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:14.312+0000 7f9e67e32700 1 -- 172.21.15.29:0/1448800867 >> 172.21.15.29:0/1448800867 conn(0x7f9e600f63c0 msgr2=0x7f9e600f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:11:14.320 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:14.312+0000 7f9e67e32700 1 -- 172.21.15.29:0/1448800867 shutdown_connections 2023-12-01T22:11:14.320 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:14.312+0000 7f9e67e32700 1 -- 172.21.15.29:0/1448800867 wait complete. 2023-12-01T22:11:14.320 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:14.313+0000 7f9e67e32700 1 Processor -- start 2023-12-01T22:11:14.320 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:14.313+0000 7f9e67e32700 1 -- start start 2023-12-01T22:11:14.320 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:14.313+0000 7f9e67e32700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e600703d0 0x7f9e6010cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:11:14.320 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:14.313+0000 7f9e67e32700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e6010d100 con 0x7f9e600703d0 2023-12-01T22:11:14.320 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:14.314+0000 7f9e65bce700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e600703d0 0x7f9e6010cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:11:14.321 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:14.314+0000 7f9e65bce700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e600703d0 0x7f9e6010cc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57304/0 (socket says 172.21.15.29:57304) 2023-12-01T22:11:14.321 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:14.314+0000 7f9e65bce700 1 -- 172.21.15.29:0/2898942425 learned_addr learned my addr 172.21.15.29:0/2898942425 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-12-01T22:11:14.321 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:14.314+0000 7f9e65bce700 1 -- 172.21.15.29:0/2898942425 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e5c00b200 con 0x7f9e600703d0 2023-12-01T22:11:14.321 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:14.314+0000 7f9e65bce700 1 --2- 172.21.15.29:0/2898942425 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e600703d0 0x7f9e6010cc10 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f9e5c00e040 tx=0x7f9e5c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:11:14.321 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:14.314+0000 7f9e4e7fc700 1 -- 172.21.15.29:0/2898942425 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9e5c00d9f0 con 0x7f9e600703d0 2023-12-01T22:11:14.321 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:14.314+0000 7f9e4e7fc700 1 -- 172.21.15.29:0/2898942425 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f9e5c01a9f0 con 0x7f9e600703d0 2023-12-01T22:11:14.321 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:14.314+0000 7f9e67e32700 1 -- 172.21.15.29:0/2898942425 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e6010d300 con 0x7f9e600703d0 2023-12-01T22:11:14.321 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:14.314+0000 7f9e4e7fc700 1 -- 172.21.15.29:0/2898942425 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9e5c01b8c0 con 0x7f9e600703d0 2023-12-01T22:11:14.321 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:14.314+0000 7f9e67e32700 1 -- 172.21.15.29:0/2898942425 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e6010d710 con 0x7f9e600703d0 2023-12-01T22:11:14.321 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:14.315+0000 7f9e4e7fc700 1 -- 172.21.15.29:0/2898942425 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 15) v1 ==== 36067+0+0 (secure 0 0 0) 0x7f9e5c00f070 con 0x7f9e600703d0 2023-12-01T22:11:14.321 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:14.315+0000 7f9e4e7fc700 1 --2- 172.21.15.29:0/2898942425 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f9e4802fdc0 0x7f9e48032280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:11:14.321 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:14.315+0000 7f9e4e7fc700 1 -- 172.21.15.29:0/2898942425 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9e5c0413a0 con 0x7f9e600703d0 2023-12-01T22:11:14.321 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:14.315+0000 7f9e67e32700 1 -- 172.21.15.29:0/2898942425 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e60045ad0 con 0x7f9e600703d0 2023-12-01T22:11:14.321 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:14.315+0000 7f9e653cd700 1 --2- 172.21.15.29:0/2898942425 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f9e4802fdc0 0x7f9e48032280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:11:14.322 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:14.316+0000 7f9e653cd700 1 --2- 172.21.15.29:0/2898942425 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f9e4802fdc0 0x7f9e48032280 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f9e500096a0 tx=0x7f9e50006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:11:14.322 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:14.320+0000 7f9e4e7fc700 1 -- 172.21.15.29:0/2898942425 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9e5c016070 con 0x7f9e600703d0 2023-12-01T22:11:14.492 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:14.490+0000 7f9e67e32700 1 -- 172.21.15.29:0/2898942425 --> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f9e60106b00 con 0x7f9e4802fdc0 2023-12-01T22:11:14.617 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:14.615+0000 7f9e4e7fc700 1 -- 172.21.15.29:0/2898942425 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 16) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f9e5c0257e0 con 0x7f9e600703d0 2023-12-01T22:11:14.644 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:14.642+0000 7f9e4e7fc700 1 -- 172.21.15.29:0/2898942425 <== mgr.14164 v2:172.21.15.29:6800/3698425734 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f9e60106b00 con 0x7f9e4802fdc0 2023-12-01T22:11:14.644 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: Self-signed certificate created 2023-12-01T22:11:14.646 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:14.644+0000 7f9e67e32700 1 -- 172.21.15.29:0/2898942425 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f9e4802fdc0 msgr2=0x7f9e48032280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:11:14.646 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:14.644+0000 7f9e67e32700 1 --2- 172.21.15.29:0/2898942425 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f9e4802fdc0 0x7f9e48032280 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f9e500096a0 tx=0x7f9e50006b40).stop 2023-12-01T22:11:14.646 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:14.644+0000 7f9e67e32700 1 -- 172.21.15.29:0/2898942425 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e600703d0 msgr2=0x7f9e6010cc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:11:14.646 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:14.644+0000 7f9e67e32700 1 --2- 172.21.15.29:0/2898942425 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e600703d0 0x7f9e6010cc10 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f9e5c00e040 tx=0x7f9e5c006ee0).stop 2023-12-01T22:11:14.647 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:14.644+0000 7f9e67e32700 1 -- 172.21.15.29:0/2898942425 shutdown_connections 2023-12-01T22:11:14.647 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:14.644+0000 7f9e67e32700 1 --2- 172.21.15.29:0/2898942425 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f9e4802fdc0 0x7f9e48032280 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:11:14.647 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:14.644+0000 7f9e67e32700 1 --2- 172.21.15.29:0/2898942425 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e600703d0 0x7f9e6010cc10 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:11:14.647 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:14.644+0000 7f9e67e32700 1 -- 172.21.15.29:0/2898942425 >> 172.21.15.29:0/2898942425 conn(0x7f9e600f63c0 msgr2=0x7f9e600f7ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:11:14.647 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:14.644+0000 7f9e67e32700 1 -- 172.21.15.29:0/2898942425 shutdown_connections 2023-12-01T22:11:14.647 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:14.644+0000 7f9e67e32700 1 -- 172.21.15.29:0/2898942425 wait complete. 2023-12-01T22:11:14.922 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:11:14 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:11:13.617432+0000 mgr.smithi029.loxrfu (mgr.14164) 1 : audit [DBG] from='client.14168 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2023-12-01T22:11:14.922 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:11:14 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:11:13.617958+0000 mon.smithi029 (mon.0) 103 : cluster [DBG] mgrmap e15: smithi029.loxrfu(active, since 1.0555s) 2023-12-01T22:11:14.923 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:11:14 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:11:13.624518+0000 mgr.smithi029.loxrfu (mgr.14164) 2 : audit [DBG] from='client.14168 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2023-12-01T22:11:14.986 INFO:teuthology.orchestra.run.smithi029.stderr:Creating initial admin user... 2023-12-01T22:11:14.987 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi029 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/5558ab6c-9096-11ee-95a2-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpou9lnrt4:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpxe9edifa:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp6hhe40g8:/tmp/dashboard.pw:z docker.io/ceph/ceph:v16.2.4 dashboard ac-user-create admin -i /tmp/dashboard.pw administrator --force-password --pwd-update-required 2023-12-01T22:11:15.400 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:15.394+0000 7f948f706700 1 Processor -- start 2023-12-01T22:11:15.400 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:15.394+0000 7f948f706700 1 -- start start 2023-12-01T22:11:15.400 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:15.394+0000 7f948f706700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94880fd2b0 0x7f94880fd6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-01T22:11:15.400 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:15.394+0000 7f948f706700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94880fdbc0 con 0x7f94880fd2b0 2023-12-01T22:11:15.400 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:15.394+0000 7f948d4a2700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94880fd2b0 0x7f94880fd6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:11:15.400 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:15.395+0000 7f948d4a2700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94880fd2b0 0x7f94880fd6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53330/0 (socket says 172.21.15.29:53330) 2023-12-01T22:11:15.400 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:15.395+0000 7f948d4a2700 1 -- 172.21.15.29:0/123021869 learned_addr learned my addr 172.21.15.29:0/123021869 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-12-01T22:11:15.400 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:15.395+0000 7f948d4a2700 1 -- 172.21.15.29:0/123021869 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94880fdd00 con 0x7f94880fd2b0 2023-12-01T22:11:15.400 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:15.395+0000 7f948d4a2700 1 --2- 172.21.15.29:0/123021869 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94880fd2b0 0x7f94880fd6d0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f948400b5f0 tx=0x7f9484006ee0).ready entity=mon.0 client_cookie=f854501eed62d6d2 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:11:15.400 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:15.395+0000 7f9477fff700 1 -- 172.21.15.29:0/123021869 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f948400d600 con 0x7f94880fd2b0 2023-12-01T22:11:15.400 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:15.395+0000 7f9477fff700 1 -- 172.21.15.29:0/123021869 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f948400dbd0 con 0x7f94880fd2b0 2023-12-01T22:11:15.400 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:15.395+0000 7f9477fff700 1 -- 172.21.15.29:0/123021869 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9484011c00 con 0x7f94880fd2b0 2023-12-01T22:11:15.400 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:15.396+0000 7f948f706700 1 -- 172.21.15.29:0/123021869 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94880fd2b0 msgr2=0x7f94880fd6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:11:15.401 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:15.396+0000 7f948f706700 1 --2- 172.21.15.29:0/123021869 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94880fd2b0 0x7f94880fd6d0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f948400b5f0 tx=0x7f9484006ee0).stop 2023-12-01T22:11:15.401 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:15.396+0000 7f948f706700 1 -- 172.21.15.29:0/123021869 shutdown_connections 2023-12-01T22:11:15.401 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:15.396+0000 7f948f706700 1 --2- 172.21.15.29:0/123021869 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94880fd2b0 0x7f94880fd6d0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:11:15.401 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:15.396+0000 7f948f706700 1 -- 172.21.15.29:0/123021869 >> 172.21.15.29:0/123021869 conn(0x7f94880f5b10 msgr2=0x7f94880f7f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:11:15.401 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:15.396+0000 7f948f706700 1 -- 172.21.15.29:0/123021869 shutdown_connections 2023-12-01T22:11:15.401 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:15.396+0000 7f948f706700 1 -- 172.21.15.29:0/123021869 wait complete. 2023-12-01T22:11:15.401 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:15.397+0000 7f948f706700 1 Processor -- start 2023-12-01T22:11:15.401 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:15.397+0000 7f948f706700 1 -- start start 2023-12-01T22:11:15.401 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:15.397+0000 7f948f706700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94880fd2b0 0x7f948810aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:11:15.401 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:15.397+0000 7f948f706700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f948810af30 con 0x7f94880fd2b0 2023-12-01T22:11:15.401 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:15.397+0000 7f948d4a2700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94880fd2b0 0x7f948810aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:11:15.401 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:15.397+0000 7f948d4a2700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94880fd2b0 0x7f948810aa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53338/0 (socket says 172.21.15.29:53338) 2023-12-01T22:11:15.401 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:15.397+0000 7f948d4a2700 1 -- 172.21.15.29:0/3363065002 learned_addr learned my addr 172.21.15.29:0/3363065002 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-12-01T22:11:15.403 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:15.397+0000 7f948d4a2700 1 -- 172.21.15.29:0/3363065002 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f948400b200 con 0x7f94880fd2b0 2023-12-01T22:11:15.403 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:15.398+0000 7f948d4a2700 1 --2- 172.21.15.29:0/3363065002 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94880fd2b0 0x7f948810aa40 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f9484002f40 tx=0x7f9484006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:11:15.403 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:15.398+0000 7f9475ffb700 1 -- 172.21.15.29:0/3363065002 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f948401b410 con 0x7f94880fd2b0 2023-12-01T22:11:15.403 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:15.398+0000 7f9475ffb700 1 -- 172.21.15.29:0/3363065002 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f948401b9e0 con 0x7f94880fd2b0 2023-12-01T22:11:15.403 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:15.398+0000 7f948f706700 1 -- 172.21.15.29:0/3363065002 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f948810b130 con 0x7f94880fd2b0 2023-12-01T22:11:15.403 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:15.398+0000 7f9475ffb700 1 -- 172.21.15.29:0/3363065002 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f948401a8b0 con 0x7f94880fd2b0 2023-12-01T22:11:15.403 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:15.398+0000 7f948f706700 1 -- 172.21.15.29:0/3363065002 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f948810b540 con 0x7f94880fd2b0 2023-12-01T22:11:15.403 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:15.399+0000 7f9475ffb700 1 -- 172.21.15.29:0/3363065002 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f948400f070 con 0x7f94880fd2b0 2023-12-01T22:11:15.403 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:15.399+0000 7f948f706700 1 -- 172.21.15.29:0/3363065002 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9488045ad0 con 0x7f94880fd2b0 2023-12-01T22:11:15.403 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:15.399+0000 7f9475ffb700 1 --2- 172.21.15.29:0/3363065002 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f947002ff00 0x7f94700323c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:11:15.403 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:15.399+0000 7f9475ffb700 1 -- 172.21.15.29:0/3363065002 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f94840413a0 con 0x7f94880fd2b0 2023-12-01T22:11:15.403 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:15.400+0000 7f948cca1700 1 --2- 172.21.15.29:0/3363065002 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f947002ff00 0x7f94700323c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:11:15.403 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:15.400+0000 7f948cca1700 1 --2- 172.21.15.29:0/3363065002 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f947002ff00 0x7f94700323c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f94780096a0 tx=0x7f9478006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:11:15.405 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:15.404+0000 7f9475ffb700 1 -- 172.21.15.29:0/3363065002 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9484019910 con 0x7f94880fd2b0 2023-12-01T22:11:15.589 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:15.587+0000 7f948f706700 1 -- 172.21.15.29:0/3363065002 --> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f9488058f20 con 0x7f947002ff00 2023-12-01T22:11:15.842 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:15.840+0000 7f9475ffb700 1 -- 172.21.15.29:0/3363065002 <== mgr.14164 v2:172.21.15.29:6800/3698425734 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f9488058f20 con 0x7f947002ff00 2023-12-01T22:11:15.842 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$ZqgY5nCVR/IKibppX.muXeG2os53U4BdBY463pP8KNDUQUX8LWNCW", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1701468675, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2023-12-01T22:11:15.844 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:15.842+0000 7f948f706700 1 -- 172.21.15.29:0/3363065002 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f947002ff00 msgr2=0x7f94700323c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:11:15.844 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:15.842+0000 7f948f706700 1 --2- 172.21.15.29:0/3363065002 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f947002ff00 0x7f94700323c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f94780096a0 tx=0x7f9478006b40).stop 2023-12-01T22:11:15.844 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:15.842+0000 7f948f706700 1 -- 172.21.15.29:0/3363065002 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94880fd2b0 msgr2=0x7f948810aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:11:15.845 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:15.842+0000 7f948f706700 1 --2- 172.21.15.29:0/3363065002 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94880fd2b0 0x7f948810aa40 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f9484002f40 tx=0x7f9484006ee0).stop 2023-12-01T22:11:15.845 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:15.842+0000 7f948f706700 1 -- 172.21.15.29:0/3363065002 shutdown_connections 2023-12-01T22:11:15.845 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:15.842+0000 7f948f706700 1 --2- 172.21.15.29:0/3363065002 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f947002ff00 0x7f94700323c0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:11:15.845 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:15.842+0000 7f948f706700 1 --2- 172.21.15.29:0/3363065002 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94880fd2b0 0x7f948810aa40 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:11:15.845 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:15.842+0000 7f948f706700 1 -- 172.21.15.29:0/3363065002 >> 172.21.15.29:0/3363065002 conn(0x7f94880f5b10 msgr2=0x7f94880ff330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:11:15.845 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:15.843+0000 7f948f706700 1 -- 172.21.15.29:0/3363065002 shutdown_connections 2023-12-01T22:11:15.845 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:15.843+0000 7f948f706700 1 -- 172.21.15.29:0/3363065002 wait complete. 2023-12-01T22:11:15.873 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:11:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:11:14.492128+0000 mgr.smithi029.loxrfu (mgr.14164) 3 : audit [DBG] from='client.14174 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2023-12-01T22:11:15.873 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:11:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:11:14.617836+0000 mon.smithi029 (mon.0) 104 : cluster [DBG] mgrmap e16: smithi029.loxrfu(active, since 2s) 2023-12-01T22:11:15.873 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:11:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:11:14.640916+0000 mon.smithi029 (mon.0) 105 : audit [INF] from='mgr.14164 172.21.15.29:0/3195927111' entity='mgr.smithi029.loxrfu' 2023-12-01T22:11:15.874 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:11:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:11:14.642965+0000 mon.smithi029 (mon.0) 106 : audit [INF] from='mgr.14164 172.21.15.29:0/3195927111' entity='mgr.smithi029.loxrfu' 2023-12-01T22:11:16.211 INFO:teuthology.orchestra.run.smithi029.stderr:Fetching dashboard port number... 2023-12-01T22:11:16.211 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi029 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/5558ab6c-9096-11ee-95a2-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpou9lnrt4:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpxe9edifa:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config get mgr mgr/dashboard/ssl_server_port 2023-12-01T22:11:16.612 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:16.609+0000 7f3886863700 1 Processor -- start 2023-12-01T22:11:16.612 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:16.609+0000 7f3886863700 1 -- start start 2023-12-01T22:11:16.612 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:16.610+0000 7f3886863700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38800703d0 0x7f38800fe940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-01T22:11:16.613 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:16.610+0000 7f3886863700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38800fee30 con 0x7f38800703d0 2023-12-01T22:11:16.613 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:16.610+0000 7f387ffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38800703d0 0x7f38800fe940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:11:16.613 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:16.610+0000 7f387ffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38800703d0 0x7f38800fe940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53352/0 (socket says 172.21.15.29:53352) 2023-12-01T22:11:16.613 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:16.610+0000 7f387ffff700 1 -- 172.21.15.29:0/52030701 learned_addr learned my addr 172.21.15.29:0/52030701 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-12-01T22:11:16.613 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:16.610+0000 7f387ffff700 1 -- 172.21.15.29:0/52030701 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38800fef70 con 0x7f38800703d0 2023-12-01T22:11:16.613 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:16.610+0000 7f387ffff700 1 --2- 172.21.15.29:0/52030701 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38800703d0 0x7f38800fe940 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f386800b5f0 tx=0x7f3868006ee0).ready entity=mon.0 client_cookie=bd3644e076f199ab server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:11:16.615 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:16.610+0000 7f387effd700 1 -- 172.21.15.29:0/52030701 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f386800d600 con 0x7f38800703d0 2023-12-01T22:11:16.616 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:16.611+0000 7f387effd700 1 -- 172.21.15.29:0/52030701 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f386800dbd0 con 0x7f38800703d0 2023-12-01T22:11:16.616 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:16.611+0000 7f387effd700 1 -- 172.21.15.29:0/52030701 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3868011c00 con 0x7f38800703d0 2023-12-01T22:11:16.616 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:16.611+0000 7f3886863700 1 -- 172.21.15.29:0/52030701 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38800703d0 msgr2=0x7f38800fe940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:11:16.616 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:16.611+0000 7f3886863700 1 --2- 172.21.15.29:0/52030701 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38800703d0 0x7f38800fe940 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f386800b5f0 tx=0x7f3868006ee0).stop 2023-12-01T22:11:16.616 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:16.611+0000 7f3886863700 1 -- 172.21.15.29:0/52030701 shutdown_connections 2023-12-01T22:11:16.616 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:16.611+0000 7f3886863700 1 --2- 172.21.15.29:0/52030701 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38800703d0 0x7f38800fe940 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:11:16.616 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:16.611+0000 7f3886863700 1 -- 172.21.15.29:0/52030701 >> 172.21.15.29:0/52030701 conn(0x7f38800f63c0 msgr2=0x7f38800f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:11:16.616 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:16.611+0000 7f3886863700 1 -- 172.21.15.29:0/52030701 shutdown_connections 2023-12-01T22:11:16.616 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:16.611+0000 7f3886863700 1 -- 172.21.15.29:0/52030701 wait complete. 2023-12-01T22:11:16.616 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:16.612+0000 7f3886863700 1 Processor -- start 2023-12-01T22:11:16.616 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:16.612+0000 7f3886863700 1 -- start start 2023-12-01T22:11:16.616 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:16.612+0000 7f3886863700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38800703d0 0x7f388010cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:11:16.616 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:16.612+0000 7f3886863700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f388010d0e0 con 0x7f38800703d0 2023-12-01T22:11:16.617 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:16.612+0000 7f387ffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38800703d0 0x7f388010cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:11:16.617 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:16.613+0000 7f387ffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38800703d0 0x7f388010cbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53360/0 (socket says 172.21.15.29:53360) 2023-12-01T22:11:16.617 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:16.613+0000 7f387ffff700 1 -- 172.21.15.29:0/2323422245 learned_addr learned my addr 172.21.15.29:0/2323422245 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-12-01T22:11:16.617 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:16.613+0000 7f387ffff700 1 -- 172.21.15.29:0/2323422245 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f386800b200 con 0x7f38800703d0 2023-12-01T22:11:16.617 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:16.613+0000 7f387ffff700 1 --2- 172.21.15.29:0/2323422245 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38800703d0 0x7f388010cbf0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f386800b190 tx=0x7f3868006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:11:16.617 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:16.613+0000 7f387cff9700 1 -- 172.21.15.29:0/2323422245 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f386801b410 con 0x7f38800703d0 2023-12-01T22:11:16.617 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:16.613+0000 7f387cff9700 1 -- 172.21.15.29:0/2323422245 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f386801b9e0 con 0x7f38800703d0 2023-12-01T22:11:16.617 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:16.613+0000 7f3886863700 1 -- 172.21.15.29:0/2323422245 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f388010d2e0 con 0x7f38800703d0 2023-12-01T22:11:16.617 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:16.613+0000 7f387cff9700 1 -- 172.21.15.29:0/2323422245 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f386801a880 con 0x7f38800703d0 2023-12-01T22:11:16.617 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:16.613+0000 7f3886863700 1 -- 172.21.15.29:0/2323422245 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f388010d6f0 con 0x7f38800703d0 2023-12-01T22:11:16.617 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:16.614+0000 7f387cff9700 1 -- 172.21.15.29:0/2323422245 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f386800f070 con 0x7f38800703d0 2023-12-01T22:11:16.618 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:16.614+0000 7f387cff9700 1 --2- 172.21.15.29:0/2323422245 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f386002ff00 0x7f38600323c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:11:16.618 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:16.614+0000 7f3886863700 1 -- 172.21.15.29:0/2323422245 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3880045ad0 con 0x7f38800703d0 2023-12-01T22:11:16.618 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:16.614+0000 7f387cff9700 1 -- 172.21.15.29:0/2323422245 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3868041500 con 0x7f38800703d0 2023-12-01T22:11:16.618 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:16.614+0000 7f387f7fe700 1 --2- 172.21.15.29:0/2323422245 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f386002ff00 0x7f38600323c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:11:16.618 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:16.615+0000 7f387f7fe700 1 --2- 172.21.15.29:0/2323422245 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f386002ff00 0x7f38600323c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f38700096a0 tx=0x7f3870006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:11:16.620 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:16.619+0000 7f387cff9700 1 -- 172.21.15.29:0/2323422245 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3868022af0 con 0x7f38800703d0 2023-12-01T22:11:16.786 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:16.783+0000 7f3886863700 1 -- 172.21.15.29:0/2323422245 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f38800ffb90 con 0x7f38800703d0 2023-12-01T22:11:16.786 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:16.784+0000 7f387cff9700 1 -- 172.21.15.29:0/2323422245 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v9) v1 ==== 112+0+5 (secure 0 0 0) 0x7f3868016070 con 0x7f38800703d0 2023-12-01T22:11:16.786 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 8443 2023-12-01T22:11:16.788 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:16.785+0000 7f3886863700 1 -- 172.21.15.29:0/2323422245 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f386002ff00 msgr2=0x7f38600323c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:11:16.788 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:16.785+0000 7f3886863700 1 --2- 172.21.15.29:0/2323422245 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f386002ff00 0x7f38600323c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f38700096a0 tx=0x7f3870006b40).stop 2023-12-01T22:11:16.788 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:16.786+0000 7f3886863700 1 -- 172.21.15.29:0/2323422245 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38800703d0 msgr2=0x7f388010cbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:11:16.788 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:16.786+0000 7f3886863700 1 --2- 172.21.15.29:0/2323422245 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38800703d0 0x7f388010cbf0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f386800b190 tx=0x7f3868006ee0).stop 2023-12-01T22:11:16.788 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:16.786+0000 7f3886863700 1 -- 172.21.15.29:0/2323422245 shutdown_connections 2023-12-01T22:11:16.788 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:16.786+0000 7f3886863700 1 --2- 172.21.15.29:0/2323422245 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f386002ff00 0x7f38600323c0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:11:16.788 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:16.786+0000 7f3886863700 1 --2- 172.21.15.29:0/2323422245 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38800703d0 0x7f388010cbf0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:11:16.788 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:16.786+0000 7f3886863700 1 -- 172.21.15.29:0/2323422245 >> 172.21.15.29:0/2323422245 conn(0x7f38800f63c0 msgr2=0x7f38800f7e40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:11:16.789 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:16.787+0000 7f3886863700 1 -- 172.21.15.29:0/2323422245 shutdown_connections 2023-12-01T22:11:16.789 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2023-12-01T22:11:16.787+0000 7f3886863700 1 -- 172.21.15.29:0/2323422245 wait complete. 2023-12-01T22:11:17.072 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:11:16 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:11:15.589396+0000 mgr.smithi029.loxrfu (mgr.14164) 4 : audit [DBG] from='client.14176 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2023-12-01T22:11:17.072 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:11:16 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:11:15.840859+0000 mon.smithi029 (mon.0) 107 : audit [INF] from='mgr.14164 172.21.15.29:0/3195927111' entity='mgr.smithi029.loxrfu' 2023-12-01T22:11:17.073 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:11:16 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:11:16.785127+0000 mon.smithi029 (mon.0) 108 : audit [DBG] from='client.? 172.21.15.29:0/2323422245' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2023-12-01T22:11:17.136 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: systemctl is-enabled firewalld.service 2023-12-01T22:11:17.147 INFO:teuthology.orchestra.run.smithi029.stderr:systemctl: disabled 2023-12-01T22:11:17.148 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: systemctl is-active firewalld.service 2023-12-01T22:11:17.157 INFO:teuthology.orchestra.run.smithi029.stderr:systemctl: inactive 2023-12-01T22:11:17.157 INFO:teuthology.orchestra.run.smithi029.stderr:firewalld.service is not enabled 2023-12-01T22:11:17.157 INFO:teuthology.orchestra.run.smithi029.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2023-12-01T22:11:17.161 INFO:teuthology.orchestra.run.smithi029.stderr:Ceph Dashboard is now available at: 2023-12-01T22:11:17.161 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-12-01T22:11:17.161 INFO:teuthology.orchestra.run.smithi029.stderr: URL: https://smithi029.front.sepia.ceph.com:8443/ 2023-12-01T22:11:17.161 INFO:teuthology.orchestra.run.smithi029.stderr: User: admin 2023-12-01T22:11:17.161 INFO:teuthology.orchestra.run.smithi029.stderr: Password: z2nj3nhbgi 2023-12-01T22:11:17.161 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-12-01T22:11:17.162 INFO:teuthology.orchestra.run.smithi029.stderr:You can access the Ceph CLI with: 2023-12-01T22:11:17.162 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-12-01T22:11:17.162 INFO:teuthology.orchestra.run.smithi029.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2023-12-01T22:11:17.162 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-12-01T22:11:17.162 INFO:teuthology.orchestra.run.smithi029.stderr:Please consider enabling telemetry to help improve Ceph: 2023-12-01T22:11:17.162 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-12-01T22:11:17.162 INFO:teuthology.orchestra.run.smithi029.stderr: ceph telemetry on 2023-12-01T22:11:17.162 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-12-01T22:11:17.162 INFO:teuthology.orchestra.run.smithi029.stderr:For more information see: 2023-12-01T22:11:17.162 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-12-01T22:11:17.163 INFO:teuthology.orchestra.run.smithi029.stderr: https://docs.ceph.com/docs/pacific/mgr/telemetry/ 2023-12-01T22:11:17.163 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-12-01T22:11:17.163 INFO:teuthology.orchestra.run.smithi029.stderr:Bootstrap complete. 2023-12-01T22:11:17.163 INFO:teuthology.orchestra.run.smithi029.stderr:Releasing lock 140445701048808 on /run/cephadm/5558ab6c-9096-11ee-95a2-87774f69a715.lock 2023-12-01T22:11:17.163 INFO:teuthology.orchestra.run.smithi029.stderr:Lock 140445701048808 released on /run/cephadm/5558ab6c-9096-11ee-95a2-87774f69a715.lock 2023-12-01T22:11:17.202 INFO:tasks.cephadm:Fetching config... 2023-12-01T22:11:17.203 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-12-01T22:11:17.203 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2023-12-01T22:11:17.265 INFO:tasks.cephadm:Fetching client.admin keyring... 2023-12-01T22:11:17.265 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-12-01T22:11:17.265 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2023-12-01T22:11:17.328 INFO:tasks.cephadm:Fetching mon keyring... 2023-12-01T22:11:17.329 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-12-01T22:11:17.329 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/var/lib/ceph/5558ab6c-9096-11ee-95a2-87774f69a715/mon.smithi029/keyring of=/dev/stdout 2023-12-01T22:11:17.404 INFO:tasks.cephadm:Fetching pub ssh key... 2023-12-01T22:11:17.405 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-12-01T22:11:17.405 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2023-12-01T22:11:17.467 INFO:tasks.cephadm:Installing pub ssh key for root users... 2023-12-01T22:11:17.468 DEBUG:teuthology.orchestra.run.smithi029:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDGfrW1MXb6ua3LhHOmlc1sD/DX4/0YK/khj7Z6/nnegRYI0nWfxKFNguc9/TKDAr9QM7A2NH4gmyTorc7ZSq1KjPv8XwgTaRdEZSFA6/H0YOViA511F3V3/irIotSuZS4xJi2kSV8pzLdPQfOUPBJ4SAw9/3JLYZtnInMI3AhDxiXDPcpDICSpyoqCsxjwwPa/n8KXTZdaPmc0L56Dfeh6Uu9Vztx+uQFt3U3RwI75U7ev3y3a3Pa0/uDw1QRj6RR8+BjL5xwOAWXPAnxp9JlHNUwi1iWolVDycAaN1jn5UOvxrVdcMR8BfB0a84WWFb/XHJxVXIiipuJG1szevgcB9F1VJGO0OEua9ZuNCVs7EtScuM0E81wFuy2GDSwCqqxw+JmEr3KS8vyYXXJ1nHJ3xZ3OZqF9VwZQ3uNe5HfS0Xg+WwfUkmDJRS6ZOWSbcO6LuNANncEfXKoT1ZxPutNOxsl3IUtoZWVgPdTxOrfNTCKh8iA7ax1QvcbJnydJeRk= ceph-5558ab6c-9096-11ee-95a2-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-12-01T22:11:17.553 INFO:teuthology.orchestra.run.smithi029.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDGfrW1MXb6ua3LhHOmlc1sD/DX4/0YK/khj7Z6/nnegRYI0nWfxKFNguc9/TKDAr9QM7A2NH4gmyTorc7ZSq1KjPv8XwgTaRdEZSFA6/H0YOViA511F3V3/irIotSuZS4xJi2kSV8pzLdPQfOUPBJ4SAw9/3JLYZtnInMI3AhDxiXDPcpDICSpyoqCsxjwwPa/n8KXTZdaPmc0L56Dfeh6Uu9Vztx+uQFt3U3RwI75U7ev3y3a3Pa0/uDw1QRj6RR8+BjL5xwOAWXPAnxp9JlHNUwi1iWolVDycAaN1jn5UOvxrVdcMR8BfB0a84WWFb/XHJxVXIiipuJG1szevgcB9F1VJGO0OEua9ZuNCVs7EtScuM0E81wFuy2GDSwCqqxw+JmEr3KS8vyYXXJ1nHJ3xZ3OZqF9VwZQ3uNe5HfS0Xg+WwfUkmDJRS6ZOWSbcO6LuNANncEfXKoT1ZxPutNOxsl3IUtoZWVgPdTxOrfNTCKh8iA7ax1QvcbJnydJeRk= ceph-5558ab6c-9096-11ee-95a2-87774f69a715 2023-12-01T22:11:17.566 DEBUG:teuthology.orchestra.run.smithi073:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDGfrW1MXb6ua3LhHOmlc1sD/DX4/0YK/khj7Z6/nnegRYI0nWfxKFNguc9/TKDAr9QM7A2NH4gmyTorc7ZSq1KjPv8XwgTaRdEZSFA6/H0YOViA511F3V3/irIotSuZS4xJi2kSV8pzLdPQfOUPBJ4SAw9/3JLYZtnInMI3AhDxiXDPcpDICSpyoqCsxjwwPa/n8KXTZdaPmc0L56Dfeh6Uu9Vztx+uQFt3U3RwI75U7ev3y3a3Pa0/uDw1QRj6RR8+BjL5xwOAWXPAnxp9JlHNUwi1iWolVDycAaN1jn5UOvxrVdcMR8BfB0a84WWFb/XHJxVXIiipuJG1szevgcB9F1VJGO0OEua9ZuNCVs7EtScuM0E81wFuy2GDSwCqqxw+JmEr3KS8vyYXXJ1nHJ3xZ3OZqF9VwZQ3uNe5HfS0Xg+WwfUkmDJRS6ZOWSbcO6LuNANncEfXKoT1ZxPutNOxsl3IUtoZWVgPdTxOrfNTCKh8iA7ax1QvcbJnydJeRk= ceph-5558ab6c-9096-11ee-95a2-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-12-01T22:11:17.613 INFO:teuthology.orchestra.run.smithi073.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDGfrW1MXb6ua3LhHOmlc1sD/DX4/0YK/khj7Z6/nnegRYI0nWfxKFNguc9/TKDAr9QM7A2NH4gmyTorc7ZSq1KjPv8XwgTaRdEZSFA6/H0YOViA511F3V3/irIotSuZS4xJi2kSV8pzLdPQfOUPBJ4SAw9/3JLYZtnInMI3AhDxiXDPcpDICSpyoqCsxjwwPa/n8KXTZdaPmc0L56Dfeh6Uu9Vztx+uQFt3U3RwI75U7ev3y3a3Pa0/uDw1QRj6RR8+BjL5xwOAWXPAnxp9JlHNUwi1iWolVDycAaN1jn5UOvxrVdcMR8BfB0a84WWFb/XHJxVXIiipuJG1szevgcB9F1VJGO0OEua9ZuNCVs7EtScuM0E81wFuy2GDSwCqqxw+JmEr3KS8vyYXXJ1nHJ3xZ3OZqF9VwZQ3uNe5HfS0Xg+WwfUkmDJRS6ZOWSbcO6LuNANncEfXKoT1ZxPutNOxsl3IUtoZWVgPdTxOrfNTCKh8iA7ax1QvcbJnydJeRk= ceph-5558ab6c-9096-11ee-95a2-87774f69a715 2023-12-01T22:11:17.625 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2023-12-01T22:11:19.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-12-01T22:11:19.344+0000 7f395c538700 1 -- 172.21.15.29:0/3374153987 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f39540f6510 msgr2=0x7f39540f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:11:19.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-12-01T22:11:19.344+0000 7f395c538700 1 --2- 172.21.15.29:0/3374153987 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f39540f6510 0x7f39540f6930 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f3948009fb0 tx=0x7f3948009b30).stop 2023-12-01T22:11:19.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-12-01T22:11:19.344+0000 7f395c538700 1 -- 172.21.15.29:0/3374153987 shutdown_connections 2023-12-01T22:11:19.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-12-01T22:11:19.344+0000 7f395c538700 1 --2- 172.21.15.29:0/3374153987 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f39540f6510 0x7f39540f6930 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:11:19.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-12-01T22:11:19.344+0000 7f395c538700 1 -- 172.21.15.29:0/3374153987 >> 172.21.15.29:0/3374153987 conn(0x7f39540f1c70 msgr2=0x7f39540f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:11:19.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-12-01T22:11:19.344+0000 7f395c538700 1 -- 172.21.15.29:0/3374153987 shutdown_connections 2023-12-01T22:11:19.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-12-01T22:11:19.344+0000 7f395c538700 1 -- 172.21.15.29:0/3374153987 wait complete. 2023-12-01T22:11:19.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-12-01T22:11:19.345+0000 7f395c538700 1 Processor -- start 2023-12-01T22:11:19.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-12-01T22:11:19.345+0000 7f395c538700 1 -- start start 2023-12-01T22:11:19.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-12-01T22:11:19.345+0000 7f395c538700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f39540f6510 0x7f395406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:11:19.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-12-01T22:11:19.345+0000 7f395c538700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3954069e20 con 0x7f39540f6510 2023-12-01T22:11:19.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-12-01T22:11:19.345+0000 7f395a2d4700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f39540f6510 0x7f395406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:11:19.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-12-01T22:11:19.345+0000 7f395a2d4700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f39540f6510 0x7f395406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53388/0 (socket says 172.21.15.29:53388) 2023-12-01T22:11:19.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-12-01T22:11:19.345+0000 7f395a2d4700 1 -- 172.21.15.29:0/2086452843 learned_addr learned my addr 172.21.15.29:0/2086452843 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-12-01T22:11:19.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-12-01T22:11:19.346+0000 7f395a2d4700 1 -- 172.21.15.29:0/2086452843 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3948005040 con 0x7f39540f6510 2023-12-01T22:11:19.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-12-01T22:11:19.346+0000 7f395a2d4700 1 --2- 172.21.15.29:0/2086452843 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f39540f6510 0x7f395406b9a0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f3948004dd0 tx=0x7f394800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:11:19.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-12-01T22:11:19.346+0000 7f3946ffd700 1 -- 172.21.15.29:0/2086452843 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3948009320 con 0x7f39540f6510 2023-12-01T22:11:19.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-12-01T22:11:19.346+0000 7f3946ffd700 1 -- 172.21.15.29:0/2086452843 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f3948013680 con 0x7f39540f6510 2023-12-01T22:11:19.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-12-01T22:11:19.346+0000 7f395c538700 1 -- 172.21.15.29:0/2086452843 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3954069fc0 con 0x7f39540f6510 2023-12-01T22:11:19.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-12-01T22:11:19.346+0000 7f3946ffd700 1 -- 172.21.15.29:0/2086452843 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f394801ecb0 con 0x7f39540f6510 2023-12-01T22:11:19.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-12-01T22:11:19.346+0000 7f395c538700 1 -- 172.21.15.29:0/2086452843 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f395406a450 con 0x7f39540f6510 2023-12-01T22:11:19.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-12-01T22:11:19.347+0000 7f3946ffd700 1 -- 172.21.15.29:0/2086452843 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f394801e420 con 0x7f39540f6510 2023-12-01T22:11:19.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-12-01T22:11:19.347+0000 7f3946ffd700 1 --2- 172.21.15.29:0/2086452843 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f394002ff60 0x7f3940032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:11:19.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-12-01T22:11:19.347+0000 7f3946ffd700 1 -- 172.21.15.29:0/2086452843 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3948040d70 con 0x7f39540f6510 2023-12-01T22:11:19.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-12-01T22:11:19.347+0000 7f395c538700 1 -- 172.21.15.29:0/2086452843 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3954059100 con 0x7f39540f6510 2023-12-01T22:11:19.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-12-01T22:11:19.347+0000 7f3959ad3700 1 --2- 172.21.15.29:0/2086452843 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f394002ff60 0x7f3940032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:11:19.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-12-01T22:11:19.348+0000 7f3959ad3700 1 --2- 172.21.15.29:0/2086452843 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f394002ff60 0x7f3940032420 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f39500096a0 tx=0x7f3950006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:11:19.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-12-01T22:11:19.352+0000 7f3946ffd700 1 -- 172.21.15.29:0/2086452843 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f39480095f0 con 0x7f39540f6510 2023-12-01T22:11:19.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-12-01T22:11:19.517+0000 7f395c538700 1 -- 172.21.15.29:0/2086452843 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f39540fbf50 con 0x7f39540f6510 2023-12-01T22:11:19.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-12-01T22:11:19.520+0000 7f3946ffd700 1 -- 172.21.15.29:0/2086452843 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v10)=0 v10) v1 ==== 125+0+0 (secure 0 0 0) 0x7f394801db70 con 0x7f39540f6510 2023-12-01T22:11:19.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-12-01T22:11:19.522+0000 7f395c538700 1 -- 172.21.15.29:0/2086452843 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f394002ff60 msgr2=0x7f3940032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:11:19.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-12-01T22:11:19.522+0000 7f395c538700 1 --2- 172.21.15.29:0/2086452843 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f394002ff60 0x7f3940032420 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f39500096a0 tx=0x7f3950006b40).stop 2023-12-01T22:11:19.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-12-01T22:11:19.522+0000 7f395c538700 1 -- 172.21.15.29:0/2086452843 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f39540f6510 msgr2=0x7f395406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:11:19.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-12-01T22:11:19.522+0000 7f395c538700 1 --2- 172.21.15.29:0/2086452843 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f39540f6510 0x7f395406b9a0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f3948004dd0 tx=0x7f394800b040).stop 2023-12-01T22:11:19.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-12-01T22:11:19.522+0000 7f395c538700 1 -- 172.21.15.29:0/2086452843 shutdown_connections 2023-12-01T22:11:19.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-12-01T22:11:19.522+0000 7f395c538700 1 --2- 172.21.15.29:0/2086452843 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f394002ff60 0x7f3940032420 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:11:19.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-12-01T22:11:19.522+0000 7f395c538700 1 --2- 172.21.15.29:0/2086452843 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f39540f6510 0x7f395406b9a0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:11:19.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-12-01T22:11:19.522+0000 7f395c538700 1 -- 172.21.15.29:0/2086452843 >> 172.21.15.29:0/2086452843 conn(0x7f39540f1c70 msgr2=0x7f39540f2740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:11:19.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-12-01T22:11:19.522+0000 7f395c538700 1 -- 172.21.15.29:0/2086452843 shutdown_connections 2023-12-01T22:11:19.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-12-01T22:11:19.522+0000 7f395c538700 1 -- 172.21.15.29:0/2086452843 wait complete. 2023-12-01T22:11:19.822 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:11:19 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:11:18.652125+0000 mon.smithi029 (mon.0) 109 : cluster [DBG] mgrmap e17: smithi029.loxrfu(active, since 6s) 2023-12-01T22:11:19.823 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:11:19 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:11:19.521394+0000 mon.smithi029 (mon.0) 110 : audit [INF] from='client.? 172.21.15.29:0/2086452843' entity='client.admin' 2023-12-01T22:11:19.904 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi073 2023-12-01T22:11:19.904 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2023-12-01T22:11:19.905 DEBUG:teuthology.orchestra.run.smithi073:> dd of=/etc/ceph/ceph.conf 2023-12-01T22:11:19.926 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2023-12-01T22:11:19.926 DEBUG:teuthology.orchestra.run.smithi073:> dd of=/etc/ceph/ceph.client.admin.keyring 2023-12-01T22:11:19.988 INFO:tasks.cephadm:Adding host smithi073 to orchestrator... 2023-12-01T22:11:19.988 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph orch host add smithi073 2023-12-01T22:11:22.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:11:21 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:11:20.847801+0000 mon.smithi029 (mon.0) 111 : audit [INF] from='mgr.14164 172.21.15.29:0/3195927111' entity='mgr.smithi029.loxrfu' 2023-12-01T22:11:22.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:11:21 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:11:21.094343+0000 mon.smithi029 (mon.0) 112 : audit [INF] from='mgr.14164 172.21.15.29:0/3195927111' entity='mgr.smithi029.loxrfu' 2023-12-01T22:11:22.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:11:21 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:11:21.780713+0000 mon.smithi029 (mon.0) 113 : audit [INF] from='mgr.14164 172.21.15.29:0/3195927111' entity='mgr.smithi029.loxrfu' 2023-12-01T22:11:25.526 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:11:25 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:11:24.235582+0000 mon.smithi029 (mon.0) 114 : audit [INF] from='mgr.14164 172.21.15.29:0/3195927111' entity='mgr.smithi029.loxrfu' 2023-12-01T22:11:25.527 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:11:25 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:11:24.533215+0000 mon.smithi029 (mon.0) 115 : audit [INF] from='mgr.14164 172.21.15.29:0/3195927111' entity='mgr.smithi029.loxrfu' 2023-12-01T22:11:25.527 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:11:25 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:11:24.535420+0000 mon.smithi029 (mon.0) 116 : audit [INF] from='mgr.14164 172.21.15.29:0/3195927111' entity='mgr.smithi029.loxrfu' 2023-12-01T22:11:25.527 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:11:25 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cephadm 2023-12-01T22:11:24.542509+0000 mgr.smithi029.loxrfu (mgr.14164) 5 : cephadm [INF] Deploying daemon alertmanager.smithi029 on smithi029 2023-12-01T22:11:34.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:11:33 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:11:32.619343+0000 mgr.smithi029.loxrfu (mgr.14164) 6 : cluster [DBG] pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:11:36.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:11:35 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:11:34.619705+0000 mgr.smithi029.loxrfu (mgr.14164) 7 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:11:38.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:11:37 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:11:36.620070+0000 mgr.smithi029.loxrfu (mgr.14164) 8 : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:11:40.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:11:39 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:11:38.620291+0000 mgr.smithi029.loxrfu (mgr.14164) 9 : cluster [DBG] pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:11:41.777 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:11:41 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:11:40.620487+0000 mgr.smithi029.loxrfu (mgr.14164) 10 : cluster [DBG] pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:11:42.338 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:42.335+0000 7f982d4ad700 1 -- 172.21.15.73:0/2996557570 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98280f47c0 msgr2=0x7f98280f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:11:42.338 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:42.335+0000 7f982d4ad700 1 --2- 172.21.15.73:0/2996557570 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98280f47c0 0x7f98280f4be0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f9818009fb0 tx=0x7f9818009b30).stop 2023-12-01T22:11:42.338 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:42.335+0000 7f982d4ad700 1 -- 172.21.15.73:0/2996557570 shutdown_connections 2023-12-01T22:11:42.338 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:42.335+0000 7f982d4ad700 1 --2- 172.21.15.73:0/2996557570 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98280f47c0 0x7f98280f4be0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:11:42.338 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:42.335+0000 7f982d4ad700 1 -- 172.21.15.73:0/2996557570 >> 172.21.15.73:0/2996557570 conn(0x7f98280f02e0 msgr2=0x7f98280f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:11:42.338 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:42.335+0000 7f982d4ad700 1 -- 172.21.15.73:0/2996557570 shutdown_connections 2023-12-01T22:11:42.339 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:42.335+0000 7f982d4ad700 1 -- 172.21.15.73:0/2996557570 wait complete. 2023-12-01T22:11:42.339 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:42.336+0000 7f982d4ad700 1 Processor -- start 2023-12-01T22:11:42.339 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:42.336+0000 7f982d4ad700 1 -- start start 2023-12-01T22:11:42.339 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:42.336+0000 7f982d4ad700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98280f47c0 0x7f9828106700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:11:42.339 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:42.336+0000 7f982d4ad700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9828106bf0 con 0x7f98280f47c0 2023-12-01T22:11:42.339 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:42.336+0000 7f9826ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98280f47c0 0x7f9828106700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:11:42.339 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:42.336+0000 7f9826ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98280f47c0 0x7f9828106700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:33256/0 (socket says 172.21.15.73:33256) 2023-12-01T22:11:42.339 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:42.336+0000 7f9826ffd700 1 -- 172.21.15.73:0/286462612 learned_addr learned my addr 172.21.15.73:0/286462612 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:11:42.340 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:42.337+0000 7f9826ffd700 1 -- 172.21.15.73:0/286462612 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9818005040 con 0x7f98280f47c0 2023-12-01T22:11:42.340 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:42.337+0000 7f9826ffd700 1 --2- 172.21.15.73:0/286462612 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98280f47c0 0x7f9828106700 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f9818013510 tx=0x7f981800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:11:42.340 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:42.337+0000 7f980f7fe700 1 -- 172.21.15.73:0/286462612 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9818009320 con 0x7f98280f47c0 2023-12-01T22:11:42.340 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:42.337+0000 7f982d4ad700 1 -- 172.21.15.73:0/286462612 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9828106df0 con 0x7f98280f47c0 2023-12-01T22:11:42.341 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:42.337+0000 7f982d4ad700 1 -- 172.21.15.73:0/286462612 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9828107280 con 0x7f98280f47c0 2023-12-01T22:11:42.341 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:42.337+0000 7f980f7fe700 1 -- 172.21.15.73:0/286462612 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9818013f00 con 0x7f98280f47c0 2023-12-01T22:11:42.341 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:42.337+0000 7f980f7fe700 1 -- 172.21.15.73:0/286462612 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9818014d90 con 0x7f98280f47c0 2023-12-01T22:11:42.341 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:42.338+0000 7f982d4ad700 1 -- 172.21.15.73:0/286462612 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9828045ad0 con 0x7f98280f47c0 2023-12-01T22:11:42.341 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:42.338+0000 7f980f7fe700 1 -- 172.21.15.73:0/286462612 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f98180095f0 con 0x7f98280f47c0 2023-12-01T22:11:42.341 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:42.338+0000 7f980f7fe700 1 --2- 172.21.15.73:0/286462612 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f981002ffb0 0x7f9810032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:11:42.341 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:42.338+0000 7f980f7fe700 1 -- 172.21.15.73:0/286462612 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f98180410c0 con 0x7f98280f47c0 2023-12-01T22:11:42.341 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:42.338+0000 7f98267fc700 1 --2- 172.21.15.73:0/286462612 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f981002ffb0 0x7f9810032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:11:42.341 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:42.339+0000 7f98267fc700 1 --2- 172.21.15.73:0/286462612 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f981002ffb0 0x7f9810032470 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f981c0096a0 tx=0x7f981c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:11:42.345 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:42.343+0000 7f980f7fe700 1 -- 172.21.15.73:0/286462612 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9818014420 con 0x7f98280f47c0 2023-12-01T22:11:42.518 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:42.516+0000 7f982d4ad700 1 -- 172.21.15.73:0/286462612 --> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi073", "target": ["mon-mgr", ""]}) v1 -- 0x7f98281083d0 con 0x7f981002ffb0 2023-12-01T22:11:43.113 INFO:teuthology.orchestra.run.smithi073.stdout:Added host 'smithi073' 2023-12-01T22:11:43.113 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:43.110+0000 7f980f7fe700 1 -- 172.21.15.73:0/286462612 <== mgr.14164 v2:172.21.15.29:6800/3698425734 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+23 (secure 0 0 0) 0x7f98281083d0 con 0x7f981002ffb0 2023-12-01T22:11:43.114 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:43.113+0000 7f982d4ad700 1 -- 172.21.15.73:0/286462612 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f981002ffb0 msgr2=0x7f9810032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:11:43.115 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:43.113+0000 7f982d4ad700 1 --2- 172.21.15.73:0/286462612 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f981002ffb0 0x7f9810032470 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f981c0096a0 tx=0x7f981c006b40).stop 2023-12-01T22:11:43.115 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:43.113+0000 7f982d4ad700 1 -- 172.21.15.73:0/286462612 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98280f47c0 msgr2=0x7f9828106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:11:43.115 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:43.113+0000 7f982d4ad700 1 --2- 172.21.15.73:0/286462612 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98280f47c0 0x7f9828106700 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f9818013510 tx=0x7f981800b040).stop 2023-12-01T22:11:43.115 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:43.113+0000 7f982d4ad700 1 -- 172.21.15.73:0/286462612 shutdown_connections 2023-12-01T22:11:43.115 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:43.113+0000 7f982d4ad700 1 --2- 172.21.15.73:0/286462612 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f981002ffb0 0x7f9810032470 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:11:43.115 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:43.113+0000 7f982d4ad700 1 --2- 172.21.15.73:0/286462612 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98280f47c0 0x7f9828106700 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:11:43.115 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:43.113+0000 7f982d4ad700 1 -- 172.21.15.73:0/286462612 >> 172.21.15.73:0/286462612 conn(0x7f98280f02e0 msgr2=0x7f98280f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:11:43.115 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:43.113+0000 7f982d4ad700 1 -- 172.21.15.73:0/286462612 shutdown_connections 2023-12-01T22:11:43.115 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:43.113+0000 7f982d4ad700 1 -- 172.21.15.73:0/286462612 wait complete. 2023-12-01T22:11:43.442 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph orch host ls --format=json 2023-12-01T22:11:44.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:11:43 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:11:42.518843+0000 mgr.smithi029.loxrfu (mgr.14164) 11 : audit [DBG] from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi073", "target": ["mon-mgr", ""]}]: dispatch 2023-12-01T22:11:44.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:11:43 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:11:42.620783+0000 mgr.smithi029.loxrfu (mgr.14164) 12 : cluster [DBG] pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:11:44.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:11:43 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:11:43.111011+0000 mon.smithi029 (mon.0) 117 : audit [INF] from='mgr.14164 172.21.15.29:0/3195927111' entity='mgr.smithi029.loxrfu' 2023-12-01T22:11:44.611 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:44.609+0000 7f5aed235700 1 -- 172.21.15.73:0/3127638891 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ae8060280 msgr2=0x7f5ae80f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:11:44.611 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:44.609+0000 7f5aed235700 1 --2- 172.21.15.73:0/3127638891 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ae8060280 0x7f5ae80f90f0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f5adc009fb0 tx=0x7f5adc009b30).stop 2023-12-01T22:11:44.611 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:44.609+0000 7f5aed235700 1 -- 172.21.15.73:0/3127638891 shutdown_connections 2023-12-01T22:11:44.611 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:44.609+0000 7f5aed235700 1 --2- 172.21.15.73:0/3127638891 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ae8060280 0x7f5ae80f90f0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:11:44.611 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:44.609+0000 7f5aed235700 1 -- 172.21.15.73:0/3127638891 >> 172.21.15.73:0/3127638891 conn(0x7f5ae80f0b70 msgr2=0x7f5ae80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:11:44.611 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:44.609+0000 7f5aed235700 1 -- 172.21.15.73:0/3127638891 shutdown_connections 2023-12-01T22:11:44.612 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:44.609+0000 7f5aed235700 1 -- 172.21.15.73:0/3127638891 wait complete. 2023-12-01T22:11:44.615 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:44.610+0000 7f5aed235700 1 Processor -- start 2023-12-01T22:11:44.615 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:44.610+0000 7f5aed235700 1 -- start start 2023-12-01T22:11:44.615 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:44.610+0000 7f5aed235700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ae8060280 0x7f5ae81066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:11:44.615 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:44.610+0000 7f5aed235700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ae8106bd0 con 0x7f5ae8060280 2023-12-01T22:11:44.615 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:44.610+0000 7f5ae6d9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ae8060280 0x7f5ae81066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:11:44.616 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:44.611+0000 7f5ae6d9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ae8060280 0x7f5ae81066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:55024/0 (socket says 172.21.15.73:55024) 2023-12-01T22:11:44.616 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:44.611+0000 7f5ae6d9d700 1 -- 172.21.15.73:0/3254288517 learned_addr learned my addr 172.21.15.73:0/3254288517 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:11:44.616 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:44.611+0000 7f5ae6d9d700 1 -- 172.21.15.73:0/3254288517 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5adc005040 con 0x7f5ae8060280 2023-12-01T22:11:44.616 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:44.611+0000 7f5ae6d9d700 1 --2- 172.21.15.73:0/3254288517 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ae8060280 0x7f5ae81066e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f5adc013510 tx=0x7f5adc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:11:44.616 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:44.611+0000 7f5acf7fe700 1 -- 172.21.15.73:0/3254288517 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5adc009320 con 0x7f5ae8060280 2023-12-01T22:11:44.616 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:44.611+0000 7f5aed235700 1 -- 172.21.15.73:0/3254288517 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ae8106dd0 con 0x7f5ae8060280 2023-12-01T22:11:44.616 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:44.611+0000 7f5acf7fe700 1 -- 172.21.15.73:0/3254288517 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5adc013f00 con 0x7f5ae8060280 2023-12-01T22:11:44.616 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:44.611+0000 7f5acf7fe700 1 -- 172.21.15.73:0/3254288517 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5adc014dc0 con 0x7f5ae8060280 2023-12-01T22:11:44.616 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:44.611+0000 7f5aed235700 1 -- 172.21.15.73:0/3254288517 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ae8107260 con 0x7f5ae8060280 2023-12-01T22:11:44.616 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:44.612+0000 7f5acf7fe700 1 -- 172.21.15.73:0/3254288517 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5adc0095f0 con 0x7f5ae8060280 2023-12-01T22:11:44.616 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:44.612+0000 7f5aed235700 1 -- 172.21.15.73:0/3254288517 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ae8045ad0 con 0x7f5ae8060280 2023-12-01T22:11:44.617 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:44.612+0000 7f5acf7fe700 1 --2- 172.21.15.73:0/3254288517 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f5ad002ff60 0x7f5ad0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:11:44.617 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:44.612+0000 7f5acf7fe700 1 -- 172.21.15.73:0/3254288517 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5adc0410a0 con 0x7f5ae8060280 2023-12-01T22:11:44.617 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:44.612+0000 7f5ae659c700 1 --2- 172.21.15.73:0/3254288517 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f5ad002ff60 0x7f5ad0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:11:44.617 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:44.613+0000 7f5ae659c700 1 --2- 172.21.15.73:0/3254288517 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f5ad002ff60 0x7f5ad0032420 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f5ad80096a0 tx=0x7f5ad8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:11:44.618 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:44.617+0000 7f5acf7fe700 1 -- 172.21.15.73:0/3254288517 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5adc014420 con 0x7f5ae8060280 2023-12-01T22:11:44.789 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:44.788+0000 7f5aed235700 1 -- 172.21.15.73:0/3254288517 --> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5ae80f1d70 con 0x7f5ad002ff60 2023-12-01T22:11:44.791 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:44.789+0000 7f5acf7fe700 1 -- 172.21.15.73:0/3254288517 <== mgr.14164 v2:172.21.15.29:6800/3698425734 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+156 (secure 0 0 0) 0x7f5ae80f1d70 con 0x7f5ad002ff60 2023-12-01T22:11:44.791 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:11:44.792 INFO:teuthology.orchestra.run.smithi073.stdout:[{"addr": "172.21.15.29", "hostname": "smithi029", "labels": [], "status": ""}, {"addr": "smithi073", "hostname": "smithi073", "labels": [], "status": ""}] 2023-12-01T22:11:44.793 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:44.791+0000 7f5aed235700 1 -- 172.21.15.73:0/3254288517 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f5ad002ff60 msgr2=0x7f5ad0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:11:44.793 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:44.791+0000 7f5aed235700 1 --2- 172.21.15.73:0/3254288517 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f5ad002ff60 0x7f5ad0032420 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f5ad80096a0 tx=0x7f5ad8006b40).stop 2023-12-01T22:11:44.793 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:44.791+0000 7f5aed235700 1 -- 172.21.15.73:0/3254288517 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ae8060280 msgr2=0x7f5ae81066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:11:44.793 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:44.791+0000 7f5aed235700 1 --2- 172.21.15.73:0/3254288517 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ae8060280 0x7f5ae81066e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f5adc013510 tx=0x7f5adc00b040).stop 2023-12-01T22:11:44.793 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:44.791+0000 7f5aed235700 1 -- 172.21.15.73:0/3254288517 shutdown_connections 2023-12-01T22:11:44.793 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:44.791+0000 7f5aed235700 1 --2- 172.21.15.73:0/3254288517 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f5ad002ff60 0x7f5ad0032420 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:11:44.794 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:44.791+0000 7f5aed235700 1 --2- 172.21.15.73:0/3254288517 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ae8060280 0x7f5ae81066e0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:11:44.794 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:44.791+0000 7f5aed235700 1 -- 172.21.15.73:0/3254288517 >> 172.21.15.73:0/3254288517 conn(0x7f5ae80f0b70 msgr2=0x7f5ae80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:11:44.794 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:44.791+0000 7f5aed235700 1 -- 172.21.15.73:0/3254288517 shutdown_connections 2023-12-01T22:11:44.794 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:44.791+0000 7f5aed235700 1 -- 172.21.15.73:0/3254288517 wait complete. 2023-12-01T22:11:45.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:11:44 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cephadm 2023-12-01T22:11:43.111434+0000 mgr.smithi029.loxrfu (mgr.14164) 13 : cephadm [INF] Added host smithi073 2023-12-01T22:11:45.177 INFO:tasks.cephadm:Setting crush tunables to default 2023-12-01T22:11:45.177 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph osd crush tunables default 2023-12-01T22:11:45.915 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:11:45 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:11:44.621208+0000 mgr.smithi029.loxrfu (mgr.14164) 14 : cluster [DBG] pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:11:45.915 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:11:45 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:11:44.790217+0000 mgr.smithi029.loxrfu (mgr.14164) 15 : audit [DBG] from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-12-01T22:11:46.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-12-01T22:11:46.428+0000 7fef4b39e700 1 -- 172.21.15.29:0/1677610225 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fef44060160 msgr2=0x7fef440f8fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:11:46.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-12-01T22:11:46.428+0000 7fef4b39e700 1 --2- 172.21.15.29:0/1677610225 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fef44060160 0x7fef440f8fa0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7fef40009fb0 tx=0x7fef40009b30).stop 2023-12-01T22:11:46.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-12-01T22:11:46.428+0000 7fef4b39e700 1 -- 172.21.15.29:0/1677610225 shutdown_connections 2023-12-01T22:11:46.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-12-01T22:11:46.428+0000 7fef4b39e700 1 --2- 172.21.15.29:0/1677610225 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fef44060160 0x7fef440f8fa0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:11:46.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-12-01T22:11:46.428+0000 7fef4b39e700 1 -- 172.21.15.29:0/1677610225 >> 172.21.15.29:0/1677610225 conn(0x7fef440f09a0 msgr2=0x7fef440f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:11:46.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-12-01T22:11:46.428+0000 7fef4b39e700 1 -- 172.21.15.29:0/1677610225 shutdown_connections 2023-12-01T22:11:46.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-12-01T22:11:46.428+0000 7fef4b39e700 1 -- 172.21.15.29:0/1677610225 wait complete. 2023-12-01T22:11:46.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-12-01T22:11:46.429+0000 7fef4b39e700 1 Processor -- start 2023-12-01T22:11:46.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-12-01T22:11:46.429+0000 7fef4b39e700 1 -- start start 2023-12-01T22:11:46.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-12-01T22:11:46.429+0000 7fef4b39e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fef44060160 0x7fef44106530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:11:46.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-12-01T22:11:46.429+0000 7fef4b39e700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef44106a20 con 0x7fef44060160 2023-12-01T22:11:46.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-12-01T22:11:46.429+0000 7fef4a39c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fef44060160 0x7fef44106530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:11:46.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-12-01T22:11:46.429+0000 7fef4a39c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fef44060160 0x7fef44106530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55726/0 (socket says 172.21.15.29:55726) 2023-12-01T22:11:46.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-12-01T22:11:46.429+0000 7fef4a39c700 1 -- 172.21.15.29:0/2673031647 learned_addr learned my addr 172.21.15.29:0/2673031647 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-12-01T22:11:46.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-12-01T22:11:46.429+0000 7fef4a39c700 1 -- 172.21.15.29:0/2673031647 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef40005040 con 0x7fef44060160 2023-12-01T22:11:46.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-12-01T22:11:46.429+0000 7fef4a39c700 1 --2- 172.21.15.29:0/2673031647 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fef44060160 0x7fef44106530 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7fef40013510 tx=0x7fef4000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:11:46.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-12-01T22:11:46.430+0000 7fef3affd700 1 -- 172.21.15.29:0/2673031647 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fef40009320 con 0x7fef44060160 2023-12-01T22:11:46.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-12-01T22:11:46.430+0000 7fef3affd700 1 -- 172.21.15.29:0/2673031647 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fef40013f00 con 0x7fef44060160 2023-12-01T22:11:46.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-12-01T22:11:46.430+0000 7fef3affd700 1 -- 172.21.15.29:0/2673031647 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fef40014d90 con 0x7fef44060160 2023-12-01T22:11:46.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-12-01T22:11:46.430+0000 7fef4b39e700 1 -- 172.21.15.29:0/2673031647 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef44106c20 con 0x7fef44060160 2023-12-01T22:11:46.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-12-01T22:11:46.430+0000 7fef4b39e700 1 -- 172.21.15.29:0/2673031647 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef441070b0 con 0x7fef44060160 2023-12-01T22:11:46.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-12-01T22:11:46.431+0000 7fef3affd700 1 -- 172.21.15.29:0/2673031647 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fef400095f0 con 0x7fef44060160 2023-12-01T22:11:46.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-12-01T22:11:46.431+0000 7fef4b39e700 1 -- 172.21.15.29:0/2673031647 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef44045ad0 con 0x7fef44060160 2023-12-01T22:11:46.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-12-01T22:11:46.431+0000 7fef3affd700 1 --2- 172.21.15.29:0/2673031647 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fef3002ffb0 0x7fef30032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:11:46.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-12-01T22:11:46.431+0000 7fef3affd700 1 -- 172.21.15.29:0/2673031647 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fef40040ff0 con 0x7fef44060160 2023-12-01T22:11:46.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-12-01T22:11:46.431+0000 7fef49b9b700 1 --2- 172.21.15.29:0/2673031647 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fef3002ffb0 0x7fef30032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:11:46.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-12-01T22:11:46.431+0000 7fef49b9b700 1 --2- 172.21.15.29:0/2673031647 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fef3002ffb0 0x7fef30032470 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7fef340096a0 tx=0x7fef34006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:11:46.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-12-01T22:11:46.435+0000 7fef3affd700 1 -- 172.21.15.29:0/2673031647 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fef4006d2b0 con 0x7fef44060160 2023-12-01T22:11:46.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-12-01T22:11:46.604+0000 7fef4b39e700 1 -- 172.21.15.29:0/2673031647 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fef44107f60 con 0x7fef44060160 2023-12-01T22:11:46.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-12-01T22:11:46.680+0000 7fef3affd700 1 -- 172.21.15.29:0/2673031647 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v5) v1 ==== 124+0+0 (secure 0 0 0) 0x7fef40014420 con 0x7fef44060160 2023-12-01T22:11:46.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-12-01T22:11:46.682+0000 7fef4b39e700 1 -- 172.21.15.29:0/2673031647 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fef3002ffb0 msgr2=0x7fef30032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:11:46.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-12-01T22:11:46.682+0000 7fef4b39e700 1 --2- 172.21.15.29:0/2673031647 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fef3002ffb0 0x7fef30032470 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7fef340096a0 tx=0x7fef34006b40).stop 2023-12-01T22:11:46.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-12-01T22:11:46.682+0000 7fef4b39e700 1 -- 172.21.15.29:0/2673031647 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fef44060160 msgr2=0x7fef44106530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:11:46.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-12-01T22:11:46.682+0000 7fef4b39e700 1 --2- 172.21.15.29:0/2673031647 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fef44060160 0x7fef44106530 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7fef40013510 tx=0x7fef4000b040).stop 2023-12-01T22:11:46.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-12-01T22:11:46.682+0000 7fef4b39e700 1 -- 172.21.15.29:0/2673031647 shutdown_connections 2023-12-01T22:11:46.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-12-01T22:11:46.682+0000 7fef4b39e700 1 --2- 172.21.15.29:0/2673031647 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fef3002ffb0 0x7fef30032470 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:11:46.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-12-01T22:11:46.682+0000 7fef4b39e700 1 --2- 172.21.15.29:0/2673031647 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fef44060160 0x7fef44106530 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:11:46.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-12-01T22:11:46.682+0000 7fef4b39e700 1 -- 172.21.15.29:0/2673031647 >> 172.21.15.29:0/2673031647 conn(0x7fef440f09a0 msgr2=0x7fef440f14a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:11:46.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-12-01T22:11:46.682+0000 7fef4b39e700 1 -- 172.21.15.29:0/2673031647 shutdown_connections 2023-12-01T22:11:46.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-12-01T22:11:46.682+0000 7fef4b39e700 1 -- 172.21.15.29:0/2673031647 wait complete. 2023-12-01T22:11:46.685 INFO:teuthology.orchestra.run.smithi029.stderr:adjusted tunables profile to default 2023-12-01T22:11:46.993 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:11:46 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:11:46.606244+0000 mon.smithi029 (mon.0) 118 : audit [INF] from='client.? 172.21.15.29:0/2673031647' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2023-12-01T22:11:47.068 INFO:tasks.cephadm:Adding mon.smithi029 on smithi029 2023-12-01T22:11:47.068 INFO:tasks.cephadm:Adding mon.smithi073 on smithi073 2023-12-01T22:11:47.069 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph orch apply mon '2;smithi029:172.21.15.29=smithi029;smithi073:172.21.15.73=smithi073' 2023-12-01T22:11:48.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:11:47 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:11:46.621572+0000 mgr.smithi029.loxrfu (mgr.14164) 16 : cluster [DBG] pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:11:48.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:11:47 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:11:46.681610+0000 mon.smithi029 (mon.0) 119 : audit [INF] from='client.? 172.21.15.29:0/2673031647' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2023-12-01T22:11:48.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:11:47 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:11:46.681654+0000 mon.smithi029 (mon.0) 120 : cluster [DBG] osdmap e5: 0 total, 0 up, 0 in 2023-12-01T22:11:48.203 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:48.201+0000 7fa7ff59e700 1 -- 172.21.15.73:0/1215202961 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8000f51a0 msgr2=0x7fa8000f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:11:48.203 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:48.201+0000 7fa7ff59e700 1 --2- 172.21.15.73:0/1215202961 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8000f51a0 0x7fa8000f55c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7fa7f4009fb0 tx=0x7fa7f4009b30).stop 2023-12-01T22:11:48.203 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:48.201+0000 7fa7ff59e700 1 -- 172.21.15.73:0/1215202961 shutdown_connections 2023-12-01T22:11:48.203 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:48.201+0000 7fa7ff59e700 1 --2- 172.21.15.73:0/1215202961 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8000f51a0 0x7fa8000f55c0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:11:48.203 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:48.201+0000 7fa7ff59e700 1 -- 172.21.15.73:0/1215202961 >> 172.21.15.73:0/1215202961 conn(0x7fa8000f09a0 msgr2=0x7fa8000f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:11:48.203 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:48.201+0000 7fa7ff59e700 1 -- 172.21.15.73:0/1215202961 shutdown_connections 2023-12-01T22:11:48.203 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:48.201+0000 7fa7ff59e700 1 -- 172.21.15.73:0/1215202961 wait complete. 2023-12-01T22:11:48.203 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:48.201+0000 7fa7ff59e700 1 Processor -- start 2023-12-01T22:11:48.203 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:48.201+0000 7fa7ff59e700 1 -- start start 2023-12-01T22:11:48.203 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:48.201+0000 7fa7ff59e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8000f51a0 0x7fa80010b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:11:48.204 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:48.201+0000 7fa7ff59e700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa800106fc0 con 0x7fa8000f51a0 2023-12-01T22:11:48.206 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:48.202+0000 7fa7fe59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8000f51a0 0x7fa80010b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:11:48.206 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:48.202+0000 7fa7fe59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8000f51a0 0x7fa80010b5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:55040/0 (socket says 172.21.15.73:55040) 2023-12-01T22:11:48.206 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:48.202+0000 7fa7fe59c700 1 -- 172.21.15.73:0/2079444260 learned_addr learned my addr 172.21.15.73:0/2079444260 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:11:48.206 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:48.202+0000 7fa7fe59c700 1 -- 172.21.15.73:0/2079444260 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7f4005040 con 0x7fa8000f51a0 2023-12-01T22:11:48.206 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:48.202+0000 7fa7fe59c700 1 --2- 172.21.15.73:0/2079444260 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8000f51a0 0x7fa80010b5d0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7fa7f4009830 tx=0x7fa7f400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:11:48.206 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:48.202+0000 7fa7eeffd700 1 -- 172.21.15.73:0/2079444260 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa7f4009320 con 0x7fa8000f51a0 2023-12-01T22:11:48.206 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:48.202+0000 7fa7eeffd700 1 -- 172.21.15.73:0/2079444260 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa7f40136c0 con 0x7fa8000f51a0 2023-12-01T22:11:48.206 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:48.202+0000 7fa7ff59e700 1 -- 172.21.15.73:0/2079444260 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa800060220 con 0x7fa8000f51a0 2023-12-01T22:11:48.206 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:48.202+0000 7fa7eeffd700 1 -- 172.21.15.73:0/2079444260 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa7f401ee90 con 0x7fa8000f51a0 2023-12-01T22:11:48.207 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:48.202+0000 7fa7ff59e700 1 -- 172.21.15.73:0/2079444260 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa80010bac0 con 0x7fa8000f51a0 2023-12-01T22:11:48.207 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:48.203+0000 7fa7eeffd700 1 -- 172.21.15.73:0/2079444260 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa7f401e420 con 0x7fa8000f51a0 2023-12-01T22:11:48.207 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:48.203+0000 7fa7ff59e700 1 -- 172.21.15.73:0/2079444260 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa800045ad0 con 0x7fa8000f51a0 2023-12-01T22:11:48.207 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:48.203+0000 7fa7eeffd700 1 --2- 172.21.15.73:0/2079444260 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fa7e802ff10 0x7fa7e80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:11:48.207 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:48.203+0000 7fa7eeffd700 1 -- 172.21.15.73:0/2079444260 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa7f4040fc0 con 0x7fa8000f51a0 2023-12-01T22:11:48.207 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:48.203+0000 7fa7fdd9b700 1 --2- 172.21.15.73:0/2079444260 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fa7e802ff10 0x7fa7e80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:11:48.207 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:48.204+0000 7fa7fdd9b700 1 --2- 172.21.15.73:0/2079444260 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fa7e802ff10 0x7fa7e80323d0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7fa7f00096a0 tx=0x7fa7f0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:11:48.209 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:48.208+0000 7fa7eeffd700 1 -- 172.21.15.73:0/2079444260 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa7f406d2b0 con 0x7fa8000f51a0 2023-12-01T22:11:48.386 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:48.385+0000 7fa7ff59e700 1 -- 172.21.15.73:0/2079444260 --> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi029:172.21.15.29=smithi029;smithi073:172.21.15.73=smithi073", "target": ["mon-mgr", ""]}) v1 -- 0x7fa8000f1c20 con 0x7fa7e802ff10 2023-12-01T22:11:48.392 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:48.390+0000 7fa7eeffd700 1 -- 172.21.15.73:0/2079444260 <== mgr.14164 v2:172.21.15.29:6800/3698425734 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fa8000f1c20 con 0x7fa7e802ff10 2023-12-01T22:11:48.392 INFO:teuthology.orchestra.run.smithi073.stdout:Scheduled mon update... 2023-12-01T22:11:48.394 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:48.392+0000 7fa7ff59e700 1 -- 172.21.15.73:0/2079444260 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fa7e802ff10 msgr2=0x7fa7e80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:11:48.394 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:48.392+0000 7fa7ff59e700 1 --2- 172.21.15.73:0/2079444260 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fa7e802ff10 0x7fa7e80323d0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7fa7f00096a0 tx=0x7fa7f0006b40).stop 2023-12-01T22:11:48.394 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:48.392+0000 7fa7ff59e700 1 -- 172.21.15.73:0/2079444260 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8000f51a0 msgr2=0x7fa80010b5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:11:48.394 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:48.392+0000 7fa7ff59e700 1 --2- 172.21.15.73:0/2079444260 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8000f51a0 0x7fa80010b5d0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7fa7f4009830 tx=0x7fa7f400b040).stop 2023-12-01T22:11:48.394 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:48.392+0000 7fa7ff59e700 1 -- 172.21.15.73:0/2079444260 shutdown_connections 2023-12-01T22:11:48.394 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:48.392+0000 7fa7ff59e700 1 --2- 172.21.15.73:0/2079444260 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fa7e802ff10 0x7fa7e80323d0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:11:48.394 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:48.392+0000 7fa7ff59e700 1 --2- 172.21.15.73:0/2079444260 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8000f51a0 0x7fa80010b5d0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:11:48.394 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:48.392+0000 7fa7ff59e700 1 -- 172.21.15.73:0/2079444260 >> 172.21.15.73:0/2079444260 conn(0x7fa8000f09a0 msgr2=0x7fa8000f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:11:48.395 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:48.392+0000 7fa7ff59e700 1 -- 172.21.15.73:0/2079444260 shutdown_connections 2023-12-01T22:11:48.395 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:48.392+0000 7fa7ff59e700 1 -- 172.21.15.73:0/2079444260 wait complete. 2023-12-01T22:11:48.777 DEBUG:teuthology.orchestra.run.smithi073:mon.smithi073> sudo journalctl -f -n 0 -u ceph-5558ab6c-9096-11ee-95a2-87774f69a715@mon.smithi073.service 2023-12-01T22:11:48.779 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:11:48.780 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:11:48.807 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:-- Logs begin at Fri 2023-12-01 21:56:25 UTC. -- 2023-12-01T22:11:49.776 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:11:49 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:11:48.387072+0000 mgr.smithi029.loxrfu (mgr.14164) 17 : audit [DBG] from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi029:172.21.15.29=smithi029;smithi073:172.21.15.73=smithi073", "target": ["mon-mgr", ""]}]: dispatch 2023-12-01T22:11:49.777 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:11:49 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cephadm 2023-12-01T22:11:48.388559+0000 mgr.smithi029.loxrfu (mgr.14164) 18 : cephadm [INF] Saving service mon spec with placement smithi029:172.21.15.29=smithi029;smithi073:172.21.15.73=smithi073;count:2 2023-12-01T22:11:49.777 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:11:49 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:11:48.391405+0000 mon.smithi029 (mon.0) 121 : audit [INF] from='mgr.14164 172.21.15.29:0/3195927111' entity='mgr.smithi029.loxrfu' 2023-12-01T22:11:49.777 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:11:49 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:11:48.621895+0000 mgr.smithi029.loxrfu (mgr.14164) 19 : cluster [DBG] pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:11:49.954 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:49.951+0000 7f6d448ed700 1 -- 172.21.15.73:0/722308613 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6d40060280 msgr2=0x7f6d400f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:11:49.955 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:49.951+0000 7f6d448ed700 1 --2- 172.21.15.73:0/722308613 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6d40060280 0x7f6d400f90f0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f6d34009fb0 tx=0x7f6d34009b30).stop 2023-12-01T22:11:49.955 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:49.951+0000 7f6d448ed700 1 -- 172.21.15.73:0/722308613 shutdown_connections 2023-12-01T22:11:49.955 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:49.951+0000 7f6d448ed700 1 --2- 172.21.15.73:0/722308613 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6d40060280 0x7f6d400f90f0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:11:49.955 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:49.951+0000 7f6d448ed700 1 -- 172.21.15.73:0/722308613 >> 172.21.15.73:0/722308613 conn(0x7f6d400f0b70 msgr2=0x7f6d400f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:11:49.955 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:49.951+0000 7f6d448ed700 1 -- 172.21.15.73:0/722308613 shutdown_connections 2023-12-01T22:11:49.955 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:49.951+0000 7f6d448ed700 1 -- 172.21.15.73:0/722308613 wait complete. 2023-12-01T22:11:49.955 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:49.952+0000 7f6d448ed700 1 Processor -- start 2023-12-01T22:11:49.955 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:49.952+0000 7f6d448ed700 1 -- start start 2023-12-01T22:11:49.955 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:49.952+0000 7f6d448ed700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6d40060280 0x7f6d401066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:11:49.955 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:49.952+0000 7f6d448ed700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d40106bd0 con 0x7f6d40060280 2023-12-01T22:11:49.955 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:49.952+0000 7f6d3e59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6d40060280 0x7f6d401066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:11:49.956 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:49.952+0000 7f6d3e59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6d40060280 0x7f6d401066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:55066/0 (socket says 172.21.15.73:55066) 2023-12-01T22:11:49.956 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:49.952+0000 7f6d3e59c700 1 -- 172.21.15.73:0/237129952 learned_addr learned my addr 172.21.15.73:0/237129952 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:11:49.956 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:49.952+0000 7f6d3e59c700 1 -- 172.21.15.73:0/237129952 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d34005040 con 0x7f6d40060280 2023-12-01T22:11:49.956 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:49.952+0000 7f6d3e59c700 1 --2- 172.21.15.73:0/237129952 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6d40060280 0x7f6d401066e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f6d34013510 tx=0x7f6d3400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:11:49.956 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:49.953+0000 7f6d2effd700 1 -- 172.21.15.73:0/237129952 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6d34009320 con 0x7f6d40060280 2023-12-01T22:11:49.956 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:49.953+0000 7f6d448ed700 1 -- 172.21.15.73:0/237129952 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d40106dd0 con 0x7f6d40060280 2023-12-01T22:11:49.956 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:49.953+0000 7f6d2effd700 1 -- 172.21.15.73:0/237129952 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6d34013f00 con 0x7f6d40060280 2023-12-01T22:11:49.956 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:49.953+0000 7f6d2effd700 1 -- 172.21.15.73:0/237129952 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6d34014dc0 con 0x7f6d40060280 2023-12-01T22:11:49.956 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:49.953+0000 7f6d448ed700 1 -- 172.21.15.73:0/237129952 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d40107260 con 0x7f6d40060280 2023-12-01T22:11:49.957 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:49.953+0000 7f6d2effd700 1 -- 172.21.15.73:0/237129952 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6d340095f0 con 0x7f6d40060280 2023-12-01T22:11:49.957 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:49.953+0000 7f6d448ed700 1 -- 172.21.15.73:0/237129952 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d40045ad0 con 0x7f6d40060280 2023-12-01T22:11:49.957 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:49.953+0000 7f6d2effd700 1 --2- 172.21.15.73:0/237129952 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f6d2802ff60 0x7f6d28032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:11:49.957 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:49.954+0000 7f6d2effd700 1 -- 172.21.15.73:0/237129952 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6d34041100 con 0x7f6d40060280 2023-12-01T22:11:49.957 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:49.954+0000 7f6d3dd9b700 1 --2- 172.21.15.73:0/237129952 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f6d2802ff60 0x7f6d28032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:11:49.957 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:49.954+0000 7f6d3dd9b700 1 --2- 172.21.15.73:0/237129952 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f6d2802ff60 0x7f6d28032420 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f6d300096a0 tx=0x7f6d30006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:11:49.960 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:49.959+0000 7f6d2effd700 1 -- 172.21.15.73:0/237129952 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6d34014420 con 0x7f6d40060280 2023-12-01T22:11:50.165 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:50.163+0000 7f6d448ed700 1 -- 172.21.15.73:0/237129952 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6d40108110 con 0x7f6d40060280 2023-12-01T22:11:50.166 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:50.164+0000 7f6d2effd700 1 -- 172.21.15.73:0/237129952 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6d34010230 con 0x7f6d40060280 2023-12-01T22:11:50.166 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:11:50.167 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:11:50.167 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:50.165+0000 7f6d448ed700 1 -- 172.21.15.73:0/237129952 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f6d2802ff60 msgr2=0x7f6d28032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:11:50.167 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:50.165+0000 7f6d448ed700 1 --2- 172.21.15.73:0/237129952 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f6d2802ff60 0x7f6d28032420 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f6d300096a0 tx=0x7f6d30006b40).stop 2023-12-01T22:11:50.167 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:50.165+0000 7f6d448ed700 1 -- 172.21.15.73:0/237129952 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6d40060280 msgr2=0x7f6d401066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:11:50.167 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:50.165+0000 7f6d448ed700 1 --2- 172.21.15.73:0/237129952 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6d40060280 0x7f6d401066e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f6d34013510 tx=0x7f6d3400b040).stop 2023-12-01T22:11:50.167 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:50.165+0000 7f6d448ed700 1 -- 172.21.15.73:0/237129952 shutdown_connections 2023-12-01T22:11:50.168 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:50.165+0000 7f6d448ed700 1 --2- 172.21.15.73:0/237129952 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f6d2802ff60 0x7f6d28032420 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:11:50.168 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:50.165+0000 7f6d448ed700 1 --2- 172.21.15.73:0/237129952 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6d40060280 0x7f6d401066e0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:11:50.168 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:50.165+0000 7f6d448ed700 1 -- 172.21.15.73:0/237129952 >> 172.21.15.73:0/237129952 conn(0x7f6d400f0b70 msgr2=0x7f6d400f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:11:50.168 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:50.166+0000 7f6d448ed700 1 -- 172.21.15.73:0/237129952 shutdown_connections 2023-12-01T22:11:50.168 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:50.166+0000 7f6d448ed700 1 -- 172.21.15.73:0/237129952 wait complete. 2023-12-01T22:11:50.168 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:11:50.776 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:11:50 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:11:50.165045+0000 mon.smithi029 (mon.0) 122 : audit [DBG] from='client.? 172.21.15.73:0/237129952' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:11:51.545 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:11:51.545 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:11:51.776 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:11:51 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:11:50.622189+0000 mgr.smithi029.loxrfu (mgr.14164) 20 : cluster [DBG] pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:11:52.746 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:52.743+0000 7f565cbad700 1 -- 172.21.15.73:0/2085514519 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56580f5410 msgr2=0x7f56580f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:11:52.747 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:52.743+0000 7f565cbad700 1 --2- 172.21.15.73:0/2085514519 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56580f5410 0x7f56580f5830 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f5648009fb0 tx=0x7f5648009b30).stop 2023-12-01T22:11:52.747 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:52.744+0000 7f565cbad700 1 -- 172.21.15.73:0/2085514519 shutdown_connections 2023-12-01T22:11:52.747 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:52.744+0000 7f565cbad700 1 --2- 172.21.15.73:0/2085514519 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56580f5410 0x7f56580f5830 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:11:52.747 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:52.744+0000 7f565cbad700 1 -- 172.21.15.73:0/2085514519 >> 172.21.15.73:0/2085514519 conn(0x7f56580f0b70 msgr2=0x7f56580f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:11:52.747 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:52.744+0000 7f565cbad700 1 -- 172.21.15.73:0/2085514519 shutdown_connections 2023-12-01T22:11:52.747 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:52.744+0000 7f565cbad700 1 -- 172.21.15.73:0/2085514519 wait complete. 2023-12-01T22:11:52.747 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:52.744+0000 7f565cbad700 1 Processor -- start 2023-12-01T22:11:52.747 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:52.744+0000 7f565cbad700 1 -- start start 2023-12-01T22:11:52.747 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:52.744+0000 7f565cbad700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56580f5410 0x7f5658106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:11:52.747 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:52.744+0000 7f565cbad700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5658106c20 con 0x7f56580f5410 2023-12-01T22:11:52.747 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:52.745+0000 7f565659c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56580f5410 0x7f5658106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:11:52.747 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:52.745+0000 7f565659c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56580f5410 0x7f5658106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:55082/0 (socket says 172.21.15.73:55082) 2023-12-01T22:11:52.748 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:52.745+0000 7f565659c700 1 -- 172.21.15.73:0/4035315774 learned_addr learned my addr 172.21.15.73:0/4035315774 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:11:52.748 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:52.745+0000 7f565659c700 1 -- 172.21.15.73:0/4035315774 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5648005040 con 0x7f56580f5410 2023-12-01T22:11:52.748 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:52.745+0000 7f565659c700 1 --2- 172.21.15.73:0/4035315774 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56580f5410 0x7f5658106730 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f5648013510 tx=0x7f564800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:11:52.749 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:52.745+0000 7f5646ffd700 1 -- 172.21.15.73:0/4035315774 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5648009320 con 0x7f56580f5410 2023-12-01T22:11:52.749 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:52.745+0000 7f565cbad700 1 -- 172.21.15.73:0/4035315774 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5658106e20 con 0x7f56580f5410 2023-12-01T22:11:52.749 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:52.745+0000 7f5646ffd700 1 -- 172.21.15.73:0/4035315774 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5648013f00 con 0x7f56580f5410 2023-12-01T22:11:52.749 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:52.745+0000 7f5646ffd700 1 -- 172.21.15.73:0/4035315774 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5648014d90 con 0x7f56580f5410 2023-12-01T22:11:52.749 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:52.745+0000 7f565cbad700 1 -- 172.21.15.73:0/4035315774 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56581072b0 con 0x7f56580f5410 2023-12-01T22:11:52.749 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:52.746+0000 7f565cbad700 1 -- 172.21.15.73:0/4035315774 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5658045ad0 con 0x7f56580f5410 2023-12-01T22:11:52.749 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:52.746+0000 7f5646ffd700 1 -- 172.21.15.73:0/4035315774 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f56480263a0 con 0x7f56580f5410 2023-12-01T22:11:52.749 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:52.747+0000 7f5646ffd700 1 --2- 172.21.15.73:0/4035315774 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f564002ffb0 0x7f5640032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:11:52.749 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:52.747+0000 7f5646ffd700 1 -- 172.21.15.73:0/4035315774 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5648040fa0 con 0x7f56580f5410 2023-12-01T22:11:52.749 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:52.747+0000 7f5655d9b700 1 --2- 172.21.15.73:0/4035315774 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f564002ffb0 0x7f5640032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:11:52.750 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:52.747+0000 7f5655d9b700 1 --2- 172.21.15.73:0/4035315774 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f564002ffb0 0x7f5640032470 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f564c0096a0 tx=0x7f564c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:11:52.753 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:52.751+0000 7f5646ffd700 1 -- 172.21.15.73:0/4035315774 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f564806d2b0 con 0x7f56580f5410 2023-12-01T22:11:52.960 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:52.958+0000 7f565cbad700 1 -- 172.21.15.73:0/4035315774 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f56580fbf90 con 0x7f56580f5410 2023-12-01T22:11:52.960 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:52.959+0000 7f5646ffd700 1 -- 172.21.15.73:0/4035315774 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f56480126f0 con 0x7f56580f5410 2023-12-01T22:11:52.960 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:11:52.961 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:11:52.962 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:52.960+0000 7f565cbad700 1 -- 172.21.15.73:0/4035315774 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f564002ffb0 msgr2=0x7f5640032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:11:52.962 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:52.960+0000 7f565cbad700 1 --2- 172.21.15.73:0/4035315774 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f564002ffb0 0x7f5640032470 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f564c0096a0 tx=0x7f564c006b40).stop 2023-12-01T22:11:52.962 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:52.960+0000 7f565cbad700 1 -- 172.21.15.73:0/4035315774 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56580f5410 msgr2=0x7f5658106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:11:52.963 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:52.960+0000 7f565cbad700 1 --2- 172.21.15.73:0/4035315774 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56580f5410 0x7f5658106730 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f5648013510 tx=0x7f564800b040).stop 2023-12-01T22:11:52.963 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:52.960+0000 7f565cbad700 1 -- 172.21.15.73:0/4035315774 shutdown_connections 2023-12-01T22:11:52.963 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:52.960+0000 7f565cbad700 1 --2- 172.21.15.73:0/4035315774 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f564002ffb0 0x7f5640032470 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:11:52.963 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:52.960+0000 7f565cbad700 1 --2- 172.21.15.73:0/4035315774 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56580f5410 0x7f5658106730 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:11:52.963 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:52.960+0000 7f565cbad700 1 -- 172.21.15.73:0/4035315774 >> 172.21.15.73:0/4035315774 conn(0x7f56580f0b70 msgr2=0x7f56580f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:11:52.963 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:52.960+0000 7f565cbad700 1 -- 172.21.15.73:0/4035315774 shutdown_connections 2023-12-01T22:11:52.963 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:52.960+0000 7f565cbad700 1 -- 172.21.15.73:0/4035315774 wait complete. 2023-12-01T22:11:52.963 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:11:54.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:11:53 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:11:52.622538+0000 mgr.smithi029.loxrfu (mgr.14164) 21 : cluster [DBG] pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:11:54.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:11:53 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:11:52.960013+0000 mon.smithi029 (mon.0) 123 : audit [DBG] from='client.? 172.21.15.73:0/4035315774' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:11:54.350 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:11:54.350 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:11:55.571 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:55.568+0000 7f93a5efa700 1 -- 172.21.15.73:0/124223202 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93a00f51a0 msgr2=0x7f93a00f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:11:55.571 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:55.568+0000 7f93a5efa700 1 --2- 172.21.15.73:0/124223202 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93a00f51a0 0x7f93a00f55c0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f9394009fb0 tx=0x7f9394009b30).stop 2023-12-01T22:11:55.571 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:55.568+0000 7f93a5efa700 1 -- 172.21.15.73:0/124223202 shutdown_connections 2023-12-01T22:11:55.571 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:55.568+0000 7f93a5efa700 1 --2- 172.21.15.73:0/124223202 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93a00f51a0 0x7f93a00f55c0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:11:55.571 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:55.568+0000 7f93a5efa700 1 -- 172.21.15.73:0/124223202 >> 172.21.15.73:0/124223202 conn(0x7f93a00f09a0 msgr2=0x7f93a00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:11:55.572 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:55.568+0000 7f93a5efa700 1 -- 172.21.15.73:0/124223202 shutdown_connections 2023-12-01T22:11:55.572 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:55.568+0000 7f93a5efa700 1 -- 172.21.15.73:0/124223202 wait complete. 2023-12-01T22:11:55.572 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:55.568+0000 7f93a5efa700 1 Processor -- start 2023-12-01T22:11:55.572 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:55.568+0000 7f93a5efa700 1 -- start start 2023-12-01T22:11:55.572 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:55.569+0000 7f93a5efa700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93a00f51a0 0x7f93a01064e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:11:55.572 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:55.569+0000 7f93a5efa700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93a01069d0 con 0x7f93a00f51a0 2023-12-01T22:11:55.572 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:55.569+0000 7f93a4ef8700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93a00f51a0 0x7f93a01064e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:11:55.572 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:55.569+0000 7f93a4ef8700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93a00f51a0 0x7f93a01064e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:52210/0 (socket says 172.21.15.73:52210) 2023-12-01T22:11:55.572 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:55.569+0000 7f93a4ef8700 1 -- 172.21.15.73:0/1655410968 learned_addr learned my addr 172.21.15.73:0/1655410968 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:11:55.572 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:55.569+0000 7f93a4ef8700 1 -- 172.21.15.73:0/1655410968 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9394005040 con 0x7f93a00f51a0 2023-12-01T22:11:55.572 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:55.569+0000 7f93a4ef8700 1 --2- 172.21.15.73:0/1655410968 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93a00f51a0 0x7f93a01064e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f9394013510 tx=0x7f939400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:11:55.572 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:55.569+0000 7f939d7fa700 1 -- 172.21.15.73:0/1655410968 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9394009320 con 0x7f93a00f51a0 2023-12-01T22:11:55.572 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:55.569+0000 7f93a5efa700 1 -- 172.21.15.73:0/1655410968 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93a0106bd0 con 0x7f93a00f51a0 2023-12-01T22:11:55.572 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:55.569+0000 7f939d7fa700 1 -- 172.21.15.73:0/1655410968 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9394013f00 con 0x7f93a00f51a0 2023-12-01T22:11:55.573 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:55.569+0000 7f939d7fa700 1 -- 172.21.15.73:0/1655410968 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9394014dc0 con 0x7f93a00f51a0 2023-12-01T22:11:55.573 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:55.569+0000 7f93a5efa700 1 -- 172.21.15.73:0/1655410968 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93a0107060 con 0x7f93a00f51a0 2023-12-01T22:11:55.573 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:55.570+0000 7f939d7fa700 1 -- 172.21.15.73:0/1655410968 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f93940095f0 con 0x7f93a00f51a0 2023-12-01T22:11:55.574 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:55.570+0000 7f93a5efa700 1 -- 172.21.15.73:0/1655410968 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93a0045ad0 con 0x7f93a00f51a0 2023-12-01T22:11:55.574 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:55.570+0000 7f939d7fa700 1 --2- 172.21.15.73:0/1655410968 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f939002ff60 0x7f9390032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:11:55.574 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:55.570+0000 7f939d7fa700 1 -- 172.21.15.73:0/1655410968 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9394041000 con 0x7f93a00f51a0 2023-12-01T22:11:55.574 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:55.571+0000 7f939ffff700 1 --2- 172.21.15.73:0/1655410968 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f939002ff60 0x7f9390032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:11:55.574 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:55.571+0000 7f939ffff700 1 --2- 172.21.15.73:0/1655410968 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f939002ff60 0x7f9390032420 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f938c0096a0 tx=0x7f938c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:11:55.577 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:55.575+0000 7f939d7fa700 1 -- 172.21.15.73:0/1655410968 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f939401c790 con 0x7f93a00f51a0 2023-12-01T22:11:55.782 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:55.780+0000 7f93a5efa700 1 -- 172.21.15.73:0/1655410968 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f93a0107f10 con 0x7f93a00f51a0 2023-12-01T22:11:55.782 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:55.781+0000 7f939d7fa700 1 -- 172.21.15.73:0/1655410968 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9394010230 con 0x7f93a00f51a0 2023-12-01T22:11:55.783 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:11:55.783 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:11:55.784 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:55.782+0000 7f93a5efa700 1 -- 172.21.15.73:0/1655410968 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f939002ff60 msgr2=0x7f9390032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:11:55.784 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:55.782+0000 7f93a5efa700 1 --2- 172.21.15.73:0/1655410968 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f939002ff60 0x7f9390032420 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f938c0096a0 tx=0x7f938c006b40).stop 2023-12-01T22:11:55.784 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:55.782+0000 7f93a5efa700 1 -- 172.21.15.73:0/1655410968 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93a00f51a0 msgr2=0x7f93a01064e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:11:55.784 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:55.782+0000 7f93a5efa700 1 --2- 172.21.15.73:0/1655410968 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93a00f51a0 0x7f93a01064e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f9394013510 tx=0x7f939400b040).stop 2023-12-01T22:11:55.784 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:55.782+0000 7f93a5efa700 1 -- 172.21.15.73:0/1655410968 shutdown_connections 2023-12-01T22:11:55.785 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:55.782+0000 7f93a5efa700 1 --2- 172.21.15.73:0/1655410968 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f939002ff60 0x7f9390032420 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:11:55.785 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:55.782+0000 7f93a5efa700 1 --2- 172.21.15.73:0/1655410968 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93a00f51a0 0x7f93a01064e0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:11:55.785 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:55.782+0000 7f93a5efa700 1 -- 172.21.15.73:0/1655410968 >> 172.21.15.73:0/1655410968 conn(0x7f93a00f09a0 msgr2=0x7f93a005fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:11:55.785 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:55.782+0000 7f93a5efa700 1 -- 172.21.15.73:0/1655410968 shutdown_connections 2023-12-01T22:11:55.785 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:55.782+0000 7f93a5efa700 1 -- 172.21.15.73:0/1655410968 wait complete. 2023-12-01T22:11:55.785 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:11:56.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:11:55 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:11:54.622883+0000 mgr.smithi029.loxrfu (mgr.14164) 22 : cluster [DBG] pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:11:57.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:11:56 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:11:55.782199+0000 mon.smithi029 (mon.0) 124 : audit [DBG] from='client.? 172.21.15.73:0/1655410968' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:11:57.171 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:11:57.171 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:11:58.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:11:57 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:11:56.623216+0000 mgr.smithi029.loxrfu (mgr.14164) 23 : cluster [DBG] pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:11:58.370 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:58.368+0000 7fae1f414700 1 -- 172.21.15.73:0/431863415 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae180f51a0 msgr2=0x7fae180f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:11:58.370 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:58.368+0000 7fae1f414700 1 --2- 172.21.15.73:0/431863415 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae180f51a0 0x7fae180f55c0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7fae14009fb0 tx=0x7fae14009b30).stop 2023-12-01T22:11:58.370 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:58.368+0000 7fae1f414700 1 -- 172.21.15.73:0/431863415 shutdown_connections 2023-12-01T22:11:58.371 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:58.368+0000 7fae1f414700 1 --2- 172.21.15.73:0/431863415 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae180f51a0 0x7fae180f55c0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:11:58.371 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:58.368+0000 7fae1f414700 1 -- 172.21.15.73:0/431863415 >> 172.21.15.73:0/431863415 conn(0x7fae180f09a0 msgr2=0x7fae180f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:11:58.371 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:58.368+0000 7fae1f414700 1 -- 172.21.15.73:0/431863415 shutdown_connections 2023-12-01T22:11:58.371 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:58.368+0000 7fae1f414700 1 -- 172.21.15.73:0/431863415 wait complete. 2023-12-01T22:11:58.371 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:58.369+0000 7fae1f414700 1 Processor -- start 2023-12-01T22:11:58.371 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:58.369+0000 7fae1f414700 1 -- start start 2023-12-01T22:11:58.371 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:58.369+0000 7fae1f414700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae180f51a0 0x7fae181053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:11:58.371 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:58.369+0000 7fae1f414700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae181058a0 con 0x7fae180f51a0 2023-12-01T22:11:58.372 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:58.369+0000 7fae1e412700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae180f51a0 0x7fae181053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:11:58.373 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:58.369+0000 7fae1e412700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae180f51a0 0x7fae181053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:52234/0 (socket says 172.21.15.73:52234) 2023-12-01T22:11:58.373 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:58.369+0000 7fae1e412700 1 -- 172.21.15.73:0/2795358888 learned_addr learned my addr 172.21.15.73:0/2795358888 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:11:58.373 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:58.369+0000 7fae1e412700 1 -- 172.21.15.73:0/2795358888 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae14005040 con 0x7fae180f51a0 2023-12-01T22:11:58.373 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:58.369+0000 7fae1e412700 1 --2- 172.21.15.73:0/2795358888 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae180f51a0 0x7fae181053b0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7fae14013510 tx=0x7fae1400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:11:58.373 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:58.370+0000 7fae0effd700 1 -- 172.21.15.73:0/2795358888 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fae14009320 con 0x7fae180f51a0 2023-12-01T22:11:58.373 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:58.370+0000 7fae1f414700 1 -- 172.21.15.73:0/2795358888 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae18105aa0 con 0x7fae180f51a0 2023-12-01T22:11:58.373 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:58.370+0000 7fae1f414700 1 -- 172.21.15.73:0/2795358888 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae18105f30 con 0x7fae180f51a0 2023-12-01T22:11:58.373 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:58.370+0000 7fae0effd700 1 -- 172.21.15.73:0/2795358888 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fae14013f00 con 0x7fae180f51a0 2023-12-01T22:11:58.373 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:58.370+0000 7fae0effd700 1 -- 172.21.15.73:0/2795358888 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fae14014d90 con 0x7fae180f51a0 2023-12-01T22:11:58.373 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:58.371+0000 7fae1f414700 1 -- 172.21.15.73:0/2795358888 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae00004fa0 con 0x7fae180f51a0 2023-12-01T22:11:58.373 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:58.371+0000 7fae0effd700 1 -- 172.21.15.73:0/2795358888 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fae140263a0 con 0x7fae180f51a0 2023-12-01T22:11:58.373 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:58.371+0000 7fae0effd700 1 --2- 172.21.15.73:0/2795358888 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fae04032170 0x7fae04034630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:11:58.373 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:58.371+0000 7fae0effd700 1 -- 172.21.15.73:0/2795358888 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fae14040fe0 con 0x7fae180f51a0 2023-12-01T22:11:58.374 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:58.371+0000 7fae1dc11700 1 --2- 172.21.15.73:0/2795358888 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fae04032170 0x7fae04034630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:11:58.374 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:58.371+0000 7fae1dc11700 1 --2- 172.21.15.73:0/2795358888 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fae04032170 0x7fae04034630 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7fae080096a0 tx=0x7fae08006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:11:58.377 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:58.375+0000 7fae0effd700 1 -- 172.21.15.73:0/2795358888 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fae14012920 con 0x7fae180f51a0 2023-12-01T22:11:58.585 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:58.583+0000 7fae1f414700 1 -- 172.21.15.73:0/2795358888 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fae00005670 con 0x7fae180f51a0 2023-12-01T22:11:58.586 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:58.584+0000 7fae0effd700 1 -- 172.21.15.73:0/2795358888 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fae14012330 con 0x7fae180f51a0 2023-12-01T22:11:58.586 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:11:58.586 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:11:58.587 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:58.586+0000 7fae1f414700 1 -- 172.21.15.73:0/2795358888 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fae04032170 msgr2=0x7fae04034630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:11:58.588 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:58.586+0000 7fae1f414700 1 --2- 172.21.15.73:0/2795358888 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fae04032170 0x7fae04034630 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7fae080096a0 tx=0x7fae08006b40).stop 2023-12-01T22:11:58.588 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:58.586+0000 7fae1f414700 1 -- 172.21.15.73:0/2795358888 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae180f51a0 msgr2=0x7fae181053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:11:58.588 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:58.586+0000 7fae1f414700 1 --2- 172.21.15.73:0/2795358888 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae180f51a0 0x7fae181053b0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7fae14013510 tx=0x7fae1400b040).stop 2023-12-01T22:11:58.588 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:58.586+0000 7fae1f414700 1 -- 172.21.15.73:0/2795358888 shutdown_connections 2023-12-01T22:11:58.588 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:58.586+0000 7fae1f414700 1 --2- 172.21.15.73:0/2795358888 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fae04032170 0x7fae04034630 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:11:58.588 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:58.586+0000 7fae1f414700 1 --2- 172.21.15.73:0/2795358888 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae180f51a0 0x7fae181053b0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:11:58.588 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:58.586+0000 7fae1f414700 1 -- 172.21.15.73:0/2795358888 >> 172.21.15.73:0/2795358888 conn(0x7fae180f09a0 msgr2=0x7fae180f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:11:58.588 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:58.586+0000 7fae1f414700 1 -- 172.21.15.73:0/2795358888 shutdown_connections 2023-12-01T22:11:58.588 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:11:58.586+0000 7fae1f414700 1 -- 172.21.15.73:0/2795358888 wait complete. 2023-12-01T22:11:58.588 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:11:59.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:11:58 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:11:58.585606+0000 mon.smithi029 (mon.0) 125 : audit [DBG] from='client.? 172.21.15.73:0/2795358888' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:11:59.985 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:11:59.986 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:12:00.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:11:59 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:11:58.623492+0000 mgr.smithi029.loxrfu (mgr.14164) 24 : cluster [DBG] pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:12:01.143 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:01.141+0000 7fd03ac81700 1 -- 172.21.15.73:0/2801845703 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0340f6510 msgr2=0x7fd0340f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:12:01.143 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:01.141+0000 7fd03ac81700 1 --2- 172.21.15.73:0/2801845703 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0340f6510 0x7fd0340f6930 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7fd028009fb0 tx=0x7fd028009b30).stop 2023-12-01T22:12:01.143 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:01.141+0000 7fd03ac81700 1 -- 172.21.15.73:0/2801845703 shutdown_connections 2023-12-01T22:12:01.143 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:01.141+0000 7fd03ac81700 1 --2- 172.21.15.73:0/2801845703 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0340f6510 0x7fd0340f6930 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:12:01.143 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:01.141+0000 7fd03ac81700 1 -- 172.21.15.73:0/2801845703 >> 172.21.15.73:0/2801845703 conn(0x7fd0340f1c70 msgr2=0x7fd0340f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:12:01.143 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:01.141+0000 7fd03ac81700 1 -- 172.21.15.73:0/2801845703 shutdown_connections 2023-12-01T22:12:01.143 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:01.141+0000 7fd03ac81700 1 -- 172.21.15.73:0/2801845703 wait complete. 2023-12-01T22:12:01.145 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:01.142+0000 7fd03ac81700 1 Processor -- start 2023-12-01T22:12:01.145 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:01.142+0000 7fd03ac81700 1 -- start start 2023-12-01T22:12:01.145 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:01.142+0000 7fd03ac81700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0340f6510 0x7fd034106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:12:01.145 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:01.142+0000 7fd03ac81700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd034106c20 con 0x7fd0340f6510 2023-12-01T22:12:01.145 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:01.142+0000 7fd038a1d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0340f6510 0x7fd034106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:12:01.145 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:01.142+0000 7fd038a1d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0340f6510 0x7fd034106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:52264/0 (socket says 172.21.15.73:52264) 2023-12-01T22:12:01.145 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:01.142+0000 7fd038a1d700 1 -- 172.21.15.73:0/1813390678 learned_addr learned my addr 172.21.15.73:0/1813390678 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:12:01.145 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:01.143+0000 7fd038a1d700 1 -- 172.21.15.73:0/1813390678 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd028005040 con 0x7fd0340f6510 2023-12-01T22:12:01.145 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:01.143+0000 7fd038a1d700 1 --2- 172.21.15.73:0/1813390678 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0340f6510 0x7fd034106730 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7fd028013510 tx=0x7fd02800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:12:01.145 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:01.143+0000 7fd0317fa700 1 -- 172.21.15.73:0/1813390678 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd028009320 con 0x7fd0340f6510 2023-12-01T22:12:01.145 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:01.143+0000 7fd03ac81700 1 -- 172.21.15.73:0/1813390678 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd034106e20 con 0x7fd0340f6510 2023-12-01T22:12:01.145 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:01.143+0000 7fd0317fa700 1 -- 172.21.15.73:0/1813390678 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd028013f00 con 0x7fd0340f6510 2023-12-01T22:12:01.146 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:01.143+0000 7fd0317fa700 1 -- 172.21.15.73:0/1813390678 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd028014d90 con 0x7fd0340f6510 2023-12-01T22:12:01.146 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:01.143+0000 7fd03ac81700 1 -- 172.21.15.73:0/1813390678 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0341072b0 con 0x7fd0340f6510 2023-12-01T22:12:01.146 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:01.144+0000 7fd0317fa700 1 -- 172.21.15.73:0/1813390678 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd0280095f0 con 0x7fd0340f6510 2023-12-01T22:12:01.147 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:01.144+0000 7fd03ac81700 1 -- 172.21.15.73:0/1813390678 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd034045ad0 con 0x7fd0340f6510 2023-12-01T22:12:01.147 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:01.144+0000 7fd0317fa700 1 --2- 172.21.15.73:0/1813390678 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fd02402ff60 0x7fd024032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:12:01.147 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:01.144+0000 7fd0317fa700 1 -- 172.21.15.73:0/1813390678 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd0280410c0 con 0x7fd0340f6510 2023-12-01T22:12:01.147 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:01.144+0000 7fd033fff700 1 --2- 172.21.15.73:0/1813390678 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fd02402ff60 0x7fd024032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:12:01.147 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:01.145+0000 7fd033fff700 1 --2- 172.21.15.73:0/1813390678 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fd02402ff60 0x7fd024032420 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7fd0200096a0 tx=0x7fd020006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:12:01.150 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:01.149+0000 7fd0317fa700 1 -- 172.21.15.73:0/1813390678 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd028014420 con 0x7fd0340f6510 2023-12-01T22:12:01.356 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:01.355+0000 7fd03ac81700 1 -- 172.21.15.73:0/1813390678 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd0340fbf60 con 0x7fd0340f6510 2023-12-01T22:12:01.357 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:01.355+0000 7fd0317fa700 1 -- 172.21.15.73:0/1813390678 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd028010230 con 0x7fd0340f6510 2023-12-01T22:12:01.357 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:12:01.357 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:12:01.359 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:01.357+0000 7fd03ac81700 1 -- 172.21.15.73:0/1813390678 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fd02402ff60 msgr2=0x7fd024032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:12:01.359 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:01.357+0000 7fd03ac81700 1 --2- 172.21.15.73:0/1813390678 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fd02402ff60 0x7fd024032420 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7fd0200096a0 tx=0x7fd020006b40).stop 2023-12-01T22:12:01.359 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:01.357+0000 7fd03ac81700 1 -- 172.21.15.73:0/1813390678 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0340f6510 msgr2=0x7fd034106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:12:01.359 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:01.357+0000 7fd03ac81700 1 --2- 172.21.15.73:0/1813390678 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0340f6510 0x7fd034106730 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7fd028013510 tx=0x7fd02800b040).stop 2023-12-01T22:12:01.359 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:01.357+0000 7fd03ac81700 1 -- 172.21.15.73:0/1813390678 shutdown_connections 2023-12-01T22:12:01.359 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:01.357+0000 7fd03ac81700 1 --2- 172.21.15.73:0/1813390678 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fd02402ff60 0x7fd024032420 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:12:01.359 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:01.357+0000 7fd03ac81700 1 --2- 172.21.15.73:0/1813390678 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0340f6510 0x7fd034106730 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:12:01.359 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:01.357+0000 7fd03ac81700 1 -- 172.21.15.73:0/1813390678 >> 172.21.15.73:0/1813390678 conn(0x7fd0340f1c70 msgr2=0x7fd0340f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:12:01.359 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:01.357+0000 7fd03ac81700 1 -- 172.21.15.73:0/1813390678 shutdown_connections 2023-12-01T22:12:01.359 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:01.357+0000 7fd03ac81700 1 -- 172.21.15.73:0/1813390678 wait complete. 2023-12-01T22:12:01.360 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:12:01.776 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:12:01 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:12:00.623804+0000 mgr.smithi029.loxrfu (mgr.14164) 25 : cluster [DBG] pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:12:01.776 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:12:01 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:12:01.356715+0000 mon.smithi029 (mon.0) 126 : audit [DBG] from='client.? 172.21.15.73:0/1813390678' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:12:02.738 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:12:02.738 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:12:03.891 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:03.888+0000 7fd13abe7700 1 -- 172.21.15.73:0/2222006886 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1340f62a0 msgr2=0x7fd1340f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:12:03.891 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:03.888+0000 7fd13abe7700 1 --2- 172.21.15.73:0/2222006886 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1340f62a0 0x7fd1340f66c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7fd124009fb0 tx=0x7fd124009b30).stop 2023-12-01T22:12:03.891 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:03.888+0000 7fd13abe7700 1 -- 172.21.15.73:0/2222006886 shutdown_connections 2023-12-01T22:12:03.891 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:03.888+0000 7fd13abe7700 1 --2- 172.21.15.73:0/2222006886 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1340f62a0 0x7fd1340f66c0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:12:03.891 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:03.888+0000 7fd13abe7700 1 -- 172.21.15.73:0/2222006886 >> 172.21.15.73:0/2222006886 conn(0x7fd1340f1ac0 msgr2=0x7fd1340f3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:12:03.891 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:03.888+0000 7fd13abe7700 1 -- 172.21.15.73:0/2222006886 shutdown_connections 2023-12-01T22:12:03.891 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:03.888+0000 7fd13abe7700 1 -- 172.21.15.73:0/2222006886 wait complete. 2023-12-01T22:12:03.891 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:03.888+0000 7fd13abe7700 1 Processor -- start 2023-12-01T22:12:03.891 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:03.888+0000 7fd13abe7700 1 -- start start 2023-12-01T22:12:03.891 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:03.889+0000 7fd13abe7700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1340f62a0 0x7fd1341086b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:12:03.891 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:03.889+0000 7fd13abe7700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd134108ba0 con 0x7fd1340f62a0 2023-12-01T22:12:03.892 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:03.889+0000 7fd139be5700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1340f62a0 0x7fd1341086b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:12:03.892 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:03.889+0000 7fd139be5700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1340f62a0 0x7fd1341086b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:52282/0 (socket says 172.21.15.73:52282) 2023-12-01T22:12:03.892 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:03.889+0000 7fd139be5700 1 -- 172.21.15.73:0/3086230267 learned_addr learned my addr 172.21.15.73:0/3086230267 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:12:03.892 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:03.889+0000 7fd139be5700 1 -- 172.21.15.73:0/3086230267 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd124005040 con 0x7fd1340f62a0 2023-12-01T22:12:03.892 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:03.889+0000 7fd139be5700 1 --2- 172.21.15.73:0/3086230267 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1340f62a0 0x7fd1341086b0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7fd124013510 tx=0x7fd12400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:12:03.893 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:03.889+0000 7fd12a7fc700 1 -- 172.21.15.73:0/3086230267 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd124009320 con 0x7fd1340f62a0 2023-12-01T22:12:03.893 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:03.890+0000 7fd12a7fc700 1 -- 172.21.15.73:0/3086230267 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd124013f00 con 0x7fd1340f62a0 2023-12-01T22:12:03.893 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:03.890+0000 7fd13abe7700 1 -- 172.21.15.73:0/3086230267 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd134108da0 con 0x7fd1340f62a0 2023-12-01T22:12:03.893 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:03.890+0000 7fd12a7fc700 1 -- 172.21.15.73:0/3086230267 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd124014dc0 con 0x7fd1340f62a0 2023-12-01T22:12:03.893 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:03.890+0000 7fd13abe7700 1 -- 172.21.15.73:0/3086230267 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd134109230 con 0x7fd1340f62a0 2023-12-01T22:12:03.893 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:03.890+0000 7fd12a7fc700 1 -- 172.21.15.73:0/3086230267 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd1240095f0 con 0x7fd1340f62a0 2023-12-01T22:12:03.893 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:03.890+0000 7fd13abe7700 1 -- 172.21.15.73:0/3086230267 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd134045ad0 con 0x7fd1340f62a0 2023-12-01T22:12:03.893 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:03.891+0000 7fd12a7fc700 1 --2- 172.21.15.73:0/3086230267 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fd12002ff60 0x7fd120032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:12:03.893 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:03.891+0000 7fd12a7fc700 1 -- 172.21.15.73:0/3086230267 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd124041040 con 0x7fd1340f62a0 2023-12-01T22:12:03.894 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:03.891+0000 7fd1393e4700 1 --2- 172.21.15.73:0/3086230267 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fd12002ff60 0x7fd120032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:12:03.894 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:03.891+0000 7fd1393e4700 1 --2- 172.21.15.73:0/3086230267 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fd12002ff60 0x7fd120032420 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7fd1300096a0 tx=0x7fd130006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:12:03.897 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:03.895+0000 7fd12a7fc700 1 -- 172.21.15.73:0/3086230267 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd124014420 con 0x7fd1340f62a0 2023-12-01T22:12:04.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:12:03 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:12:02.624166+0000 mgr.smithi029.loxrfu (mgr.14164) 26 : cluster [DBG] pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:12:04.106 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:04.104+0000 7fd13abe7700 1 -- 172.21.15.73:0/3086230267 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd13406b1d0 con 0x7fd1340f62a0 2023-12-01T22:12:04.107 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:04.105+0000 7fd12a7fc700 1 -- 172.21.15.73:0/3086230267 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd124010230 con 0x7fd1340f62a0 2023-12-01T22:12:04.107 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:12:04.107 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:12:04.109 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:04.106+0000 7fd13abe7700 1 -- 172.21.15.73:0/3086230267 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fd12002ff60 msgr2=0x7fd120032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:12:04.109 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:04.106+0000 7fd13abe7700 1 --2- 172.21.15.73:0/3086230267 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fd12002ff60 0x7fd120032420 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7fd1300096a0 tx=0x7fd130006b40).stop 2023-12-01T22:12:04.109 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:04.106+0000 7fd13abe7700 1 -- 172.21.15.73:0/3086230267 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1340f62a0 msgr2=0x7fd1341086b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:12:04.109 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:04.106+0000 7fd13abe7700 1 --2- 172.21.15.73:0/3086230267 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1340f62a0 0x7fd1341086b0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7fd124013510 tx=0x7fd12400b040).stop 2023-12-01T22:12:04.109 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:04.106+0000 7fd13abe7700 1 -- 172.21.15.73:0/3086230267 shutdown_connections 2023-12-01T22:12:04.109 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:04.106+0000 7fd13abe7700 1 --2- 172.21.15.73:0/3086230267 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fd12002ff60 0x7fd120032420 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:12:04.109 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:04.106+0000 7fd13abe7700 1 --2- 172.21.15.73:0/3086230267 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1340f62a0 0x7fd1341086b0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:12:04.109 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:04.106+0000 7fd13abe7700 1 -- 172.21.15.73:0/3086230267 >> 172.21.15.73:0/3086230267 conn(0x7fd1340f1ac0 msgr2=0x7fd1340f28c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:12:04.110 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:04.107+0000 7fd13abe7700 1 -- 172.21.15.73:0/3086230267 shutdown_connections 2023-12-01T22:12:04.110 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:04.107+0000 7fd13abe7700 1 -- 172.21.15.73:0/3086230267 wait complete. 2023-12-01T22:12:04.110 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:12:05.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:12:04 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:12:04.106355+0000 mon.smithi029 (mon.0) 127 : audit [DBG] from='client.? 172.21.15.73:0/3086230267' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:12:05.507 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:12:05.507 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:12:06.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:12:05 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:12:04.624538+0000 mgr.smithi029.loxrfu (mgr.14164) 27 : cluster [DBG] pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:12:06.625 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:06.622+0000 7f40ae92e700 1 -- 172.21.15.73:0/355862826 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40a80f5410 msgr2=0x7f40a80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:12:06.625 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:06.622+0000 7f40ae92e700 1 --2- 172.21.15.73:0/355862826 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40a80f5410 0x7f40a80f5830 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f4098009fb0 tx=0x7f4098009b30).stop 2023-12-01T22:12:06.626 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:06.622+0000 7f40ae92e700 1 -- 172.21.15.73:0/355862826 shutdown_connections 2023-12-01T22:12:06.626 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:06.622+0000 7f40ae92e700 1 --2- 172.21.15.73:0/355862826 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40a80f5410 0x7f40a80f5830 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:12:06.626 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:06.622+0000 7f40ae92e700 1 -- 172.21.15.73:0/355862826 >> 172.21.15.73:0/355862826 conn(0x7f40a80f0b70 msgr2=0x7f40a80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:12:06.626 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:06.623+0000 7f40ae92e700 1 -- 172.21.15.73:0/355862826 shutdown_connections 2023-12-01T22:12:06.626 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:06.623+0000 7f40ae92e700 1 -- 172.21.15.73:0/355862826 wait complete. 2023-12-01T22:12:06.626 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:06.623+0000 7f40ae92e700 1 Processor -- start 2023-12-01T22:12:06.626 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:06.623+0000 7f40ae92e700 1 -- start start 2023-12-01T22:12:06.626 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:06.623+0000 7f40ae92e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40a80f5410 0x7f40a80fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:12:06.626 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:06.623+0000 7f40ae92e700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40a80fb820 con 0x7f40a80f5410 2023-12-01T22:12:06.626 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:06.624+0000 7f40a7fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40a80f5410 0x7f40a80fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:12:06.626 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:06.624+0000 7f40a7fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40a80f5410 0x7f40a80fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:55964/0 (socket says 172.21.15.73:55964) 2023-12-01T22:12:06.626 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:06.624+0000 7f40a7fff700 1 -- 172.21.15.73:0/3786729038 learned_addr learned my addr 172.21.15.73:0/3786729038 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:12:06.626 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:06.624+0000 7f40a7fff700 1 -- 172.21.15.73:0/3786729038 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4098005040 con 0x7f40a80f5410 2023-12-01T22:12:06.627 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:06.624+0000 7f40a7fff700 1 --2- 172.21.15.73:0/3786729038 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40a80f5410 0x7f40a80fd3a0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f4098013510 tx=0x7f409800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:12:06.630 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:06.624+0000 7f40a4ff9700 1 -- 172.21.15.73:0/3786729038 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4098009320 con 0x7f40a80f5410 2023-12-01T22:12:06.630 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:06.624+0000 7f40ae92e700 1 -- 172.21.15.73:0/3786729038 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40a80fb9c0 con 0x7f40a80f5410 2023-12-01T22:12:06.630 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:06.624+0000 7f40a4ff9700 1 -- 172.21.15.73:0/3786729038 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4098013f00 con 0x7f40a80f5410 2023-12-01T22:12:06.630 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:06.624+0000 7f40a4ff9700 1 -- 172.21.15.73:0/3786729038 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4098014d90 con 0x7f40a80f5410 2023-12-01T22:12:06.630 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:06.624+0000 7f40ae92e700 1 -- 172.21.15.73:0/3786729038 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40a80fbe50 con 0x7f40a80f5410 2023-12-01T22:12:06.630 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:06.625+0000 7f40a4ff9700 1 -- 172.21.15.73:0/3786729038 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f40980263a0 con 0x7f40a80f5410 2023-12-01T22:12:06.630 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:06.625+0000 7f40a4ff9700 1 --2- 172.21.15.73:0/3786729038 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f409002ffb0 0x7f4090032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:12:06.630 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:06.625+0000 7f40a4ff9700 1 -- 172.21.15.73:0/3786729038 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f40980410d0 con 0x7f40a80f5410 2023-12-01T22:12:06.631 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:06.625+0000 7f40ae92e700 1 -- 172.21.15.73:0/3786729038 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40a8059100 con 0x7f40a80f5410 2023-12-01T22:12:06.631 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:06.625+0000 7f40a77fe700 1 --2- 172.21.15.73:0/3786729038 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f409002ffb0 0x7f4090032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:12:06.631 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:06.626+0000 7f40a77fe700 1 --2- 172.21.15.73:0/3786729038 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f409002ffb0 0x7f4090032470 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f409c0096a0 tx=0x7f409c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:12:06.632 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:06.630+0000 7f40a4ff9700 1 -- 172.21.15.73:0/3786729038 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4098012920 con 0x7f40a80f5410 2023-12-01T22:12:06.838 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:06.836+0000 7f40ae92e700 1 -- 172.21.15.73:0/3786729038 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f40a80fc5b0 con 0x7f40a80f5410 2023-12-01T22:12:06.838 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:06.838+0000 7f40a4ff9700 1 -- 172.21.15.73:0/3786729038 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f409802b020 con 0x7f40a80f5410 2023-12-01T22:12:06.839 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:12:06.839 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:12:06.840 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:06.838+0000 7f40ae92e700 1 -- 172.21.15.73:0/3786729038 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f409002ffb0 msgr2=0x7f4090032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:12:06.840 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:06.838+0000 7f40ae92e700 1 --2- 172.21.15.73:0/3786729038 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f409002ffb0 0x7f4090032470 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f409c0096a0 tx=0x7f409c006b40).stop 2023-12-01T22:12:06.840 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:06.838+0000 7f40ae92e700 1 -- 172.21.15.73:0/3786729038 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40a80f5410 msgr2=0x7f40a80fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:12:06.840 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:06.838+0000 7f40ae92e700 1 --2- 172.21.15.73:0/3786729038 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40a80f5410 0x7f40a80fd3a0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f4098013510 tx=0x7f409800b040).stop 2023-12-01T22:12:06.840 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:06.838+0000 7f40ae92e700 1 -- 172.21.15.73:0/3786729038 shutdown_connections 2023-12-01T22:12:06.840 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:06.838+0000 7f40ae92e700 1 --2- 172.21.15.73:0/3786729038 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f409002ffb0 0x7f4090032470 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:12:06.840 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:06.838+0000 7f40ae92e700 1 --2- 172.21.15.73:0/3786729038 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40a80f5410 0x7f40a80fd3a0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:12:06.841 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:06.838+0000 7f40ae92e700 1 -- 172.21.15.73:0/3786729038 >> 172.21.15.73:0/3786729038 conn(0x7f40a80f0b70 msgr2=0x7f40a805fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:12:06.841 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:06.839+0000 7f40ae92e700 1 -- 172.21.15.73:0/3786729038 shutdown_connections 2023-12-01T22:12:06.841 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:06.839+0000 7f40ae92e700 1 -- 172.21.15.73:0/3786729038 wait complete. 2023-12-01T22:12:06.841 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:12:08.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:12:07 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:12:06.624750+0000 mgr.smithi029.loxrfu (mgr.14164) 28 : cluster [DBG] pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:12:08.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:12:07 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:12:06.838194+0000 mon.smithi029 (mon.0) 128 : audit [DBG] from='client.? 172.21.15.73:0/3786729038' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:12:08.226 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:12:08.227 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:12:10.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:12:09 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:12:08.625116+0000 mgr.smithi029.loxrfu (mgr.14164) 29 : cluster [DBG] pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:12:11.776 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:12:11 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:12:10.625498+0000 mgr.smithi029.loxrfu (mgr.14164) 30 : cluster [DBG] pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:12:13.717 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:13.590+0000 7f0b39d9b700 1 -- 172.21.15.73:0/1580807252 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b340f6510 msgr2=0x7f0b340f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:12:13.717 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:13.590+0000 7f0b39d9b700 1 --2- 172.21.15.73:0/1580807252 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b340f6510 0x7f0b340f6930 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f0b28009fb0 tx=0x7f0b28009b30).stop 2023-12-01T22:12:13.717 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:13.591+0000 7f0b39d9b700 1 -- 172.21.15.73:0/1580807252 shutdown_connections 2023-12-01T22:12:13.717 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:13.591+0000 7f0b39d9b700 1 --2- 172.21.15.73:0/1580807252 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b340f6510 0x7f0b340f6930 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:12:13.717 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:13.591+0000 7f0b39d9b700 1 -- 172.21.15.73:0/1580807252 >> 172.21.15.73:0/1580807252 conn(0x7f0b340f1c70 msgr2=0x7f0b340f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:12:13.717 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:13.591+0000 7f0b39d9b700 1 -- 172.21.15.73:0/1580807252 shutdown_connections 2023-12-01T22:12:13.718 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:13.591+0000 7f0b39d9b700 1 -- 172.21.15.73:0/1580807252 wait complete. 2023-12-01T22:12:13.718 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:13.591+0000 7f0b39d9b700 1 Processor -- start 2023-12-01T22:12:13.718 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:13.592+0000 7f0b39d9b700 1 -- start start 2023-12-01T22:12:13.718 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:13.592+0000 7f0b39d9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b340f6510 0x7f0b34106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:12:13.718 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:13.592+0000 7f0b39d9b700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b34106c20 con 0x7f0b340f6510 2023-12-01T22:12:13.718 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:13.592+0000 7f0b337fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b340f6510 0x7f0b34106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:12:13.718 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:13.592+0000 7f0b337fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b340f6510 0x7f0b34106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:55974/0 (socket says 172.21.15.73:55974) 2023-12-01T22:12:13.718 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:13.592+0000 7f0b337fe700 1 -- 172.21.15.73:0/997494227 learned_addr learned my addr 172.21.15.73:0/997494227 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:12:13.718 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:13.593+0000 7f0b337fe700 1 -- 172.21.15.73:0/997494227 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b28005040 con 0x7f0b340f6510 2023-12-01T22:12:13.718 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:13.593+0000 7f0b337fe700 1 --2- 172.21.15.73:0/997494227 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b340f6510 0x7f0b34106730 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f0b28013510 tx=0x7f0b2800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:12:13.718 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:13.593+0000 7f0b1bfff700 1 -- 172.21.15.73:0/997494227 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0b28009320 con 0x7f0b340f6510 2023-12-01T22:12:13.718 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:13.593+0000 7f0b39d9b700 1 -- 172.21.15.73:0/997494227 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b34106e20 con 0x7f0b340f6510 2023-12-01T22:12:13.718 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:13.593+0000 7f0b1bfff700 1 -- 172.21.15.73:0/997494227 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0b28013f00 con 0x7f0b340f6510 2023-12-01T22:12:13.718 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:13.593+0000 7f0b1bfff700 1 -- 172.21.15.73:0/997494227 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0b28014d90 con 0x7f0b340f6510 2023-12-01T22:12:13.719 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:13.593+0000 7f0b39d9b700 1 -- 172.21.15.73:0/997494227 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b341072b0 con 0x7f0b340f6510 2023-12-01T22:12:13.719 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:13.594+0000 7f0b1bfff700 1 -- 172.21.15.73:0/997494227 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0b28014420 con 0x7f0b340f6510 2023-12-01T22:12:13.719 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:13.594+0000 7f0b39d9b700 1 -- 172.21.15.73:0/997494227 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b34045ad0 con 0x7f0b340f6510 2023-12-01T22:12:13.719 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:13.594+0000 7f0b1bfff700 1 --2- 172.21.15.73:0/997494227 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f0b1c02ff60 0x7f0b1c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:12:13.719 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:13.594+0000 7f0b1bfff700 1 -- 172.21.15.73:0/997494227 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0b28041000 con 0x7f0b340f6510 2023-12-01T22:12:13.719 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:13.594+0000 7f0b32ffd700 1 --2- 172.21.15.73:0/997494227 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f0b1c02ff60 0x7f0b1c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:12:13.719 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:13.594+0000 7f0b32ffd700 1 --2- 172.21.15.73:0/997494227 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f0b1c02ff60 0x7f0b1c032420 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f0b240096a0 tx=0x7f0b24006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:12:13.719 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:13.599+0000 7f0b1bfff700 1 -- 172.21.15.73:0/997494227 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0b280095f0 con 0x7f0b340f6510 2023-12-01T22:12:13.805 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:13.804+0000 7f0b39d9b700 1 -- 172.21.15.73:0/997494227 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0b340fbf60 con 0x7f0b340f6510 2023-12-01T22:12:13.806 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:13.805+0000 7f0b1bfff700 1 -- 172.21.15.73:0/997494227 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0b2801db70 con 0x7f0b340f6510 2023-12-01T22:12:13.806 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:12:13.806 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:12:13.808 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:13.807+0000 7f0b39d9b700 1 -- 172.21.15.73:0/997494227 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f0b1c02ff60 msgr2=0x7f0b1c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:12:13.808 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:13.807+0000 7f0b39d9b700 1 --2- 172.21.15.73:0/997494227 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f0b1c02ff60 0x7f0b1c032420 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f0b240096a0 tx=0x7f0b24006b40).stop 2023-12-01T22:12:13.808 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:13.807+0000 7f0b39d9b700 1 -- 172.21.15.73:0/997494227 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b340f6510 msgr2=0x7f0b34106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:12:13.808 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:13.807+0000 7f0b39d9b700 1 --2- 172.21.15.73:0/997494227 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b340f6510 0x7f0b34106730 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f0b28013510 tx=0x7f0b2800b040).stop 2023-12-01T22:12:13.808 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:13.807+0000 7f0b39d9b700 1 -- 172.21.15.73:0/997494227 shutdown_connections 2023-12-01T22:12:13.808 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:13.807+0000 7f0b39d9b700 1 --2- 172.21.15.73:0/997494227 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f0b1c02ff60 0x7f0b1c032420 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:12:13.808 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:13.807+0000 7f0b39d9b700 1 --2- 172.21.15.73:0/997494227 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b340f6510 0x7f0b34106730 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:12:13.808 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:13.807+0000 7f0b39d9b700 1 -- 172.21.15.73:0/997494227 >> 172.21.15.73:0/997494227 conn(0x7f0b340f1c70 msgr2=0x7f0b340f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:12:13.808 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:13.807+0000 7f0b39d9b700 1 -- 172.21.15.73:0/997494227 shutdown_connections 2023-12-01T22:12:13.808 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:13.807+0000 7f0b39d9b700 1 -- 172.21.15.73:0/997494227 wait complete. 2023-12-01T22:12:13.809 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:12:14.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:12:13 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:12:12.625732+0000 mgr.smithi029.loxrfu (mgr.14164) 31 : cluster [DBG] pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:12:14.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:12:13 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:12:12.776137+0000 mon.smithi029 (mon.0) 129 : audit [INF] from='mgr.14164 172.21.15.29:0/3195927111' entity='mgr.smithi029.loxrfu' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi029.loxrfu/mirror_snapshot_schedule"}]: dispatch 2023-12-01T22:12:14.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:12:13 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:12:12.780327+0000 mon.smithi029 (mon.0) 130 : audit [INF] from='mgr.14164 172.21.15.29:0/3195927111' entity='mgr.smithi029.loxrfu' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi029.loxrfu/trash_purge_schedule"}]: dispatch 2023-12-01T22:12:15.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:12:14 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:12:13.805462+0000 mon.smithi029 (mon.0) 131 : audit [DBG] from='client.? 172.21.15.73:0/997494227' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:12:16.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:12:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:12:14.626033+0000 mgr.smithi029.loxrfu (mgr.14164) 32 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:12:16.636 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:12:16.637 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:12:17.762 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:17.761+0000 7ff838658700 1 -- 172.21.15.73:0/2868533130 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8300f5410 msgr2=0x7ff8300f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:12:17.762 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:17.761+0000 7ff838658700 1 --2- 172.21.15.73:0/2868533130 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8300f5410 0x7ff8300f5830 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7ff82c009fb0 tx=0x7ff82c009b30).stop 2023-12-01T22:12:17.763 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:17.761+0000 7ff838658700 1 -- 172.21.15.73:0/2868533130 shutdown_connections 2023-12-01T22:12:17.763 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:17.761+0000 7ff838658700 1 --2- 172.21.15.73:0/2868533130 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8300f5410 0x7ff8300f5830 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:12:17.763 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:17.761+0000 7ff838658700 1 -- 172.21.15.73:0/2868533130 >> 172.21.15.73:0/2868533130 conn(0x7ff8300f0b70 msgr2=0x7ff8300f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:12:17.763 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:17.761+0000 7ff838658700 1 -- 172.21.15.73:0/2868533130 shutdown_connections 2023-12-01T22:12:17.763 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:17.761+0000 7ff838658700 1 -- 172.21.15.73:0/2868533130 wait complete. 2023-12-01T22:12:17.763 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:17.761+0000 7ff838658700 1 Processor -- start 2023-12-01T22:12:17.763 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:17.761+0000 7ff838658700 1 -- start start 2023-12-01T22:12:17.763 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:17.762+0000 7ff838658700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8300f5410 0x7ff8300fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:12:17.763 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:17.762+0000 7ff838658700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8300fb820 con 0x7ff8300f5410 2023-12-01T22:12:17.763 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:17.762+0000 7ff8363f4700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8300f5410 0x7ff8300fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:12:17.763 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:17.762+0000 7ff8363f4700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8300f5410 0x7ff8300fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:39570/0 (socket says 172.21.15.73:39570) 2023-12-01T22:12:17.763 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:17.762+0000 7ff8363f4700 1 -- 172.21.15.73:0/1404370922 learned_addr learned my addr 172.21.15.73:0/1404370922 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:12:17.765 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:17.762+0000 7ff8363f4700 1 -- 172.21.15.73:0/1404370922 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff82c005040 con 0x7ff8300f5410 2023-12-01T22:12:17.765 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:17.762+0000 7ff8363f4700 1 --2- 172.21.15.73:0/1404370922 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8300f5410 0x7ff8300fd3a0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7ff82c013510 tx=0x7ff82c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:12:17.765 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:17.763+0000 7ff826ffd700 1 -- 172.21.15.73:0/1404370922 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff82c009320 con 0x7ff8300f5410 2023-12-01T22:12:17.765 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:17.763+0000 7ff838658700 1 -- 172.21.15.73:0/1404370922 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8300fb9c0 con 0x7ff8300f5410 2023-12-01T22:12:17.765 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:17.763+0000 7ff838658700 1 -- 172.21.15.73:0/1404370922 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8300fbe50 con 0x7ff8300f5410 2023-12-01T22:12:17.765 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:17.763+0000 7ff826ffd700 1 -- 172.21.15.73:0/1404370922 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff82c013f00 con 0x7ff8300f5410 2023-12-01T22:12:17.765 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:17.763+0000 7ff826ffd700 1 -- 172.21.15.73:0/1404370922 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff82c014ce0 con 0x7ff8300f5410 2023-12-01T22:12:17.765 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:17.763+0000 7ff826ffd700 1 -- 172.21.15.73:0/1404370922 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff82c0263a0 con 0x7ff8300f5410 2023-12-01T22:12:17.765 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:17.764+0000 7ff826ffd700 1 --2- 172.21.15.73:0/1404370922 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7ff81c02ff60 0x7ff81c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:12:17.765 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:17.764+0000 7ff826ffd700 1 -- 172.21.15.73:0/1404370922 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff82c040ed0 con 0x7ff8300f5410 2023-12-01T22:12:17.765 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:17.764+0000 7ff838658700 1 -- 172.21.15.73:0/1404370922 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff818004fa0 con 0x7ff8300f5410 2023-12-01T22:12:17.765 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:17.764+0000 7ff835bf3700 1 --2- 172.21.15.73:0/1404370922 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7ff81c02ff60 0x7ff81c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:12:17.766 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:17.764+0000 7ff835bf3700 1 --2- 172.21.15.73:0/1404370922 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7ff81c02ff60 0x7ff81c032420 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7ff8200096a0 tx=0x7ff820006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:12:17.769 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:17.769+0000 7ff826ffd700 1 -- 172.21.15.73:0/1404370922 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff82c01d380 con 0x7ff8300f5410 2023-12-01T22:12:17.979 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:17.979+0000 7ff838658700 1 -- 172.21.15.73:0/1404370922 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff818005670 con 0x7ff8300f5410 2023-12-01T22:12:17.980 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:17.979+0000 7ff826ffd700 1 -- 172.21.15.73:0/1404370922 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff82c014e60 con 0x7ff8300f5410 2023-12-01T22:12:17.980 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:12:17.980 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:12:17.982 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:17.981+0000 7ff838658700 1 -- 172.21.15.73:0/1404370922 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7ff81c02ff60 msgr2=0x7ff81c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:12:17.983 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:17.981+0000 7ff838658700 1 --2- 172.21.15.73:0/1404370922 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7ff81c02ff60 0x7ff81c032420 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7ff8200096a0 tx=0x7ff820006b40).stop 2023-12-01T22:12:17.983 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:17.981+0000 7ff838658700 1 -- 172.21.15.73:0/1404370922 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8300f5410 msgr2=0x7ff8300fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:12:17.983 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:17.981+0000 7ff838658700 1 --2- 172.21.15.73:0/1404370922 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8300f5410 0x7ff8300fd3a0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7ff82c013510 tx=0x7ff82c00b040).stop 2023-12-01T22:12:17.983 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:17.981+0000 7ff838658700 1 -- 172.21.15.73:0/1404370922 shutdown_connections 2023-12-01T22:12:17.983 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:17.981+0000 7ff838658700 1 --2- 172.21.15.73:0/1404370922 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7ff81c02ff60 0x7ff81c032420 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:12:17.983 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:17.981+0000 7ff838658700 1 --2- 172.21.15.73:0/1404370922 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8300f5410 0x7ff8300fd3a0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:12:17.983 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:17.981+0000 7ff838658700 1 -- 172.21.15.73:0/1404370922 >> 172.21.15.73:0/1404370922 conn(0x7ff8300f0b70 msgr2=0x7ff8300f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:12:17.983 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:17.981+0000 7ff838658700 1 -- 172.21.15.73:0/1404370922 shutdown_connections 2023-12-01T22:12:17.983 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:17.981+0000 7ff838658700 1 -- 172.21.15.73:0/1404370922 wait complete. 2023-12-01T22:12:17.983 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:12:18.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:12:17 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:12:16.626274+0000 mgr.smithi029.loxrfu (mgr.14164) 33 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:12:19.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:12:18 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:12:17.979822+0000 mon.smithi029 (mon.0) 132 : audit [DBG] from='client.? 172.21.15.73:0/1404370922' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:12:19.386 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:12:19.386 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:12:20.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:12:19 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:12:18.626631+0000 mgr.smithi029.loxrfu (mgr.14164) 34 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:12:20.541 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:20.540+0000 7f0651366700 1 -- 172.21.15.73:0/1692120443 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f064c0f47c0 msgr2=0x7f064c0f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:12:20.541 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:20.540+0000 7f0651366700 1 --2- 172.21.15.73:0/1692120443 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f064c0f47c0 0x7f064c0f4be0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f0640009fb0 tx=0x7f0640009b30).stop 2023-12-01T22:12:20.541 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:20.540+0000 7f0651366700 1 -- 172.21.15.73:0/1692120443 shutdown_connections 2023-12-01T22:12:20.541 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:20.540+0000 7f0651366700 1 --2- 172.21.15.73:0/1692120443 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f064c0f47c0 0x7f064c0f4be0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:12:20.541 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:20.540+0000 7f0651366700 1 -- 172.21.15.73:0/1692120443 >> 172.21.15.73:0/1692120443 conn(0x7f064c0f02e0 msgr2=0x7f064c0f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:12:20.541 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:20.540+0000 7f0651366700 1 -- 172.21.15.73:0/1692120443 shutdown_connections 2023-12-01T22:12:20.541 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:20.540+0000 7f0651366700 1 -- 172.21.15.73:0/1692120443 wait complete. 2023-12-01T22:12:20.543 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:20.541+0000 7f0651366700 1 Processor -- start 2023-12-01T22:12:20.544 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:20.541+0000 7f0651366700 1 -- start start 2023-12-01T22:12:20.544 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:20.541+0000 7f0651366700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f064c0f47c0 0x7f064c0f7370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:12:20.544 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:20.541+0000 7f0651366700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f064c0f57f0 con 0x7f064c0f47c0 2023-12-01T22:12:20.544 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:20.541+0000 7f064affd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f064c0f47c0 0x7f064c0f7370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:12:20.544 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:20.541+0000 7f064affd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f064c0f47c0 0x7f064c0f7370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:39598/0 (socket says 172.21.15.73:39598) 2023-12-01T22:12:20.544 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:20.541+0000 7f064affd700 1 -- 172.21.15.73:0/771322621 learned_addr learned my addr 172.21.15.73:0/771322621 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:12:20.544 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:20.541+0000 7f064affd700 1 -- 172.21.15.73:0/771322621 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0640005040 con 0x7f064c0f47c0 2023-12-01T22:12:20.544 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:20.542+0000 7f064affd700 1 --2- 172.21.15.73:0/771322621 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f064c0f47c0 0x7f064c0f7370 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f0640013510 tx=0x7f064000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:12:20.544 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:20.542+0000 7f06337fe700 1 -- 172.21.15.73:0/771322621 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0640009320 con 0x7f064c0f47c0 2023-12-01T22:12:20.544 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:20.542+0000 7f06337fe700 1 -- 172.21.15.73:0/771322621 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0640013f00 con 0x7f064c0f47c0 2023-12-01T22:12:20.544 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:20.542+0000 7f0651366700 1 -- 172.21.15.73:0/771322621 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f064c0f5990 con 0x7f064c0f47c0 2023-12-01T22:12:20.544 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:20.542+0000 7f06337fe700 1 -- 172.21.15.73:0/771322621 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0640014d90 con 0x7f064c0f47c0 2023-12-01T22:12:20.544 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:20.542+0000 7f0651366700 1 -- 172.21.15.73:0/771322621 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f064c0f5e20 con 0x7f064c0f47c0 2023-12-01T22:12:20.544 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:20.543+0000 7f06337fe700 1 -- 172.21.15.73:0/771322621 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f06400263a0 con 0x7f064c0f47c0 2023-12-01T22:12:20.544 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:20.543+0000 7f0651366700 1 -- 172.21.15.73:0/771322621 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f064c059100 con 0x7f064c0f47c0 2023-12-01T22:12:20.545 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:20.543+0000 7f06337fe700 1 --2- 172.21.15.73:0/771322621 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f063402ff60 0x7f0634032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:12:20.545 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:20.543+0000 7f06337fe700 1 -- 172.21.15.73:0/771322621 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0640040ff0 con 0x7f064c0f47c0 2023-12-01T22:12:20.545 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:20.543+0000 7f064a7fc700 1 --2- 172.21.15.73:0/771322621 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f063402ff60 0x7f0634032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:12:20.545 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:20.543+0000 7f064a7fc700 1 --2- 172.21.15.73:0/771322621 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f063402ff60 0x7f0634032420 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f063c0096a0 tx=0x7f063c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:12:20.548 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:20.548+0000 7f06337fe700 1 -- 172.21.15.73:0/771322621 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f06400095f0 con 0x7f064c0f47c0 2023-12-01T22:12:20.754 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:20.754+0000 7f0651366700 1 -- 172.21.15.73:0/771322621 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f064c0fbf60 con 0x7f064c0f47c0 2023-12-01T22:12:20.755 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:20.755+0000 7f06337fe700 1 -- 172.21.15.73:0/771322621 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f064001db70 con 0x7f064c0f47c0 2023-12-01T22:12:20.755 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:12:20.755 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:12:20.757 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:20.756+0000 7f0651366700 1 -- 172.21.15.73:0/771322621 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f063402ff60 msgr2=0x7f0634032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:12:20.757 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:20.756+0000 7f0651366700 1 --2- 172.21.15.73:0/771322621 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f063402ff60 0x7f0634032420 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f063c0096a0 tx=0x7f063c006b40).stop 2023-12-01T22:12:20.757 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:20.756+0000 7f0651366700 1 -- 172.21.15.73:0/771322621 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f064c0f47c0 msgr2=0x7f064c0f7370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:12:20.758 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:20.756+0000 7f0651366700 1 --2- 172.21.15.73:0/771322621 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f064c0f47c0 0x7f064c0f7370 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f0640013510 tx=0x7f064000b040).stop 2023-12-01T22:12:20.758 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:20.757+0000 7f0651366700 1 -- 172.21.15.73:0/771322621 shutdown_connections 2023-12-01T22:12:20.758 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:20.757+0000 7f0651366700 1 --2- 172.21.15.73:0/771322621 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f063402ff60 0x7f0634032420 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:12:20.758 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:20.757+0000 7f0651366700 1 --2- 172.21.15.73:0/771322621 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f064c0f47c0 0x7f064c0f7370 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:12:20.758 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:20.757+0000 7f0651366700 1 -- 172.21.15.73:0/771322621 >> 172.21.15.73:0/771322621 conn(0x7f064c0f02e0 msgr2=0x7f064c0f0e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:12:20.758 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:20.757+0000 7f0651366700 1 -- 172.21.15.73:0/771322621 shutdown_connections 2023-12-01T22:12:20.758 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:20.757+0000 7f0651366700 1 -- 172.21.15.73:0/771322621 wait complete. 2023-12-01T22:12:20.758 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:12:21.776 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:12:21 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:12:20.626959+0000 mgr.smithi029.loxrfu (mgr.14164) 35 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:12:21.777 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:12:21 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:12:20.754870+0000 mon.smithi029 (mon.0) 133 : audit [DBG] from='client.? 172.21.15.73:0/771322621' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:12:22.174 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:12:22.174 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:12:23.358 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:23.354+0000 7f6b3b851700 1 -- 172.21.15.73:0/3329537275 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6b34060280 msgr2=0x7f6b340f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:12:23.358 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:23.354+0000 7f6b3b851700 1 --2- 172.21.15.73:0/3329537275 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6b34060280 0x7f6b340f90f0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f6b24009fb0 tx=0x7f6b24009b30).stop 2023-12-01T22:12:23.358 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:23.354+0000 7f6b3b851700 1 -- 172.21.15.73:0/3329537275 shutdown_connections 2023-12-01T22:12:23.358 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:23.354+0000 7f6b3b851700 1 --2- 172.21.15.73:0/3329537275 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6b34060280 0x7f6b340f90f0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:12:23.358 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:23.354+0000 7f6b3b851700 1 -- 172.21.15.73:0/3329537275 >> 172.21.15.73:0/3329537275 conn(0x7f6b340f0b70 msgr2=0x7f6b340f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:12:23.358 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:23.354+0000 7f6b3b851700 1 -- 172.21.15.73:0/3329537275 shutdown_connections 2023-12-01T22:12:23.358 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:23.354+0000 7f6b3b851700 1 -- 172.21.15.73:0/3329537275 wait complete. 2023-12-01T22:12:23.359 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:23.355+0000 7f6b3b851700 1 Processor -- start 2023-12-01T22:12:23.359 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:23.355+0000 7f6b3b851700 1 -- start start 2023-12-01T22:12:23.359 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:23.355+0000 7f6b3b851700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6b341067b0 0x7f6b34106bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:12:23.359 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:23.355+0000 7f6b3b851700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b341070c0 con 0x7f6b341067b0 2023-12-01T22:12:23.359 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:23.355+0000 7f6b395ed700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6b341067b0 0x7f6b34106bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:12:23.359 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:23.355+0000 7f6b395ed700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6b341067b0 0x7f6b34106bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:39618/0 (socket says 172.21.15.73:39618) 2023-12-01T22:12:23.359 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:23.355+0000 7f6b395ed700 1 -- 172.21.15.73:0/3990611837 learned_addr learned my addr 172.21.15.73:0/3990611837 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:12:23.359 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:23.356+0000 7f6b395ed700 1 -- 172.21.15.73:0/3990611837 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b24005040 con 0x7f6b341067b0 2023-12-01T22:12:23.359 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:23.356+0000 7f6b395ed700 1 --2- 172.21.15.73:0/3990611837 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6b341067b0 0x7f6b34106bd0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f6b24009f80 tx=0x7f6b24009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:12:23.359 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:23.356+0000 7f6b29ffb700 1 -- 172.21.15.73:0/3990611837 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6b240136c0 con 0x7f6b341067b0 2023-12-01T22:12:23.359 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:23.356+0000 7f6b3b851700 1 -- 172.21.15.73:0/3990611837 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b341072c0 con 0x7f6b341067b0 2023-12-01T22:12:23.359 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:23.356+0000 7f6b29ffb700 1 -- 172.21.15.73:0/3990611837 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6b240095f0 con 0x7f6b341067b0 2023-12-01T22:12:23.359 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:23.356+0000 7f6b29ffb700 1 -- 172.21.15.73:0/3990611837 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6b24014420 con 0x7f6b341067b0 2023-12-01T22:12:23.360 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:23.356+0000 7f6b3b851700 1 -- 172.21.15.73:0/3990611837 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b34107e10 con 0x7f6b341067b0 2023-12-01T22:12:23.360 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:23.357+0000 7f6b3b851700 1 -- 172.21.15.73:0/3990611837 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b34059100 con 0x7f6b341067b0 2023-12-01T22:12:23.360 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:23.357+0000 7f6b29ffb700 1 -- 172.21.15.73:0/3990611837 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6b24025060 con 0x7f6b341067b0 2023-12-01T22:12:23.360 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:23.357+0000 7f6b29ffb700 1 --2- 172.21.15.73:0/3990611837 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f6b2002ff60 0x7f6b20032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:12:23.361 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:23.357+0000 7f6b29ffb700 1 -- 172.21.15.73:0/3990611837 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6b24040e40 con 0x7f6b341067b0 2023-12-01T22:12:23.361 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:23.358+0000 7f6b38dec700 1 --2- 172.21.15.73:0/3990611837 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f6b2002ff60 0x7f6b20032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:12:23.361 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:23.358+0000 7f6b38dec700 1 --2- 172.21.15.73:0/3990611837 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f6b2002ff60 0x7f6b20032420 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7f6b300096a0 tx=0x7f6b30006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:12:23.364 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:23.362+0000 7f6b29ffb700 1 -- 172.21.15.73:0/3990611837 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6b2401c950 con 0x7f6b341067b0 2023-12-01T22:12:23.568 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:23.566+0000 7f6b3b851700 1 -- 172.21.15.73:0/3990611837 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6b341085d0 con 0x7f6b341067b0 2023-12-01T22:12:23.569 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:23.567+0000 7f6b29ffb700 1 -- 172.21.15.73:0/3990611837 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6b24042090 con 0x7f6b341067b0 2023-12-01T22:12:23.569 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:12:23.569 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:12:23.570 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:23.569+0000 7f6b3b851700 1 -- 172.21.15.73:0/3990611837 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f6b2002ff60 msgr2=0x7f6b20032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:12:23.571 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:23.569+0000 7f6b3b851700 1 --2- 172.21.15.73:0/3990611837 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f6b2002ff60 0x7f6b20032420 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7f6b300096a0 tx=0x7f6b30006b40).stop 2023-12-01T22:12:23.571 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:23.569+0000 7f6b3b851700 1 -- 172.21.15.73:0/3990611837 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6b341067b0 msgr2=0x7f6b34106bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:12:23.571 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:23.569+0000 7f6b3b851700 1 --2- 172.21.15.73:0/3990611837 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6b341067b0 0x7f6b34106bd0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f6b24009f80 tx=0x7f6b24009b30).stop 2023-12-01T22:12:23.571 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:23.569+0000 7f6b3b851700 1 -- 172.21.15.73:0/3990611837 shutdown_connections 2023-12-01T22:12:23.571 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:23.569+0000 7f6b3b851700 1 --2- 172.21.15.73:0/3990611837 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f6b2002ff60 0x7f6b20032420 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:12:23.571 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:23.569+0000 7f6b3b851700 1 --2- 172.21.15.73:0/3990611837 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6b341067b0 0x7f6b34106bd0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:12:23.571 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:23.569+0000 7f6b3b851700 1 -- 172.21.15.73:0/3990611837 >> 172.21.15.73:0/3990611837 conn(0x7f6b340f0b70 msgr2=0x7f6b340f1680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:12:23.571 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:23.569+0000 7f6b3b851700 1 -- 172.21.15.73:0/3990611837 shutdown_connections 2023-12-01T22:12:23.571 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:23.569+0000 7f6b3b851700 1 -- 172.21.15.73:0/3990611837 wait complete. 2023-12-01T22:12:23.571 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:12:24.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:12:23 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:12:22.627239+0000 mgr.smithi029.loxrfu (mgr.14164) 36 : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:12:24.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:12:23 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:12:23.568429+0000 mon.smithi029 (mon.0) 134 : audit [DBG] from='client.? 172.21.15.73:0/3990611837' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:12:24.989 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:12:24.990 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:12:26.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:12:25 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:12:24.627593+0000 mgr.smithi029.loxrfu (mgr.14164) 37 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:12:26.212 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:26.209+0000 7f2a45b26700 1 -- 172.21.15.73:0/864010174 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a400f5410 msgr2=0x7f2a400f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:12:26.213 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:26.209+0000 7f2a45b26700 1 --2- 172.21.15.73:0/864010174 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a400f5410 0x7f2a400f5830 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f2a28009fb0 tx=0x7f2a28009b30).stop 2023-12-01T22:12:26.213 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:26.209+0000 7f2a45b26700 1 -- 172.21.15.73:0/864010174 shutdown_connections 2023-12-01T22:12:26.213 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:26.209+0000 7f2a45b26700 1 --2- 172.21.15.73:0/864010174 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a400f5410 0x7f2a400f5830 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:12:26.213 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:26.209+0000 7f2a45b26700 1 -- 172.21.15.73:0/864010174 >> 172.21.15.73:0/864010174 conn(0x7f2a400f0b70 msgr2=0x7f2a400f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:12:26.213 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:26.209+0000 7f2a45b26700 1 -- 172.21.15.73:0/864010174 shutdown_connections 2023-12-01T22:12:26.213 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:26.209+0000 7f2a45b26700 1 -- 172.21.15.73:0/864010174 wait complete. 2023-12-01T22:12:26.213 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:26.210+0000 7f2a45b26700 1 Processor -- start 2023-12-01T22:12:26.213 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:26.210+0000 7f2a45b26700 1 -- start start 2023-12-01T22:12:26.213 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:26.210+0000 7f2a45b26700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a400f5410 0x7f2a400fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:12:26.213 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:26.210+0000 7f2a45b26700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a400fb820 con 0x7f2a400f5410 2023-12-01T22:12:26.213 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:26.210+0000 7f2a3f7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a400f5410 0x7f2a400fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:12:26.213 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:26.210+0000 7f2a3f7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a400f5410 0x7f2a400fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:54616/0 (socket says 172.21.15.73:54616) 2023-12-01T22:12:26.213 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:26.210+0000 7f2a3f7fe700 1 -- 172.21.15.73:0/749479312 learned_addr learned my addr 172.21.15.73:0/749479312 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:12:26.214 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:26.210+0000 7f2a3f7fe700 1 -- 172.21.15.73:0/749479312 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a28005040 con 0x7f2a400f5410 2023-12-01T22:12:26.214 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:26.211+0000 7f2a3f7fe700 1 --2- 172.21.15.73:0/749479312 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a400f5410 0x7f2a400fd3a0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f2a28013510 tx=0x7f2a2800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:12:26.214 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:26.211+0000 7f2a37fff700 1 -- 172.21.15.73:0/749479312 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2a28009320 con 0x7f2a400f5410 2023-12-01T22:12:26.214 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:26.211+0000 7f2a45b26700 1 -- 172.21.15.73:0/749479312 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a400fb9c0 con 0x7f2a400f5410 2023-12-01T22:12:26.214 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:26.211+0000 7f2a37fff700 1 -- 172.21.15.73:0/749479312 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2a28013f00 con 0x7f2a400f5410 2023-12-01T22:12:26.214 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:26.211+0000 7f2a37fff700 1 -- 172.21.15.73:0/749479312 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2a28014db0 con 0x7f2a400f5410 2023-12-01T22:12:26.214 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:26.211+0000 7f2a45b26700 1 -- 172.21.15.73:0/749479312 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a400fbe50 con 0x7f2a400f5410 2023-12-01T22:12:26.215 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:26.212+0000 7f2a37fff700 1 -- 172.21.15.73:0/749479312 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2a280095f0 con 0x7f2a400f5410 2023-12-01T22:12:26.215 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:26.212+0000 7f2a45b26700 1 -- 172.21.15.73:0/749479312 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a40059100 con 0x7f2a400f5410 2023-12-01T22:12:26.215 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:26.212+0000 7f2a37fff700 1 --2- 172.21.15.73:0/749479312 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f2a2c02ff60 0x7f2a2c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:12:26.215 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:26.212+0000 7f2a37fff700 1 -- 172.21.15.73:0/749479312 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2a28040fa0 con 0x7f2a400f5410 2023-12-01T22:12:26.215 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:26.212+0000 7f2a3effd700 1 --2- 172.21.15.73:0/749479312 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f2a2c02ff60 0x7f2a2c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:12:26.215 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:26.212+0000 7f2a3effd700 1 --2- 172.21.15.73:0/749479312 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f2a2c02ff60 0x7f2a2c032420 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f2a300096a0 tx=0x7f2a30006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:12:26.218 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:26.217+0000 7f2a37fff700 1 -- 172.21.15.73:0/749479312 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2a28014420 con 0x7f2a400f5410 2023-12-01T22:12:26.421 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:26.419+0000 7f2a45b26700 1 -- 172.21.15.73:0/749479312 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2a400fae60 con 0x7f2a400f5410 2023-12-01T22:12:26.422 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:26.420+0000 7f2a37fff700 1 -- 172.21.15.73:0/749479312 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2a2801c920 con 0x7f2a400f5410 2023-12-01T22:12:26.422 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:12:26.422 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:12:26.424 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:26.421+0000 7f2a45b26700 1 -- 172.21.15.73:0/749479312 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f2a2c02ff60 msgr2=0x7f2a2c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:12:26.424 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:26.422+0000 7f2a45b26700 1 --2- 172.21.15.73:0/749479312 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f2a2c02ff60 0x7f2a2c032420 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f2a300096a0 tx=0x7f2a30006b40).stop 2023-12-01T22:12:26.424 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:26.422+0000 7f2a45b26700 1 -- 172.21.15.73:0/749479312 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a400f5410 msgr2=0x7f2a400fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:12:26.424 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:26.422+0000 7f2a45b26700 1 --2- 172.21.15.73:0/749479312 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a400f5410 0x7f2a400fd3a0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f2a28013510 tx=0x7f2a2800b040).stop 2023-12-01T22:12:26.424 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:26.422+0000 7f2a45b26700 1 -- 172.21.15.73:0/749479312 shutdown_connections 2023-12-01T22:12:26.424 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:26.422+0000 7f2a45b26700 1 --2- 172.21.15.73:0/749479312 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f2a2c02ff60 0x7f2a2c032420 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:12:26.424 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:26.422+0000 7f2a45b26700 1 --2- 172.21.15.73:0/749479312 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a400f5410 0x7f2a400fd3a0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:12:26.424 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:26.422+0000 7f2a45b26700 1 -- 172.21.15.73:0/749479312 >> 172.21.15.73:0/749479312 conn(0x7f2a400f0b70 msgr2=0x7f2a4005fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:12:26.424 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:26.422+0000 7f2a45b26700 1 -- 172.21.15.73:0/749479312 shutdown_connections 2023-12-01T22:12:26.424 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:26.422+0000 7f2a45b26700 1 -- 172.21.15.73:0/749479312 wait complete. 2023-12-01T22:12:26.425 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:12:27.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:12:26 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:12:26.421302+0000 mon.smithi029 (mon.0) 135 : audit [DBG] from='client.? 172.21.15.73:0/749479312' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:12:27.853 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:12:27.854 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:12:28.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:12:27 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:12:26.627957+0000 mgr.smithi029.loxrfu (mgr.14164) 38 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:12:28.992 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:28.987+0000 7f37588b5700 1 -- 172.21.15.73:0/2781894444 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f37540f4920 msgr2=0x7f37540f4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:12:28.992 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:28.987+0000 7f37588b5700 1 --2- 172.21.15.73:0/2781894444 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f37540f4920 0x7f37540f4d40 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f3744009fb0 tx=0x7f3744009b30).stop 2023-12-01T22:12:28.992 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:28.987+0000 7f37588b5700 1 -- 172.21.15.73:0/2781894444 shutdown_connections 2023-12-01T22:12:28.993 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:28.987+0000 7f37588b5700 1 --2- 172.21.15.73:0/2781894444 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f37540f4920 0x7f37540f4d40 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:12:28.993 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:28.987+0000 7f37588b5700 1 -- 172.21.15.73:0/2781894444 >> 172.21.15.73:0/2781894444 conn(0x7f37540f0120 msgr2=0x7f37540f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:12:28.993 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:28.987+0000 7f37588b5700 1 -- 172.21.15.73:0/2781894444 shutdown_connections 2023-12-01T22:12:28.993 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:28.987+0000 7f37588b5700 1 -- 172.21.15.73:0/2781894444 wait complete. 2023-12-01T22:12:28.993 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:28.988+0000 7f37588b5700 1 Processor -- start 2023-12-01T22:12:28.993 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:28.988+0000 7f37588b5700 1 -- start start 2023-12-01T22:12:28.993 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:28.988+0000 7f37588b5700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f37540f4920 0x7f375410b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:12:28.993 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:28.988+0000 7f37588b5700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3754106fc0 con 0x7f37540f4920 2023-12-01T22:12:28.993 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:28.988+0000 7f37537fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f37540f4920 0x7f375410b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:12:28.993 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:28.988+0000 7f37537fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f37540f4920 0x7f375410b5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:54626/0 (socket says 172.21.15.73:54626) 2023-12-01T22:12:28.993 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:28.988+0000 7f37537fe700 1 -- 172.21.15.73:0/820243333 learned_addr learned my addr 172.21.15.73:0/820243333 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:12:28.993 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:28.988+0000 7f37537fe700 1 -- 172.21.15.73:0/820243333 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3744005040 con 0x7f37540f4920 2023-12-01T22:12:28.994 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:28.989+0000 7f37537fe700 1 --2- 172.21.15.73:0/820243333 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f37540f4920 0x7f375410b5d0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f3744013510 tx=0x7f374400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:12:28.994 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:28.989+0000 7f373bfff700 1 -- 172.21.15.73:0/820243333 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3744009320 con 0x7f37540f4920 2023-12-01T22:12:28.994 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:28.989+0000 7f37588b5700 1 -- 172.21.15.73:0/820243333 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37540f16e0 con 0x7f37540f4920 2023-12-01T22:12:28.994 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:28.989+0000 7f373bfff700 1 -- 172.21.15.73:0/820243333 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3744013f00 con 0x7f37540f4920 2023-12-01T22:12:28.994 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:28.989+0000 7f37588b5700 1 -- 172.21.15.73:0/820243333 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f375410bac0 con 0x7f37540f4920 2023-12-01T22:12:28.994 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:28.989+0000 7f373bfff700 1 -- 172.21.15.73:0/820243333 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3744014de0 con 0x7f37540f4920 2023-12-01T22:12:28.994 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:28.990+0000 7f373bfff700 1 -- 172.21.15.73:0/820243333 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f374401e420 con 0x7f37540f4920 2023-12-01T22:12:28.994 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:28.990+0000 7f373bfff700 1 --2- 172.21.15.73:0/820243333 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f373c02ff60 0x7f373c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:12:28.994 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:28.990+0000 7f373bfff700 1 -- 172.21.15.73:0/820243333 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3744041090 con 0x7f37540f4920 2023-12-01T22:12:28.994 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:28.990+0000 7f37588b5700 1 -- 172.21.15.73:0/820243333 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3754045ad0 con 0x7f37540f4920 2023-12-01T22:12:28.994 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:28.990+0000 7f3752ffd700 1 --2- 172.21.15.73:0/820243333 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f373c02ff60 0x7f373c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:12:28.994 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:28.990+0000 7f3752ffd700 1 --2- 172.21.15.73:0/820243333 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f373c02ff60 0x7f373c032420 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7f37480096a0 tx=0x7f3748006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:12:28.996 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:28.995+0000 7f373bfff700 1 -- 172.21.15.73:0/820243333 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f37440273f0 con 0x7f37540f4920 2023-12-01T22:12:29.204 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:29.202+0000 7f37588b5700 1 -- 172.21.15.73:0/820243333 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f37540008d0 con 0x7f37540f4920 2023-12-01T22:12:29.205 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:29.203+0000 7f373bfff700 1 -- 172.21.15.73:0/820243333 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f374401c970 con 0x7f37540f4920 2023-12-01T22:12:29.205 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:12:29.205 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:12:29.206 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:29.205+0000 7f37588b5700 1 -- 172.21.15.73:0/820243333 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f373c02ff60 msgr2=0x7f373c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:12:29.207 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:29.205+0000 7f37588b5700 1 --2- 172.21.15.73:0/820243333 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f373c02ff60 0x7f373c032420 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7f37480096a0 tx=0x7f3748006b40).stop 2023-12-01T22:12:29.207 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:29.205+0000 7f37588b5700 1 -- 172.21.15.73:0/820243333 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f37540f4920 msgr2=0x7f375410b5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:12:29.207 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:29.205+0000 7f37588b5700 1 --2- 172.21.15.73:0/820243333 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f37540f4920 0x7f375410b5d0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f3744013510 tx=0x7f374400b040).stop 2023-12-01T22:12:29.207 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:29.205+0000 7f37588b5700 1 -- 172.21.15.73:0/820243333 shutdown_connections 2023-12-01T22:12:29.207 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:29.205+0000 7f37588b5700 1 --2- 172.21.15.73:0/820243333 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f373c02ff60 0x7f373c032420 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:12:29.207 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:29.205+0000 7f37588b5700 1 --2- 172.21.15.73:0/820243333 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f37540f4920 0x7f375410b5d0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:12:29.207 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:29.205+0000 7f37588b5700 1 -- 172.21.15.73:0/820243333 >> 172.21.15.73:0/820243333 conn(0x7f37540f0120 msgr2=0x7f37540f2380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:12:29.207 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:29.205+0000 7f37588b5700 1 -- 172.21.15.73:0/820243333 shutdown_connections 2023-12-01T22:12:29.207 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:29.205+0000 7f37588b5700 1 -- 172.21.15.73:0/820243333 wait complete. 2023-12-01T22:12:29.207 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:12:30.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:12:29 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:12:28.628280+0000 mgr.smithi029.loxrfu (mgr.14164) 39 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:12:30.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:12:29 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:12:29.204480+0000 mon.smithi029 (mon.0) 136 : audit [DBG] from='client.? 172.21.15.73:0/820243333' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:12:30.634 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:12:30.634 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:12:31.776 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:12:31 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:12:30.628542+0000 mgr.smithi029.loxrfu (mgr.14164) 40 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:12:31.777 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:31.775+0000 7f5bad3b4700 1 -- 172.21.15.73:0/3532322533 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ba80f47c0 msgr2=0x7f5ba80f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:12:31.777 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:31.775+0000 7f5bad3b4700 1 --2- 172.21.15.73:0/3532322533 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ba80f47c0 0x7f5ba80f4be0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f5ba0009960 tx=0x7f5ba0004d10).stop 2023-12-01T22:12:31.777 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:31.775+0000 7f5bad3b4700 1 -- 172.21.15.73:0/3532322533 shutdown_connections 2023-12-01T22:12:31.777 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:31.775+0000 7f5bad3b4700 1 --2- 172.21.15.73:0/3532322533 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ba80f47c0 0x7f5ba80f4be0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:12:31.777 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:31.775+0000 7f5bad3b4700 1 -- 172.21.15.73:0/3532322533 >> 172.21.15.73:0/3532322533 conn(0x7f5ba80f02e0 msgr2=0x7f5ba80f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:12:31.778 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:31.775+0000 7f5bad3b4700 1 -- 172.21.15.73:0/3532322533 shutdown_connections 2023-12-01T22:12:31.778 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:31.775+0000 7f5bad3b4700 1 -- 172.21.15.73:0/3532322533 wait complete. 2023-12-01T22:12:31.779 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:31.776+0000 7f5bad3b4700 1 Processor -- start 2023-12-01T22:12:31.779 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:31.776+0000 7f5bad3b4700 1 -- start start 2023-12-01T22:12:31.780 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:31.776+0000 7f5bad3b4700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ba80f47c0 0x7f5ba80f7370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:12:31.780 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:31.776+0000 7f5bad3b4700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ba80f57f0 con 0x7f5ba80f47c0 2023-12-01T22:12:31.780 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:31.776+0000 7f5ba6ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ba80f47c0 0x7f5ba80f7370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:12:31.780 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:31.776+0000 7f5ba6ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ba80f47c0 0x7f5ba80f7370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:54646/0 (socket says 172.21.15.73:54646) 2023-12-01T22:12:31.780 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:31.776+0000 7f5ba6ffd700 1 -- 172.21.15.73:0/1287198079 learned_addr learned my addr 172.21.15.73:0/1287198079 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:12:31.780 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:31.777+0000 7f5ba6ffd700 1 -- 172.21.15.73:0/1287198079 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ba0009640 con 0x7f5ba80f47c0 2023-12-01T22:12:31.780 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:31.777+0000 7f5ba6ffd700 1 --2- 172.21.15.73:0/1287198079 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ba80f47c0 0x7f5ba80f7370 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f5ba0004b70 tx=0x7f5ba0004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:12:31.780 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:31.777+0000 7f5b9f7fe700 1 -- 172.21.15.73:0/1287198079 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5ba0018070 con 0x7f5ba80f47c0 2023-12-01T22:12:31.780 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:31.777+0000 7f5bad3b4700 1 -- 172.21.15.73:0/1287198079 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ba80f5990 con 0x7f5ba80f47c0 2023-12-01T22:12:31.780 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:31.777+0000 7f5b9f7fe700 1 -- 172.21.15.73:0/1287198079 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5ba000b870 con 0x7f5ba80f47c0 2023-12-01T22:12:31.780 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:31.777+0000 7f5b9f7fe700 1 -- 172.21.15.73:0/1287198079 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5ba001c560 con 0x7f5ba80f47c0 2023-12-01T22:12:31.780 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:31.777+0000 7f5bad3b4700 1 -- 172.21.15.73:0/1287198079 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ba80f5e20 con 0x7f5ba80f47c0 2023-12-01T22:12:31.781 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:31.778+0000 7f5b9f7fe700 1 -- 172.21.15.73:0/1287198079 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5ba0013630 con 0x7f5ba80f47c0 2023-12-01T22:12:31.781 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:31.778+0000 7f5bad3b4700 1 -- 172.21.15.73:0/1287198079 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ba8059100 con 0x7f5ba80f47c0 2023-12-01T22:12:31.781 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:31.778+0000 7f5b9f7fe700 1 --2- 172.21.15.73:0/1287198079 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f5b9402ff60 0x7f5b94032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:12:31.781 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:31.778+0000 7f5b9f7fe700 1 -- 172.21.15.73:0/1287198079 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5ba0041290 con 0x7f5ba80f47c0 2023-12-01T22:12:31.781 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:31.778+0000 7f5ba67fc700 1 --2- 172.21.15.73:0/1287198079 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f5b9402ff60 0x7f5b94032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:12:31.781 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:31.778+0000 7f5ba67fc700 1 --2- 172.21.15.73:0/1287198079 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f5b9402ff60 0x7f5b94032420 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7f5b900096a0 tx=0x7f5b90006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:12:31.784 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:31.783+0000 7f5b9f7fe700 1 -- 172.21.15.73:0/1287198079 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5ba0004810 con 0x7f5ba80f47c0 2023-12-01T22:12:31.991 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:31.989+0000 7f5bad3b4700 1 -- 172.21.15.73:0/1287198079 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5ba8045ad0 con 0x7f5ba80f47c0 2023-12-01T22:12:31.991 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:31.989+0000 7f5b9f7fe700 1 -- 172.21.15.73:0/1287198079 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5ba0004e10 con 0x7f5ba80f47c0 2023-12-01T22:12:31.992 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:12:31.992 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:12:31.993 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:31.991+0000 7f5bad3b4700 1 -- 172.21.15.73:0/1287198079 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f5b9402ff60 msgr2=0x7f5b94032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:12:31.993 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:31.991+0000 7f5bad3b4700 1 --2- 172.21.15.73:0/1287198079 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f5b9402ff60 0x7f5b94032420 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7f5b900096a0 tx=0x7f5b90006b40).stop 2023-12-01T22:12:31.993 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:31.991+0000 7f5bad3b4700 1 -- 172.21.15.73:0/1287198079 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ba80f47c0 msgr2=0x7f5ba80f7370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:12:31.993 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:31.991+0000 7f5bad3b4700 1 --2- 172.21.15.73:0/1287198079 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ba80f47c0 0x7f5ba80f7370 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f5ba0004b70 tx=0x7f5ba0004d10).stop 2023-12-01T22:12:31.993 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:31.991+0000 7f5bad3b4700 1 -- 172.21.15.73:0/1287198079 shutdown_connections 2023-12-01T22:12:31.993 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:31.991+0000 7f5bad3b4700 1 --2- 172.21.15.73:0/1287198079 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f5b9402ff60 0x7f5b94032420 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:12:31.993 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:31.991+0000 7f5bad3b4700 1 --2- 172.21.15.73:0/1287198079 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ba80f47c0 0x7f5ba80f7370 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:12:31.993 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:31.991+0000 7f5bad3b4700 1 -- 172.21.15.73:0/1287198079 >> 172.21.15.73:0/1287198079 conn(0x7f5ba80f02e0 msgr2=0x7f5ba80f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:12:31.993 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:31.991+0000 7f5bad3b4700 1 -- 172.21.15.73:0/1287198079 shutdown_connections 2023-12-01T22:12:31.994 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:31.991+0000 7f5bad3b4700 1 -- 172.21.15.73:0/1287198079 wait complete. 2023-12-01T22:12:31.994 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:12:32.776 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:12:32 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:12:31.990914+0000 mon.smithi029 (mon.0) 137 : audit [DBG] from='client.? 172.21.15.73:0/1287198079' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:12:33.378 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:12:33.379 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:12:33.776 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:12:33 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:12:32.628895+0000 mgr.smithi029.loxrfu (mgr.14164) 41 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:12:34.519 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:34.517+0000 7f23600b0700 1 -- 172.21.15.73:0/882872353 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f23580f5410 msgr2=0x7f23580f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:12:34.520 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:34.517+0000 7f23600b0700 1 --2- 172.21.15.73:0/882872353 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f23580f5410 0x7f23580f5830 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f2348009fb0 tx=0x7f2348009b30).stop 2023-12-01T22:12:34.520 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:34.517+0000 7f23600b0700 1 -- 172.21.15.73:0/882872353 shutdown_connections 2023-12-01T22:12:34.520 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:34.517+0000 7f23600b0700 1 --2- 172.21.15.73:0/882872353 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f23580f5410 0x7f23580f5830 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:12:34.520 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:34.517+0000 7f23600b0700 1 -- 172.21.15.73:0/882872353 >> 172.21.15.73:0/882872353 conn(0x7f23580f0b70 msgr2=0x7f23580f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:12:34.520 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:34.517+0000 7f23600b0700 1 -- 172.21.15.73:0/882872353 shutdown_connections 2023-12-01T22:12:34.520 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:34.517+0000 7f23600b0700 1 -- 172.21.15.73:0/882872353 wait complete. 2023-12-01T22:12:34.520 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:34.518+0000 7f23600b0700 1 Processor -- start 2023-12-01T22:12:34.520 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:34.518+0000 7f23600b0700 1 -- start start 2023-12-01T22:12:34.520 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:34.518+0000 7f23600b0700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f23580f5410 0x7f2358109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:12:34.520 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:34.518+0000 7f23600b0700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2358060240 con 0x7f23580f5410 2023-12-01T22:12:34.522 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:34.518+0000 7f235de4c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f23580f5410 0x7f2358109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:12:34.522 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:34.518+0000 7f235de4c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f23580f5410 0x7f2358109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:46368/0 (socket says 172.21.15.73:46368) 2023-12-01T22:12:34.522 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:34.518+0000 7f235de4c700 1 -- 172.21.15.73:0/2126109298 learned_addr learned my addr 172.21.15.73:0/2126109298 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:12:34.522 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:34.519+0000 7f235de4c700 1 -- 172.21.15.73:0/2126109298 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2348005040 con 0x7f23580f5410 2023-12-01T22:12:34.522 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:34.519+0000 7f235de4c700 1 --2- 172.21.15.73:0/2126109298 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f23580f5410 0x7f2358109430 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f2348014000 tx=0x7f234800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:12:34.522 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:34.519+0000 7f234e7fc700 1 -- 172.21.15.73:0/2126109298 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f23480095f0 con 0x7f23580f5410 2023-12-01T22:12:34.522 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:34.519+0000 7f23600b0700 1 -- 172.21.15.73:0/2126109298 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2358060440 con 0x7f23580f5410 2023-12-01T22:12:34.522 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:34.519+0000 7f234e7fc700 1 -- 172.21.15.73:0/2126109298 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2348007e70 con 0x7f23580f5410 2023-12-01T22:12:34.523 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:34.519+0000 7f234e7fc700 1 -- 172.21.15.73:0/2126109298 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2348014ea0 con 0x7f23580f5410 2023-12-01T22:12:34.523 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:34.519+0000 7f23600b0700 1 -- 172.21.15.73:0/2126109298 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2358109ae0 con 0x7f23580f5410 2023-12-01T22:12:34.523 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:34.520+0000 7f234e7fc700 1 -- 172.21.15.73:0/2126109298 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2348025070 con 0x7f23580f5410 2023-12-01T22:12:34.523 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:34.520+0000 7f23600b0700 1 -- 172.21.15.73:0/2126109298 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2358045ad0 con 0x7f23580f5410 2023-12-01T22:12:34.523 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:34.520+0000 7f234e7fc700 1 --2- 172.21.15.73:0/2126109298 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f234402ff60 0x7f2344032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:12:34.523 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:34.520+0000 7f234e7fc700 1 -- 172.21.15.73:0/2126109298 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2348040970 con 0x7f23580f5410 2023-12-01T22:12:34.523 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:34.520+0000 7f235d64b700 1 --2- 172.21.15.73:0/2126109298 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f234402ff60 0x7f2344032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:12:34.523 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:34.520+0000 7f235d64b700 1 --2- 172.21.15.73:0/2126109298 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f234402ff60 0x7f2344032420 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7f23540096a0 tx=0x7f2354006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:12:34.526 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:34.524+0000 7f234e7fc700 1 -- 172.21.15.73:0/2126109298 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f23480079c0 con 0x7f23580f5410 2023-12-01T22:12:34.732 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:34.730+0000 7f23600b0700 1 -- 172.21.15.73:0/2126109298 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f23580fae60 con 0x7f23580f5410 2023-12-01T22:12:34.733 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:34.731+0000 7f234e7fc700 1 -- 172.21.15.73:0/2126109298 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f234800f300 con 0x7f23580f5410 2023-12-01T22:12:34.733 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:12:34.733 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:12:34.734 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:34.733+0000 7f23600b0700 1 -- 172.21.15.73:0/2126109298 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f234402ff60 msgr2=0x7f2344032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:12:34.735 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:34.733+0000 7f23600b0700 1 --2- 172.21.15.73:0/2126109298 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f234402ff60 0x7f2344032420 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7f23540096a0 tx=0x7f2354006b40).stop 2023-12-01T22:12:34.735 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:34.733+0000 7f23600b0700 1 -- 172.21.15.73:0/2126109298 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f23580f5410 msgr2=0x7f2358109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:12:34.735 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:34.733+0000 7f23600b0700 1 --2- 172.21.15.73:0/2126109298 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f23580f5410 0x7f2358109430 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f2348014000 tx=0x7f234800b040).stop 2023-12-01T22:12:34.735 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:34.733+0000 7f23600b0700 1 -- 172.21.15.73:0/2126109298 shutdown_connections 2023-12-01T22:12:34.735 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:34.733+0000 7f23600b0700 1 --2- 172.21.15.73:0/2126109298 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f234402ff60 0x7f2344032420 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:12:34.735 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:34.733+0000 7f23600b0700 1 --2- 172.21.15.73:0/2126109298 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f23580f5410 0x7f2358109430 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:12:34.735 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:34.733+0000 7f23600b0700 1 -- 172.21.15.73:0/2126109298 >> 172.21.15.73:0/2126109298 conn(0x7f23580f0b70 msgr2=0x7f23580f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:12:34.735 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:34.733+0000 7f23600b0700 1 -- 172.21.15.73:0/2126109298 shutdown_connections 2023-12-01T22:12:34.735 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:34.733+0000 7f23600b0700 1 -- 172.21.15.73:0/2126109298 wait complete. 2023-12-01T22:12:34.736 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:12:36.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:12:35 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:12:34.629245+0000 mgr.smithi029.loxrfu (mgr.14164) 42 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:12:36.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:12:35 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:12:34.732629+0000 mon.smithi029 (mon.0) 138 : audit [DBG] from='client.? 172.21.15.73:0/2126109298' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:12:36.132 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:12:36.133 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:12:37.336 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:37.334+0000 7f45d2089700 1 -- 172.21.15.73:0/2324043340 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45cc0f6510 msgr2=0x7f45cc0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:12:37.336 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:37.334+0000 7f45d2089700 1 --2- 172.21.15.73:0/2324043340 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45cc0f6510 0x7f45cc0f6930 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f45bc009fb0 tx=0x7f45bc009b30).stop 2023-12-01T22:12:37.336 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:37.334+0000 7f45d2089700 1 -- 172.21.15.73:0/2324043340 shutdown_connections 2023-12-01T22:12:37.336 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:37.334+0000 7f45d2089700 1 --2- 172.21.15.73:0/2324043340 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45cc0f6510 0x7f45cc0f6930 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:12:37.336 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:37.334+0000 7f45d2089700 1 -- 172.21.15.73:0/2324043340 >> 172.21.15.73:0/2324043340 conn(0x7f45cc0f1c70 msgr2=0x7f45cc0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:12:37.336 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:37.334+0000 7f45d2089700 1 -- 172.21.15.73:0/2324043340 shutdown_connections 2023-12-01T22:12:37.336 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:37.334+0000 7f45d2089700 1 -- 172.21.15.73:0/2324043340 wait complete. 2023-12-01T22:12:37.336 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:37.334+0000 7f45d2089700 1 Processor -- start 2023-12-01T22:12:37.337 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:37.334+0000 7f45d2089700 1 -- start start 2023-12-01T22:12:37.338 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:37.335+0000 7f45d2089700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45cc0f6510 0x7f45cc106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:12:37.338 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:37.335+0000 7f45d2089700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45cc106c20 con 0x7f45cc0f6510 2023-12-01T22:12:37.338 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:37.335+0000 7f45cb7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45cc0f6510 0x7f45cc106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:12:37.338 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:37.335+0000 7f45cb7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45cc0f6510 0x7f45cc106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:46378/0 (socket says 172.21.15.73:46378) 2023-12-01T22:12:37.338 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:37.335+0000 7f45cb7fe700 1 -- 172.21.15.73:0/2538946767 learned_addr learned my addr 172.21.15.73:0/2538946767 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:12:37.338 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:37.335+0000 7f45cb7fe700 1 -- 172.21.15.73:0/2538946767 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45bc005040 con 0x7f45cc0f6510 2023-12-01T22:12:37.338 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:37.335+0000 7f45cb7fe700 1 --2- 172.21.15.73:0/2538946767 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45cc0f6510 0x7f45cc106730 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f45bc013510 tx=0x7f45bc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:12:37.338 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:37.335+0000 7f45b3fff700 1 -- 172.21.15.73:0/2538946767 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f45bc009320 con 0x7f45cc0f6510 2023-12-01T22:12:37.338 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:37.335+0000 7f45d2089700 1 -- 172.21.15.73:0/2538946767 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45cc106e20 con 0x7f45cc0f6510 2023-12-01T22:12:37.338 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:37.336+0000 7f45d2089700 1 -- 172.21.15.73:0/2538946767 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45cc1072b0 con 0x7f45cc0f6510 2023-12-01T22:12:37.338 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:37.336+0000 7f45b3fff700 1 -- 172.21.15.73:0/2538946767 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f45bc013f00 con 0x7f45cc0f6510 2023-12-01T22:12:37.338 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:37.336+0000 7f45b3fff700 1 -- 172.21.15.73:0/2538946767 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f45bc014d60 con 0x7f45cc0f6510 2023-12-01T22:12:37.339 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:37.336+0000 7f45b3fff700 1 -- 172.21.15.73:0/2538946767 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f45bc0263a0 con 0x7f45cc0f6510 2023-12-01T22:12:37.339 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:37.336+0000 7f45d2089700 1 -- 172.21.15.73:0/2538946767 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45ac004fa0 con 0x7f45cc0f6510 2023-12-01T22:12:37.339 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:37.336+0000 7f45b3fff700 1 --2- 172.21.15.73:0/2538946767 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f45b402ff60 0x7f45b4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:12:37.339 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:37.337+0000 7f45b3fff700 1 -- 172.21.15.73:0/2538946767 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f45bc041230 con 0x7f45cc0f6510 2023-12-01T22:12:37.340 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:37.337+0000 7f45caffd700 1 --2- 172.21.15.73:0/2538946767 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f45b402ff60 0x7f45b4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:12:37.340 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:37.337+0000 7f45caffd700 1 --2- 172.21.15.73:0/2538946767 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f45b402ff60 0x7f45b4032420 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f45c00096a0 tx=0x7f45c0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:12:37.343 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:37.341+0000 7f45b3fff700 1 -- 172.21.15.73:0/2538946767 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f45bc012920 con 0x7f45cc0f6510 2023-12-01T22:12:37.548 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:37.546+0000 7f45d2089700 1 -- 172.21.15.73:0/2538946767 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f45ac005670 con 0x7f45cc0f6510 2023-12-01T22:12:37.548 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:37.547+0000 7f45b3fff700 1 -- 172.21.15.73:0/2538946767 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f45bc012330 con 0x7f45cc0f6510 2023-12-01T22:12:37.549 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:12:37.549 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:12:37.550 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:37.549+0000 7f45d2089700 1 -- 172.21.15.73:0/2538946767 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f45b402ff60 msgr2=0x7f45b4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:12:37.550 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:37.549+0000 7f45d2089700 1 --2- 172.21.15.73:0/2538946767 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f45b402ff60 0x7f45b4032420 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f45c00096a0 tx=0x7f45c0006b40).stop 2023-12-01T22:12:37.550 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:37.549+0000 7f45d2089700 1 -- 172.21.15.73:0/2538946767 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45cc0f6510 msgr2=0x7f45cc106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:12:37.551 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:37.549+0000 7f45d2089700 1 --2- 172.21.15.73:0/2538946767 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45cc0f6510 0x7f45cc106730 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f45bc013510 tx=0x7f45bc00b040).stop 2023-12-01T22:12:37.551 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:37.549+0000 7f45d2089700 1 -- 172.21.15.73:0/2538946767 shutdown_connections 2023-12-01T22:12:37.551 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:37.549+0000 7f45d2089700 1 --2- 172.21.15.73:0/2538946767 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f45b402ff60 0x7f45b4032420 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:12:37.551 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:37.549+0000 7f45d2089700 1 --2- 172.21.15.73:0/2538946767 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45cc0f6510 0x7f45cc106730 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:12:37.551 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:37.549+0000 7f45d2089700 1 -- 172.21.15.73:0/2538946767 >> 172.21.15.73:0/2538946767 conn(0x7f45cc0f1c70 msgr2=0x7f45cc0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:12:37.551 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:37.549+0000 7f45d2089700 1 -- 172.21.15.73:0/2538946767 shutdown_connections 2023-12-01T22:12:37.551 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:37.549+0000 7f45d2089700 1 -- 172.21.15.73:0/2538946767 wait complete. 2023-12-01T22:12:37.551 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:12:38.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:12:37 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:12:36.629576+0000 mgr.smithi029.loxrfu (mgr.14164) 43 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:12:38.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:12:37 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:12:37.548286+0000 mon.smithi029 (mon.0) 139 : audit [DBG] from='client.? 172.21.15.73:0/2538946767' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:12:38.950 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:12:38.950 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:12:40.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:12:39 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:12:38.629945+0000 mgr.smithi029.loxrfu (mgr.14164) 44 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:12:40.181 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:40.179+0000 7f2fe4c37700 1 -- 172.21.15.73:0/2778932886 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2fe00f62a0 msgr2=0x7f2fe00f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:12:40.181 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:40.179+0000 7f2fe4c37700 1 --2- 172.21.15.73:0/2778932886 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2fe00f62a0 0x7f2fe00f66c0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f2fd4009fb0 tx=0x7f2fd4009b30).stop 2023-12-01T22:12:40.181 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:40.179+0000 7f2fe4c37700 1 -- 172.21.15.73:0/2778932886 shutdown_connections 2023-12-01T22:12:40.181 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:40.179+0000 7f2fe4c37700 1 --2- 172.21.15.73:0/2778932886 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2fe00f62a0 0x7f2fe00f66c0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:12:40.181 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:40.179+0000 7f2fe4c37700 1 -- 172.21.15.73:0/2778932886 >> 172.21.15.73:0/2778932886 conn(0x7f2fe00f1ac0 msgr2=0x7f2fe00f3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:12:40.181 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:40.179+0000 7f2fe4c37700 1 -- 172.21.15.73:0/2778932886 shutdown_connections 2023-12-01T22:12:40.182 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:40.179+0000 7f2fe4c37700 1 -- 172.21.15.73:0/2778932886 wait complete. 2023-12-01T22:12:40.183 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:40.179+0000 7f2fe4c37700 1 Processor -- start 2023-12-01T22:12:40.183 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:40.180+0000 7f2fe4c37700 1 -- start start 2023-12-01T22:12:40.183 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:40.180+0000 7f2fe4c37700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2fe00f62a0 0x7f2fe01064a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:12:40.183 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:40.180+0000 7f2fe4c37700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fe0106990 con 0x7f2fe00f62a0 2023-12-01T22:12:40.184 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:40.180+0000 7f2fdf7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2fe00f62a0 0x7f2fe01064a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:12:40.184 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:40.180+0000 7f2fdf7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2fe00f62a0 0x7f2fe01064a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:46398/0 (socket says 172.21.15.73:46398) 2023-12-01T22:12:40.184 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:40.180+0000 7f2fdf7fe700 1 -- 172.21.15.73:0/227480874 learned_addr learned my addr 172.21.15.73:0/227480874 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:12:40.184 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:40.181+0000 7f2fdf7fe700 1 -- 172.21.15.73:0/227480874 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fd4005040 con 0x7f2fe00f62a0 2023-12-01T22:12:40.184 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:40.181+0000 7f2fdf7fe700 1 --2- 172.21.15.73:0/227480874 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2fe00f62a0 0x7f2fe01064a0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f2fd4013510 tx=0x7f2fd400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:12:40.184 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:40.181+0000 7f2fc7fff700 1 -- 172.21.15.73:0/227480874 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2fd4009320 con 0x7f2fe00f62a0 2023-12-01T22:12:40.184 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:40.181+0000 7f2fe4c37700 1 -- 172.21.15.73:0/227480874 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fe0106b90 con 0x7f2fe00f62a0 2023-12-01T22:12:40.184 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:40.181+0000 7f2fc7fff700 1 -- 172.21.15.73:0/227480874 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2fd4013f00 con 0x7f2fe00f62a0 2023-12-01T22:12:40.184 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:40.181+0000 7f2fc7fff700 1 -- 172.21.15.73:0/227480874 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2fd4014dc0 con 0x7f2fe00f62a0 2023-12-01T22:12:40.185 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:40.181+0000 7f2fe4c37700 1 -- 172.21.15.73:0/227480874 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fe0107020 con 0x7f2fe00f62a0 2023-12-01T22:12:40.185 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:40.182+0000 7f2fc7fff700 1 -- 172.21.15.73:0/227480874 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2fd40263a0 con 0x7f2fe00f62a0 2023-12-01T22:12:40.186 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:40.182+0000 7f2fe4c37700 1 -- 172.21.15.73:0/227480874 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fe0045ad0 con 0x7f2fe00f62a0 2023-12-01T22:12:40.186 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:40.182+0000 7f2fc7fff700 1 --2- 172.21.15.73:0/227480874 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f2fc802ffb0 0x7f2fc8032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:12:40.186 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:40.182+0000 7f2fc7fff700 1 -- 172.21.15.73:0/227480874 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2fd40412f0 con 0x7f2fe00f62a0 2023-12-01T22:12:40.186 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:40.182+0000 7f2fdeffd700 1 --2- 172.21.15.73:0/227480874 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f2fc802ffb0 0x7f2fc8032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:12:40.186 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:40.182+0000 7f2fdeffd700 1 --2- 172.21.15.73:0/227480874 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f2fc802ffb0 0x7f2fc8032470 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f2fd00096a0 tx=0x7f2fd0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:12:40.188 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:40.186+0000 7f2fc7fff700 1 -- 172.21.15.73:0/227480874 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2fd401cad0 con 0x7f2fe00f62a0 2023-12-01T22:12:40.392 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:40.390+0000 7f2fe4c37700 1 -- 172.21.15.73:0/227480874 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2fe0107ed0 con 0x7f2fe00f62a0 2023-12-01T22:12:40.393 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:40.391+0000 7f2fc7fff700 1 -- 172.21.15.73:0/227480874 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2fd401cad0 con 0x7f2fe00f62a0 2023-12-01T22:12:40.393 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:12:40.393 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:12:40.395 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:40.392+0000 7f2fe4c37700 1 -- 172.21.15.73:0/227480874 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f2fc802ffb0 msgr2=0x7f2fc8032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:12:40.395 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:40.392+0000 7f2fe4c37700 1 --2- 172.21.15.73:0/227480874 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f2fc802ffb0 0x7f2fc8032470 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f2fd00096a0 tx=0x7f2fd0006b40).stop 2023-12-01T22:12:40.395 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:40.393+0000 7f2fe4c37700 1 -- 172.21.15.73:0/227480874 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2fe00f62a0 msgr2=0x7f2fe01064a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:12:40.395 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:40.393+0000 7f2fe4c37700 1 --2- 172.21.15.73:0/227480874 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2fe00f62a0 0x7f2fe01064a0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f2fd4013510 tx=0x7f2fd400b040).stop 2023-12-01T22:12:40.395 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:40.393+0000 7f2fe4c37700 1 -- 172.21.15.73:0/227480874 shutdown_connections 2023-12-01T22:12:40.395 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:40.393+0000 7f2fe4c37700 1 --2- 172.21.15.73:0/227480874 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f2fc802ffb0 0x7f2fc8032470 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:12:40.395 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:40.393+0000 7f2fe4c37700 1 --2- 172.21.15.73:0/227480874 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2fe00f62a0 0x7f2fe01064a0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:12:40.395 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:40.393+0000 7f2fe4c37700 1 -- 172.21.15.73:0/227480874 >> 172.21.15.73:0/227480874 conn(0x7f2fe00f1ac0 msgr2=0x7f2fe00f2560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:12:40.395 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:40.393+0000 7f2fe4c37700 1 -- 172.21.15.73:0/227480874 shutdown_connections 2023-12-01T22:12:40.395 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:40.393+0000 7f2fe4c37700 1 -- 172.21.15.73:0/227480874 wait complete. 2023-12-01T22:12:40.396 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:12:41.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:12:40 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:12:40.392229+0000 mon.smithi029 (mon.0) 140 : audit [DBG] from='client.? 172.21.15.73:0/227480874' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:12:41.777 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:12:41.778 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:12:42.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:12:41 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:12:40.630337+0000 mgr.smithi029.loxrfu (mgr.14164) 45 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:12:42.959 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:42.957+0000 7fece6a2c700 1 -- 172.21.15.73:0/3542349244 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fece00f5410 msgr2=0x7fece00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:12:42.959 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:42.957+0000 7fece6a2c700 1 --2- 172.21.15.73:0/3542349244 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fece00f5410 0x7fece00f5830 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7fecd0009fb0 tx=0x7fecd0009b30).stop 2023-12-01T22:12:42.959 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:42.957+0000 7fece6a2c700 1 -- 172.21.15.73:0/3542349244 shutdown_connections 2023-12-01T22:12:42.959 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:42.957+0000 7fece6a2c700 1 --2- 172.21.15.73:0/3542349244 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fece00f5410 0x7fece00f5830 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:12:42.959 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:42.957+0000 7fece6a2c700 1 -- 172.21.15.73:0/3542349244 >> 172.21.15.73:0/3542349244 conn(0x7fece00f0b70 msgr2=0x7fece00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:12:42.959 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:42.957+0000 7fece6a2c700 1 -- 172.21.15.73:0/3542349244 shutdown_connections 2023-12-01T22:12:42.959 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:42.957+0000 7fece6a2c700 1 -- 172.21.15.73:0/3542349244 wait complete. 2023-12-01T22:12:42.960 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:42.958+0000 7fece6a2c700 1 Processor -- start 2023-12-01T22:12:42.961 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:42.958+0000 7fece6a2c700 1 -- start start 2023-12-01T22:12:42.961 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:42.958+0000 7fece6a2c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fece00f5410 0x7fece0109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:12:42.961 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:42.958+0000 7fece6a2c700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fece0060220 con 0x7fece00f5410 2023-12-01T22:12:42.961 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:42.958+0000 7fecdffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fece00f5410 0x7fece0109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:12:42.961 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:42.958+0000 7fecdffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fece00f5410 0x7fece0109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:46418/0 (socket says 172.21.15.73:46418) 2023-12-01T22:12:42.961 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:42.958+0000 7fecdffff700 1 -- 172.21.15.73:0/3903118293 learned_addr learned my addr 172.21.15.73:0/3903118293 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:12:42.961 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:42.959+0000 7fecdffff700 1 -- 172.21.15.73:0/3903118293 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecd0005040 con 0x7fece00f5410 2023-12-01T22:12:42.961 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:42.959+0000 7fecdffff700 1 --2- 172.21.15.73:0/3903118293 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fece00f5410 0x7fece0109450 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7fecd0013510 tx=0x7fecd000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:12:42.961 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:42.959+0000 7fecdcff9700 1 -- 172.21.15.73:0/3903118293 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fecd0009320 con 0x7fece00f5410 2023-12-01T22:12:42.961 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:42.959+0000 7fece6a2c700 1 -- 172.21.15.73:0/3903118293 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fece0060420 con 0x7fece00f5410 2023-12-01T22:12:42.961 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:42.959+0000 7fecdcff9700 1 -- 172.21.15.73:0/3903118293 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fecd0013f00 con 0x7fece00f5410 2023-12-01T22:12:42.961 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:42.959+0000 7fecdcff9700 1 -- 172.21.15.73:0/3903118293 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fecd0014d90 con 0x7fece00f5410 2023-12-01T22:12:42.961 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:42.959+0000 7fece6a2c700 1 -- 172.21.15.73:0/3903118293 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fece0109b00 con 0x7fece00f5410 2023-12-01T22:12:42.962 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:42.960+0000 7fecdcff9700 1 -- 172.21.15.73:0/3903118293 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fecd0014420 con 0x7fece00f5410 2023-12-01T22:12:42.962 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:42.960+0000 7fece6a2c700 1 -- 172.21.15.73:0/3903118293 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fece0059100 con 0x7fece00f5410 2023-12-01T22:12:42.962 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:42.960+0000 7fecdcff9700 1 --2- 172.21.15.73:0/3903118293 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fecc802ff60 0x7fecc8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:12:42.962 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:42.960+0000 7fecdcff9700 1 -- 172.21.15.73:0/3903118293 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fecd00410c0 con 0x7fece00f5410 2023-12-01T22:12:42.963 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:42.960+0000 7fecdf7fe700 1 --2- 172.21.15.73:0/3903118293 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fecc802ff60 0x7fecc8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:12:42.963 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:42.960+0000 7fecdf7fe700 1 --2- 172.21.15.73:0/3903118293 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fecc802ff60 0x7fecc8032420 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7fecd40096a0 tx=0x7fecd4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:12:42.966 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:42.965+0000 7fecdcff9700 1 -- 172.21.15.73:0/3903118293 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fecd00095f0 con 0x7fece00f5410 2023-12-01T22:12:43.172 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:43.170+0000 7fece6a2c700 1 -- 172.21.15.73:0/3903118293 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fece0045ad0 con 0x7fece00f5410 2023-12-01T22:12:43.172 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:43.171+0000 7fecdcff9700 1 -- 172.21.15.73:0/3903118293 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fecd001db70 con 0x7fece00f5410 2023-12-01T22:12:43.173 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:12:43.173 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:12:43.174 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:43.172+0000 7fece6a2c700 1 -- 172.21.15.73:0/3903118293 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fecc802ff60 msgr2=0x7fecc8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:12:43.174 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:43.172+0000 7fece6a2c700 1 --2- 172.21.15.73:0/3903118293 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fecc802ff60 0x7fecc8032420 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7fecd40096a0 tx=0x7fecd4006b40).stop 2023-12-01T22:12:43.174 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:43.172+0000 7fece6a2c700 1 -- 172.21.15.73:0/3903118293 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fece00f5410 msgr2=0x7fece0109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:12:43.174 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:43.172+0000 7fece6a2c700 1 --2- 172.21.15.73:0/3903118293 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fece00f5410 0x7fece0109450 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7fecd0013510 tx=0x7fecd000b040).stop 2023-12-01T22:12:43.174 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:43.172+0000 7fece6a2c700 1 -- 172.21.15.73:0/3903118293 shutdown_connections 2023-12-01T22:12:43.175 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:43.172+0000 7fece6a2c700 1 --2- 172.21.15.73:0/3903118293 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fecc802ff60 0x7fecc8032420 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:12:43.175 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:43.172+0000 7fece6a2c700 1 --2- 172.21.15.73:0/3903118293 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fece00f5410 0x7fece0109450 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:12:43.175 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:43.172+0000 7fece6a2c700 1 -- 172.21.15.73:0/3903118293 >> 172.21.15.73:0/3903118293 conn(0x7fece00f0b70 msgr2=0x7fece00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:12:43.175 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:43.173+0000 7fece6a2c700 1 -- 172.21.15.73:0/3903118293 shutdown_connections 2023-12-01T22:12:43.175 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:43.173+0000 7fece6a2c700 1 -- 172.21.15.73:0/3903118293 wait complete. 2023-12-01T22:12:43.175 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:12:44.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:12:43 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:12:42.630672+0000 mgr.smithi029.loxrfu (mgr.14164) 46 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:12:44.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:12:43 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:12:43.172313+0000 mon.smithi029 (mon.0) 141 : audit [DBG] from='client.? 172.21.15.73:0/3903118293' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:12:44.569 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:12:44.569 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:12:45.797 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:45.793+0000 7fe3969bf700 1 -- 172.21.15.73:0/1301000765 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe390060220 msgr2=0x7fe3900f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:12:45.797 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:45.793+0000 7fe3969bf700 1 --2- 172.21.15.73:0/1301000765 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe390060220 0x7fe3900f9150 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7fe380009fb0 tx=0x7fe380009b30).stop 2023-12-01T22:12:45.797 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:45.793+0000 7fe3969bf700 1 -- 172.21.15.73:0/1301000765 shutdown_connections 2023-12-01T22:12:45.797 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:45.793+0000 7fe3969bf700 1 --2- 172.21.15.73:0/1301000765 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe390060220 0x7fe3900f9150 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:12:45.797 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:45.793+0000 7fe3969bf700 1 -- 172.21.15.73:0/1301000765 >> 172.21.15.73:0/1301000765 conn(0x7fe3900f0b70 msgr2=0x7fe3900f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:12:45.797 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:45.794+0000 7fe3969bf700 1 -- 172.21.15.73:0/1301000765 shutdown_connections 2023-12-01T22:12:45.797 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:45.794+0000 7fe3969bf700 1 -- 172.21.15.73:0/1301000765 wait complete. 2023-12-01T22:12:45.797 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:45.794+0000 7fe3969bf700 1 Processor -- start 2023-12-01T22:12:45.797 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:45.794+0000 7fe3969bf700 1 -- start start 2023-12-01T22:12:45.797 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:45.794+0000 7fe3969bf700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe390060220 0x7fe3901066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:12:45.797 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:45.794+0000 7fe3969bf700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe390106bd0 con 0x7fe390060220 2023-12-01T22:12:45.797 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:45.794+0000 7fe38ffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe390060220 0x7fe3901066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:12:45.798 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:45.795+0000 7fe38ffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe390060220 0x7fe3901066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:35772/0 (socket says 172.21.15.73:35772) 2023-12-01T22:12:45.798 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:45.795+0000 7fe38ffff700 1 -- 172.21.15.73:0/3608154302 learned_addr learned my addr 172.21.15.73:0/3608154302 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:12:45.798 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:45.795+0000 7fe38ffff700 1 -- 172.21.15.73:0/3608154302 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe380005040 con 0x7fe390060220 2023-12-01T22:12:45.798 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:45.795+0000 7fe38ffff700 1 --2- 172.21.15.73:0/3608154302 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe390060220 0x7fe3901066e0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7fe380013510 tx=0x7fe38000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:12:45.798 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:45.795+0000 7fe38cff9700 1 -- 172.21.15.73:0/3608154302 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe380009320 con 0x7fe390060220 2023-12-01T22:12:45.798 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:45.795+0000 7fe3969bf700 1 -- 172.21.15.73:0/3608154302 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe390106dd0 con 0x7fe390060220 2023-12-01T22:12:45.799 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:45.795+0000 7fe38cff9700 1 -- 172.21.15.73:0/3608154302 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe380013f00 con 0x7fe390060220 2023-12-01T22:12:45.799 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:45.795+0000 7fe38cff9700 1 -- 172.21.15.73:0/3608154302 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe380014dc0 con 0x7fe390060220 2023-12-01T22:12:45.800 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:45.795+0000 7fe3969bf700 1 -- 172.21.15.73:0/3608154302 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe390107260 con 0x7fe390060220 2023-12-01T22:12:45.800 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:45.796+0000 7fe38cff9700 1 -- 172.21.15.73:0/3608154302 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe3800263a0 con 0x7fe390060220 2023-12-01T22:12:45.800 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:45.796+0000 7fe3969bf700 1 -- 172.21.15.73:0/3608154302 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe390045ad0 con 0x7fe390060220 2023-12-01T22:12:45.800 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:45.796+0000 7fe38cff9700 1 --2- 172.21.15.73:0/3608154302 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fe37802ffb0 0x7fe378032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:12:45.800 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:45.796+0000 7fe38cff9700 1 -- 172.21.15.73:0/3608154302 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe380041340 con 0x7fe390060220 2023-12-01T22:12:45.800 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:45.796+0000 7fe38f7fe700 1 --2- 172.21.15.73:0/3608154302 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fe37802ffb0 0x7fe378032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:12:45.800 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:45.797+0000 7fe38f7fe700 1 --2- 172.21.15.73:0/3608154302 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fe37802ffb0 0x7fe378032470 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7fe3840096a0 tx=0x7fe384006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:12:45.802 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:45.801+0000 7fe38cff9700 1 -- 172.21.15.73:0/3608154302 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe380014600 con 0x7fe390060220 2023-12-01T22:12:46.006 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:46.004+0000 7fe3969bf700 1 -- 172.21.15.73:0/3608154302 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe390108110 con 0x7fe390060220 2023-12-01T22:12:46.007 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:46.006+0000 7fe38cff9700 1 -- 172.21.15.73:0/3608154302 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe380014420 con 0x7fe390060220 2023-12-01T22:12:46.007 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:12:46.007 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:12:46.008 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:46.007+0000 7fe3969bf700 1 -- 172.21.15.73:0/3608154302 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fe37802ffb0 msgr2=0x7fe378032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:12:46.008 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:46.007+0000 7fe3969bf700 1 --2- 172.21.15.73:0/3608154302 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fe37802ffb0 0x7fe378032470 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7fe3840096a0 tx=0x7fe384006b40).stop 2023-12-01T22:12:46.009 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:46.007+0000 7fe3969bf700 1 -- 172.21.15.73:0/3608154302 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe390060220 msgr2=0x7fe3901066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:12:46.009 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:46.007+0000 7fe3969bf700 1 --2- 172.21.15.73:0/3608154302 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe390060220 0x7fe3901066e0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7fe380013510 tx=0x7fe38000b040).stop 2023-12-01T22:12:46.009 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:46.007+0000 7fe3969bf700 1 -- 172.21.15.73:0/3608154302 shutdown_connections 2023-12-01T22:12:46.009 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:46.007+0000 7fe3969bf700 1 --2- 172.21.15.73:0/3608154302 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fe37802ffb0 0x7fe378032470 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:12:46.009 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:46.007+0000 7fe3969bf700 1 --2- 172.21.15.73:0/3608154302 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe390060220 0x7fe3901066e0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:12:46.009 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:46.007+0000 7fe3969bf700 1 -- 172.21.15.73:0/3608154302 >> 172.21.15.73:0/3608154302 conn(0x7fe3900f0b70 msgr2=0x7fe3900f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:12:46.009 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:46.007+0000 7fe3969bf700 1 -- 172.21.15.73:0/3608154302 shutdown_connections 2023-12-01T22:12:46.009 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:46.007+0000 7fe3969bf700 1 -- 172.21.15.73:0/3608154302 wait complete. 2023-12-01T22:12:46.009 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:12:46.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:12:45 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:12:44.630988+0000 mgr.smithi029.loxrfu (mgr.14164) 47 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:12:47.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:12:46 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:12:46.006289+0000 mon.smithi029 (mon.0) 142 : audit [DBG] from='client.? 172.21.15.73:0/3608154302' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:12:47.416 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:12:47.417 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:12:48.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:12:47 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:12:46.631242+0000 mgr.smithi029.loxrfu (mgr.14164) 48 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:12:48.592 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:48.588+0000 7f5ba9248700 1 -- 172.21.15.73:0/1189565821 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ba40f5410 msgr2=0x7f5ba40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:12:48.592 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:48.588+0000 7f5ba9248700 1 --2- 172.21.15.73:0/1189565821 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ba40f5410 0x7f5ba40f5830 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f5b8c009fb0 tx=0x7f5b8c009b30).stop 2023-12-01T22:12:48.592 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:48.589+0000 7f5ba9248700 1 -- 172.21.15.73:0/1189565821 shutdown_connections 2023-12-01T22:12:48.592 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:48.589+0000 7f5ba9248700 1 --2- 172.21.15.73:0/1189565821 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ba40f5410 0x7f5ba40f5830 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:12:48.592 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:48.589+0000 7f5ba9248700 1 -- 172.21.15.73:0/1189565821 >> 172.21.15.73:0/1189565821 conn(0x7f5ba40f0b70 msgr2=0x7f5ba40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:12:48.592 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:48.589+0000 7f5ba9248700 1 -- 172.21.15.73:0/1189565821 shutdown_connections 2023-12-01T22:12:48.592 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:48.589+0000 7f5ba9248700 1 -- 172.21.15.73:0/1189565821 wait complete. 2023-12-01T22:12:48.593 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:48.589+0000 7f5ba9248700 1 Processor -- start 2023-12-01T22:12:48.593 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:48.589+0000 7f5ba9248700 1 -- start start 2023-12-01T22:12:48.593 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:48.589+0000 7f5ba9248700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ba40f5410 0x7f5ba4109420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:12:48.593 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:48.589+0000 7f5ba9248700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ba4060220 con 0x7f5ba40f5410 2023-12-01T22:12:48.593 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:48.590+0000 7f5ba2d9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ba40f5410 0x7f5ba4109420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:12:48.593 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:48.590+0000 7f5ba2d9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ba40f5410 0x7f5ba4109420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:35786/0 (socket says 172.21.15.73:35786) 2023-12-01T22:12:48.593 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:48.590+0000 7f5ba2d9d700 1 -- 172.21.15.73:0/4246660885 learned_addr learned my addr 172.21.15.73:0/4246660885 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:12:48.593 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:48.590+0000 7f5ba2d9d700 1 -- 172.21.15.73:0/4246660885 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b8c005040 con 0x7f5ba40f5410 2023-12-01T22:12:48.593 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:48.590+0000 7f5ba2d9d700 1 --2- 172.21.15.73:0/4246660885 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ba40f5410 0x7f5ba4109420 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f5b8c014000 tx=0x7f5b8c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:12:48.593 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:48.590+0000 7f5b9b7fe700 1 -- 172.21.15.73:0/4246660885 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5b8c0095f0 con 0x7f5ba40f5410 2023-12-01T22:12:48.593 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:48.590+0000 7f5ba9248700 1 -- 172.21.15.73:0/4246660885 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ba4060420 con 0x7f5ba40f5410 2023-12-01T22:12:48.593 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:48.590+0000 7f5b9b7fe700 1 -- 172.21.15.73:0/4246660885 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5b8c007e70 con 0x7f5ba40f5410 2023-12-01T22:12:48.593 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:48.590+0000 7f5b9b7fe700 1 -- 172.21.15.73:0/4246660885 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5b8c014ea0 con 0x7f5ba40f5410 2023-12-01T22:12:48.593 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:48.590+0000 7f5ba9248700 1 -- 172.21.15.73:0/4246660885 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ba4109ad0 con 0x7f5ba40f5410 2023-12-01T22:12:48.594 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:48.591+0000 7f5b9b7fe700 1 -- 172.21.15.73:0/4246660885 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5b8c0079c0 con 0x7f5ba40f5410 2023-12-01T22:12:48.595 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:48.591+0000 7f5ba9248700 1 -- 172.21.15.73:0/4246660885 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ba4059100 con 0x7f5ba40f5410 2023-12-01T22:12:48.595 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:48.591+0000 7f5b9b7fe700 1 --2- 172.21.15.73:0/4246660885 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f5b9002ff60 0x7f5b90032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:12:48.595 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:48.591+0000 7f5b9b7fe700 1 -- 172.21.15.73:0/4246660885 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5b8c040860 con 0x7f5ba40f5410 2023-12-01T22:12:48.595 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:48.591+0000 7f5ba259c700 1 --2- 172.21.15.73:0/4246660885 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f5b9002ff60 0x7f5b90032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:12:48.595 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:48.592+0000 7f5ba259c700 1 --2- 172.21.15.73:0/4246660885 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f5b9002ff60 0x7f5b90032420 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f5b940096a0 tx=0x7f5b94006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:12:48.597 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:48.596+0000 7f5b9b7fe700 1 -- 172.21.15.73:0/4246660885 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5b8c00f950 con 0x7f5ba40f5410 2023-12-01T22:12:48.804 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:48.803+0000 7f5ba9248700 1 -- 172.21.15.73:0/4246660885 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5ba4045ad0 con 0x7f5ba40f5410 2023-12-01T22:12:48.805 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:48.804+0000 7f5b9b7fe700 1 -- 172.21.15.73:0/4246660885 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5b8c010230 con 0x7f5ba40f5410 2023-12-01T22:12:48.806 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:12:48.806 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:12:48.807 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:48.805+0000 7f5ba9248700 1 -- 172.21.15.73:0/4246660885 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f5b9002ff60 msgr2=0x7f5b90032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:12:48.807 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:48.805+0000 7f5ba9248700 1 --2- 172.21.15.73:0/4246660885 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f5b9002ff60 0x7f5b90032420 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f5b940096a0 tx=0x7f5b94006b40).stop 2023-12-01T22:12:48.807 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:48.805+0000 7f5ba9248700 1 -- 172.21.15.73:0/4246660885 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ba40f5410 msgr2=0x7f5ba4109420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:12:48.807 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:48.805+0000 7f5ba9248700 1 --2- 172.21.15.73:0/4246660885 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ba40f5410 0x7f5ba4109420 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f5b8c014000 tx=0x7f5b8c00b040).stop 2023-12-01T22:12:48.807 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:48.805+0000 7f5ba9248700 1 -- 172.21.15.73:0/4246660885 shutdown_connections 2023-12-01T22:12:48.807 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:48.805+0000 7f5ba9248700 1 --2- 172.21.15.73:0/4246660885 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f5b9002ff60 0x7f5b90032420 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:12:48.807 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:48.805+0000 7f5ba9248700 1 --2- 172.21.15.73:0/4246660885 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ba40f5410 0x7f5ba4109420 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:12:48.808 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:48.805+0000 7f5ba9248700 1 -- 172.21.15.73:0/4246660885 >> 172.21.15.73:0/4246660885 conn(0x7f5ba40f0b70 msgr2=0x7f5ba40f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:12:48.808 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:48.805+0000 7f5ba9248700 1 -- 172.21.15.73:0/4246660885 shutdown_connections 2023-12-01T22:12:48.808 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:48.805+0000 7f5ba9248700 1 -- 172.21.15.73:0/4246660885 wait complete. 2023-12-01T22:12:48.808 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:12:50.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:12:49 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:12:48.631584+0000 mgr.smithi029.loxrfu (mgr.14164) 49 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:12:50.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:12:49 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:12:48.804930+0000 mon.smithi029 (mon.0) 143 : audit [DBG] from='client.? 172.21.15.73:0/4246660885' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:12:50.206 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:12:50.206 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:12:51.382 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:51.379+0000 7f199c788700 1 -- 172.21.15.73:0/206708389 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f19940f6510 msgr2=0x7f19940f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:12:51.383 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:51.379+0000 7f199c788700 1 --2- 172.21.15.73:0/206708389 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f19940f6510 0x7f19940f6930 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f1984009fb0 tx=0x7f1984009b30).stop 2023-12-01T22:12:51.383 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:51.379+0000 7f199c788700 1 -- 172.21.15.73:0/206708389 shutdown_connections 2023-12-01T22:12:51.383 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:51.379+0000 7f199c788700 1 --2- 172.21.15.73:0/206708389 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f19940f6510 0x7f19940f6930 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:12:51.383 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:51.379+0000 7f199c788700 1 -- 172.21.15.73:0/206708389 >> 172.21.15.73:0/206708389 conn(0x7f19940f1c70 msgr2=0x7f19940f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:12:51.383 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:51.379+0000 7f199c788700 1 -- 172.21.15.73:0/206708389 shutdown_connections 2023-12-01T22:12:51.383 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:51.379+0000 7f199c788700 1 -- 172.21.15.73:0/206708389 wait complete. 2023-12-01T22:12:51.383 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:51.380+0000 7f199c788700 1 Processor -- start 2023-12-01T22:12:51.383 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:51.380+0000 7f199c788700 1 -- start start 2023-12-01T22:12:51.383 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:51.380+0000 7f199c788700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f19940f6510 0x7f199406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:12:51.383 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:51.380+0000 7f199c788700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1994069e20 con 0x7f19940f6510 2023-12-01T22:12:51.383 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:51.380+0000 7f199a524700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f19940f6510 0x7f199406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:12:51.383 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:51.380+0000 7f199a524700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f19940f6510 0x7f199406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:35796/0 (socket says 172.21.15.73:35796) 2023-12-01T22:12:51.384 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:51.380+0000 7f199a524700 1 -- 172.21.15.73:0/2250331106 learned_addr learned my addr 172.21.15.73:0/2250331106 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:12:51.384 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:51.380+0000 7f199a524700 1 -- 172.21.15.73:0/2250331106 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1984005040 con 0x7f19940f6510 2023-12-01T22:12:51.384 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:51.381+0000 7f199a524700 1 --2- 172.21.15.73:0/2250331106 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f19940f6510 0x7f199406b9a0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f1984013510 tx=0x7f198400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:12:51.384 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:51.381+0000 7f198affd700 1 -- 172.21.15.73:0/2250331106 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1984009320 con 0x7f19940f6510 2023-12-01T22:12:51.384 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:51.381+0000 7f198affd700 1 -- 172.21.15.73:0/2250331106 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1984013f00 con 0x7f19940f6510 2023-12-01T22:12:51.384 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:51.381+0000 7f198affd700 1 -- 172.21.15.73:0/2250331106 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1984014d90 con 0x7f19940f6510 2023-12-01T22:12:51.384 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:51.381+0000 7f199c788700 1 -- 172.21.15.73:0/2250331106 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1994069fc0 con 0x7f19940f6510 2023-12-01T22:12:51.384 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:51.381+0000 7f199c788700 1 -- 172.21.15.73:0/2250331106 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f199406a450 con 0x7f19940f6510 2023-12-01T22:12:51.385 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:51.382+0000 7f198affd700 1 -- 172.21.15.73:0/2250331106 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f19840095f0 con 0x7f19940f6510 2023-12-01T22:12:51.385 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:51.382+0000 7f199c788700 1 -- 172.21.15.73:0/2250331106 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1994059100 con 0x7f19940f6510 2023-12-01T22:12:51.385 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:51.382+0000 7f198affd700 1 --2- 172.21.15.73:0/2250331106 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f198002ffb0 0x7f1980032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:12:51.385 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:51.382+0000 7f198affd700 1 -- 172.21.15.73:0/2250331106 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1984040f90 con 0x7f19940f6510 2023-12-01T22:12:51.385 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:51.382+0000 7f1999d23700 1 --2- 172.21.15.73:0/2250331106 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f198002ffb0 0x7f1980032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:12:51.385 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:51.382+0000 7f1999d23700 1 --2- 172.21.15.73:0/2250331106 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f198002ffb0 0x7f1980032470 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f19900096a0 tx=0x7f1990006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:12:51.388 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:51.387+0000 7f198affd700 1 -- 172.21.15.73:0/2250331106 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1984014420 con 0x7f19940f6510 2023-12-01T22:12:51.595 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:51.593+0000 7f199c788700 1 -- 172.21.15.73:0/2250331106 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f19940fbf60 con 0x7f19940f6510 2023-12-01T22:12:51.595 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:51.595+0000 7f198affd700 1 -- 172.21.15.73:0/2250331106 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f198401d7f0 con 0x7f19940f6510 2023-12-01T22:12:51.596 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:12:51.596 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:12:51.597 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:51.595+0000 7f199c788700 1 -- 172.21.15.73:0/2250331106 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f198002ffb0 msgr2=0x7f1980032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:12:51.597 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:51.595+0000 7f199c788700 1 --2- 172.21.15.73:0/2250331106 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f198002ffb0 0x7f1980032470 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f19900096a0 tx=0x7f1990006b40).stop 2023-12-01T22:12:51.597 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:51.595+0000 7f199c788700 1 -- 172.21.15.73:0/2250331106 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f19940f6510 msgr2=0x7f199406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:12:51.598 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:51.595+0000 7f199c788700 1 --2- 172.21.15.73:0/2250331106 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f19940f6510 0x7f199406b9a0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f1984013510 tx=0x7f198400b040).stop 2023-12-01T22:12:51.598 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:51.595+0000 7f199c788700 1 -- 172.21.15.73:0/2250331106 shutdown_connections 2023-12-01T22:12:51.598 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:51.595+0000 7f199c788700 1 --2- 172.21.15.73:0/2250331106 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f198002ffb0 0x7f1980032470 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:12:51.598 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:51.596+0000 7f199c788700 1 --2- 172.21.15.73:0/2250331106 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f19940f6510 0x7f199406b9a0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:12:51.598 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:51.596+0000 7f199c788700 1 -- 172.21.15.73:0/2250331106 >> 172.21.15.73:0/2250331106 conn(0x7f19940f1c70 msgr2=0x7f19940f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:12:51.598 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:51.596+0000 7f199c788700 1 -- 172.21.15.73:0/2250331106 shutdown_connections 2023-12-01T22:12:51.598 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:51.596+0000 7f199c788700 1 -- 172.21.15.73:0/2250331106 wait complete. 2023-12-01T22:12:51.598 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:12:51.776 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:12:51 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:12:50.631876+0000 mgr.smithi029.loxrfu (mgr.14164) 50 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:12:52.776 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:12:52 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:12:51.595114+0000 mon.smithi029 (mon.0) 144 : audit [DBG] from='client.? 172.21.15.73:0/2250331106' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:12:52.994 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:12:52.994 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:12:53.776 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:12:53 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:12:52.632140+0000 mgr.smithi029.loxrfu (mgr.14164) 51 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:12:54.209 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:54.205+0000 7f32aa579700 1 -- 172.21.15.73:0/968864349 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32a40f51a0 msgr2=0x7f32a40f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:12:54.209 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:54.205+0000 7f32aa579700 1 --2- 172.21.15.73:0/968864349 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32a40f51a0 0x7f32a40f55c0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f32a0009fb0 tx=0x7f32a0009b30).stop 2023-12-01T22:12:54.209 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:54.205+0000 7f32aa579700 1 -- 172.21.15.73:0/968864349 shutdown_connections 2023-12-01T22:12:54.209 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:54.205+0000 7f32aa579700 1 --2- 172.21.15.73:0/968864349 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32a40f51a0 0x7f32a40f55c0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:12:54.209 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:54.205+0000 7f32aa579700 1 -- 172.21.15.73:0/968864349 >> 172.21.15.73:0/968864349 conn(0x7f32a40f09a0 msgr2=0x7f32a40f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:12:54.209 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:54.205+0000 7f32aa579700 1 -- 172.21.15.73:0/968864349 shutdown_connections 2023-12-01T22:12:54.210 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:54.205+0000 7f32aa579700 1 -- 172.21.15.73:0/968864349 wait complete. 2023-12-01T22:12:54.210 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:54.206+0000 7f32aa579700 1 Processor -- start 2023-12-01T22:12:54.210 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:54.206+0000 7f32aa579700 1 -- start start 2023-12-01T22:12:54.210 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:54.206+0000 7f32aa579700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32a40f51a0 0x7f32a40fd120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:12:54.210 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:54.206+0000 7f32aa579700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32a40fb5a0 con 0x7f32a40f51a0 2023-12-01T22:12:54.210 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:54.206+0000 7f32a9577700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32a40f51a0 0x7f32a40fd120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:12:54.210 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:54.206+0000 7f32a9577700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32a40f51a0 0x7f32a40fd120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:35812/0 (socket says 172.21.15.73:35812) 2023-12-01T22:12:54.210 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:54.206+0000 7f32a9577700 1 -- 172.21.15.73:0/97856679 learned_addr learned my addr 172.21.15.73:0/97856679 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:12:54.210 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:54.206+0000 7f32a9577700 1 -- 172.21.15.73:0/97856679 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32a0005040 con 0x7f32a40f51a0 2023-12-01T22:12:54.210 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:54.207+0000 7f32a9577700 1 --2- 172.21.15.73:0/97856679 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32a40f51a0 0x7f32a40fd120 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f32a0013510 tx=0x7f32a000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:12:54.210 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:54.207+0000 7f3299ffb700 1 -- 172.21.15.73:0/97856679 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f32a0009320 con 0x7f32a40f51a0 2023-12-01T22:12:54.210 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:54.207+0000 7f3299ffb700 1 -- 172.21.15.73:0/97856679 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f32a0013f00 con 0x7f32a40f51a0 2023-12-01T22:12:54.210 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:54.207+0000 7f32aa579700 1 -- 172.21.15.73:0/97856679 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32a40fb740 con 0x7f32a40f51a0 2023-12-01T22:12:54.211 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:54.207+0000 7f3299ffb700 1 -- 172.21.15.73:0/97856679 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f32a0014dc0 con 0x7f32a40f51a0 2023-12-01T22:12:54.211 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:54.207+0000 7f32aa579700 1 -- 172.21.15.73:0/97856679 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32a40fbbd0 con 0x7f32a40f51a0 2023-12-01T22:12:54.211 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:54.208+0000 7f3299ffb700 1 -- 172.21.15.73:0/97856679 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f32a001d3d0 con 0x7f32a40f51a0 2023-12-01T22:12:54.211 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:54.208+0000 7f3299ffb700 1 --2- 172.21.15.73:0/97856679 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f329002ff60 0x7f3290032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:12:54.212 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:54.208+0000 7f32aa579700 1 -- 172.21.15.73:0/97856679 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32a4059100 con 0x7f32a40f51a0 2023-12-01T22:12:54.212 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:54.208+0000 7f3299ffb700 1 -- 172.21.15.73:0/97856679 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f32a0041100 con 0x7f32a40f51a0 2023-12-01T22:12:54.212 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:54.208+0000 7f32a8d76700 1 --2- 172.21.15.73:0/97856679 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f329002ff60 0x7f3290032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:12:54.212 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:54.209+0000 7f32a8d76700 1 --2- 172.21.15.73:0/97856679 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f329002ff60 0x7f3290032420 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f32940096a0 tx=0x7f3294006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:12:54.214 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:54.214+0000 7f3299ffb700 1 -- 172.21.15.73:0/97856679 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f32a0012920 con 0x7f32a40f51a0 2023-12-01T22:12:54.421 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:54.419+0000 7f32aa579700 1 -- 172.21.15.73:0/97856679 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f32a40fabe0 con 0x7f32a40f51a0 2023-12-01T22:12:54.421 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:54.421+0000 7f3299ffb700 1 -- 172.21.15.73:0/97856679 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f32a003d020 con 0x7f32a40f51a0 2023-12-01T22:12:54.422 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:12:54.422 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:12:54.423 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:54.421+0000 7f32aa579700 1 -- 172.21.15.73:0/97856679 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f329002ff60 msgr2=0x7f3290032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:12:54.424 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:54.421+0000 7f32aa579700 1 --2- 172.21.15.73:0/97856679 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f329002ff60 0x7f3290032420 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f32940096a0 tx=0x7f3294006b40).stop 2023-12-01T22:12:54.424 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:54.421+0000 7f32aa579700 1 -- 172.21.15.73:0/97856679 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32a40f51a0 msgr2=0x7f32a40fd120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:12:54.424 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:54.421+0000 7f32aa579700 1 --2- 172.21.15.73:0/97856679 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32a40f51a0 0x7f32a40fd120 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f32a0013510 tx=0x7f32a000b040).stop 2023-12-01T22:12:54.424 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:54.421+0000 7f32aa579700 1 -- 172.21.15.73:0/97856679 shutdown_connections 2023-12-01T22:12:54.424 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:54.421+0000 7f32aa579700 1 --2- 172.21.15.73:0/97856679 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f329002ff60 0x7f3290032420 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:12:54.424 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:54.421+0000 7f32aa579700 1 --2- 172.21.15.73:0/97856679 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32a40f51a0 0x7f32a40fd120 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:12:54.424 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:54.421+0000 7f32aa579700 1 -- 172.21.15.73:0/97856679 >> 172.21.15.73:0/97856679 conn(0x7f32a40f09a0 msgr2=0x7f32a405fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:12:54.424 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:54.421+0000 7f32aa579700 1 -- 172.21.15.73:0/97856679 shutdown_connections 2023-12-01T22:12:54.424 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:54.421+0000 7f32aa579700 1 -- 172.21.15.73:0/97856679 wait complete. 2023-12-01T22:12:54.424 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:12:54.776 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:12:54 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:12:54.421097+0000 mon.smithi029 (mon.0) 145 : audit [DBG] from='client.? 172.21.15.73:0/97856679' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:12:55.776 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:12:55 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:12:54.632530+0000 mgr.smithi029.loxrfu (mgr.14164) 52 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:12:55.811 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:12:55.811 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:12:56.993 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:56.989+0000 7f93d9962700 1 -- 172.21.15.73:0/1787650613 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93d40f6510 msgr2=0x7f93d40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:12:56.994 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:56.989+0000 7f93d9962700 1 --2- 172.21.15.73:0/1787650613 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93d40f6510 0x7f93d40f6930 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f93c4009fb0 tx=0x7f93c4009b30).stop 2023-12-01T22:12:56.994 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:56.989+0000 7f93d9962700 1 -- 172.21.15.73:0/1787650613 shutdown_connections 2023-12-01T22:12:56.994 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:56.989+0000 7f93d9962700 1 --2- 172.21.15.73:0/1787650613 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93d40f6510 0x7f93d40f6930 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:12:56.994 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:56.989+0000 7f93d9962700 1 -- 172.21.15.73:0/1787650613 >> 172.21.15.73:0/1787650613 conn(0x7f93d40f1c70 msgr2=0x7f93d40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:12:56.994 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:56.989+0000 7f93d9962700 1 -- 172.21.15.73:0/1787650613 shutdown_connections 2023-12-01T22:12:56.994 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:56.989+0000 7f93d9962700 1 -- 172.21.15.73:0/1787650613 wait complete. 2023-12-01T22:12:56.994 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:56.989+0000 7f93d9962700 1 Processor -- start 2023-12-01T22:12:56.994 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:56.989+0000 7f93d9962700 1 -- start start 2023-12-01T22:12:56.994 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:56.990+0000 7f93d9962700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93d40f6510 0x7f93d4106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:12:56.994 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:56.990+0000 7f93d9962700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93d4106c20 con 0x7f93d40f6510 2023-12-01T22:12:56.994 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:56.990+0000 7f93d2ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93d40f6510 0x7f93d4106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:12:56.995 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:56.990+0000 7f93d2ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93d40f6510 0x7f93d4106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:57580/0 (socket says 172.21.15.73:57580) 2023-12-01T22:12:56.995 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:56.990+0000 7f93d2ffd700 1 -- 172.21.15.73:0/3404071833 learned_addr learned my addr 172.21.15.73:0/3404071833 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:12:56.995 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:56.990+0000 7f93d2ffd700 1 -- 172.21.15.73:0/3404071833 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93c4005040 con 0x7f93d40f6510 2023-12-01T22:12:56.995 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:56.990+0000 7f93d2ffd700 1 --2- 172.21.15.73:0/3404071833 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93d40f6510 0x7f93d4106730 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f93c4013510 tx=0x7f93c400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:12:56.995 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:56.990+0000 7f93bb7fe700 1 -- 172.21.15.73:0/3404071833 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f93c4009320 con 0x7f93d40f6510 2023-12-01T22:12:56.995 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:56.991+0000 7f93d9962700 1 -- 172.21.15.73:0/3404071833 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93d4106e20 con 0x7f93d40f6510 2023-12-01T22:12:56.995 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:56.991+0000 7f93bb7fe700 1 -- 172.21.15.73:0/3404071833 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f93c4013f00 con 0x7f93d40f6510 2023-12-01T22:12:56.995 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:56.991+0000 7f93bb7fe700 1 -- 172.21.15.73:0/3404071833 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f93c4014d90 con 0x7f93d40f6510 2023-12-01T22:12:56.995 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:56.991+0000 7f93d9962700 1 -- 172.21.15.73:0/3404071833 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93d41072b0 con 0x7f93d40f6510 2023-12-01T22:12:56.995 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:56.991+0000 7f93bb7fe700 1 -- 172.21.15.73:0/3404071833 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f93c4014420 con 0x7f93d40f6510 2023-12-01T22:12:56.995 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:56.991+0000 7f93d9962700 1 -- 172.21.15.73:0/3404071833 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93d4045ad0 con 0x7f93d40f6510 2023-12-01T22:12:56.995 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:56.992+0000 7f93bb7fe700 1 --2- 172.21.15.73:0/3404071833 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f93bc02ff60 0x7f93bc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:12:56.995 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:56.992+0000 7f93bb7fe700 1 -- 172.21.15.73:0/3404071833 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f93c4040fb0 con 0x7f93d40f6510 2023-12-01T22:12:56.995 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:56.992+0000 7f93d27fc700 1 --2- 172.21.15.73:0/3404071833 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f93bc02ff60 0x7f93bc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:12:56.996 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:56.992+0000 7f93d27fc700 1 --2- 172.21.15.73:0/3404071833 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f93bc02ff60 0x7f93bc032420 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f93c80096a0 tx=0x7f93c8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:12:56.997 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:56.997+0000 7f93bb7fe700 1 -- 172.21.15.73:0/3404071833 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f93c40095f0 con 0x7f93d40f6510 2023-12-01T22:12:57.202 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:57.202+0000 7f93d9962700 1 -- 172.21.15.73:0/3404071833 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f93d40fbf60 con 0x7f93d40f6510 2023-12-01T22:12:57.203 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:57.202+0000 7f93bb7fe700 1 -- 172.21.15.73:0/3404071833 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f93c401db70 con 0x7f93d40f6510 2023-12-01T22:12:57.203 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:12:57.203 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:12:57.205 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:57.204+0000 7f93d9962700 1 -- 172.21.15.73:0/3404071833 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f93bc02ff60 msgr2=0x7f93bc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:12:57.205 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:57.204+0000 7f93d9962700 1 --2- 172.21.15.73:0/3404071833 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f93bc02ff60 0x7f93bc032420 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f93c80096a0 tx=0x7f93c8006b40).stop 2023-12-01T22:12:57.205 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:57.204+0000 7f93d9962700 1 -- 172.21.15.73:0/3404071833 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93d40f6510 msgr2=0x7f93d4106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:12:57.205 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:57.204+0000 7f93d9962700 1 --2- 172.21.15.73:0/3404071833 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93d40f6510 0x7f93d4106730 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f93c4013510 tx=0x7f93c400b040).stop 2023-12-01T22:12:57.205 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:57.204+0000 7f93d9962700 1 -- 172.21.15.73:0/3404071833 shutdown_connections 2023-12-01T22:12:57.205 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:57.204+0000 7f93d9962700 1 --2- 172.21.15.73:0/3404071833 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f93bc02ff60 0x7f93bc032420 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:12:57.205 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:57.204+0000 7f93d9962700 1 --2- 172.21.15.73:0/3404071833 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93d40f6510 0x7f93d4106730 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:12:57.205 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:57.204+0000 7f93d9962700 1 -- 172.21.15.73:0/3404071833 >> 172.21.15.73:0/3404071833 conn(0x7f93d40f1c70 msgr2=0x7f93d40f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:12:57.205 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:57.204+0000 7f93d9962700 1 -- 172.21.15.73:0/3404071833 shutdown_connections 2023-12-01T22:12:57.205 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:57.204+0000 7f93d9962700 1 -- 172.21.15.73:0/3404071833 wait complete. 2023-12-01T22:12:57.206 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:12:58.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:12:57 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:12:56.632907+0000 mgr.smithi029.loxrfu (mgr.14164) 53 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:12:58.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:12:57 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:12:57.202884+0000 mon.smithi029 (mon.0) 146 : audit [DBG] from='client.? 172.21.15.73:0/3404071833' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:12:58.579 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:12:58.579 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:12:59.740 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:59.739+0000 7fe3f6d98700 1 -- 172.21.15.73:0/106665680 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3f0060200 msgr2=0x7fe3f00fb570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:12:59.741 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:59.739+0000 7fe3f6d98700 1 --2- 172.21.15.73:0/106665680 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3f0060200 0x7fe3f00fb570 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7fe3e4009fb0 tx=0x7fe3e4009b30).stop 2023-12-01T22:12:59.741 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:59.739+0000 7fe3f6d98700 1 -- 172.21.15.73:0/106665680 shutdown_connections 2023-12-01T22:12:59.741 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:59.739+0000 7fe3f6d98700 1 --2- 172.21.15.73:0/106665680 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3f0060200 0x7fe3f00fb570 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:12:59.741 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:59.739+0000 7fe3f6d98700 1 -- 172.21.15.73:0/106665680 >> 172.21.15.73:0/106665680 conn(0x7fe3f00f0b70 msgr2=0x7fe3f00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:12:59.741 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:59.739+0000 7fe3f6d98700 1 -- 172.21.15.73:0/106665680 shutdown_connections 2023-12-01T22:12:59.741 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:59.739+0000 7fe3f6d98700 1 -- 172.21.15.73:0/106665680 wait complete. 2023-12-01T22:12:59.741 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:59.740+0000 7fe3f6d98700 1 Processor -- start 2023-12-01T22:12:59.741 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:59.740+0000 7fe3f6d98700 1 -- start start 2023-12-01T22:12:59.742 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:59.740+0000 7fe3f6d98700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3f0060200 0x7fe3f01088d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:12:59.742 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:59.740+0000 7fe3f6d98700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3f0108dc0 con 0x7fe3f0060200 2023-12-01T22:12:59.743 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:59.740+0000 7fe3f4b34700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3f0060200 0x7fe3f01088d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:12:59.743 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:59.740+0000 7fe3f4b34700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3f0060200 0x7fe3f01088d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:57610/0 (socket says 172.21.15.73:57610) 2023-12-01T22:12:59.743 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:59.740+0000 7fe3f4b34700 1 -- 172.21.15.73:0/3609239530 learned_addr learned my addr 172.21.15.73:0/3609239530 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:12:59.743 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:59.741+0000 7fe3f4b34700 1 -- 172.21.15.73:0/3609239530 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3e4005040 con 0x7fe3f0060200 2023-12-01T22:12:59.743 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:59.741+0000 7fe3f4b34700 1 --2- 172.21.15.73:0/3609239530 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3f0060200 0x7fe3f01088d0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7fe3e4013510 tx=0x7fe3e400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:12:59.743 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:59.741+0000 7fe3ed7fa700 1 -- 172.21.15.73:0/3609239530 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe3e4009320 con 0x7fe3f0060200 2023-12-01T22:12:59.743 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:59.741+0000 7fe3f6d98700 1 -- 172.21.15.73:0/3609239530 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3f0108fc0 con 0x7fe3f0060200 2023-12-01T22:12:59.743 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:59.741+0000 7fe3ed7fa700 1 -- 172.21.15.73:0/3609239530 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe3e4013f00 con 0x7fe3f0060200 2023-12-01T22:12:59.743 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:59.741+0000 7fe3ed7fa700 1 -- 172.21.15.73:0/3609239530 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe3e4014dc0 con 0x7fe3f0060200 2023-12-01T22:12:59.743 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:59.741+0000 7fe3f6d98700 1 -- 172.21.15.73:0/3609239530 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3f0109450 con 0x7fe3f0060200 2023-12-01T22:12:59.743 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:59.742+0000 7fe3ed7fa700 1 -- 172.21.15.73:0/3609239530 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe3e40095f0 con 0x7fe3f0060200 2023-12-01T22:12:59.743 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:59.742+0000 7fe3f6d98700 1 -- 172.21.15.73:0/3609239530 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3f0045ad0 con 0x7fe3f0060200 2023-12-01T22:12:59.743 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:59.742+0000 7fe3ed7fa700 1 --2- 172.21.15.73:0/3609239530 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fe3e002ff60 0x7fe3e0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:12:59.743 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:59.742+0000 7fe3ed7fa700 1 -- 172.21.15.73:0/3609239530 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe3e40410c0 con 0x7fe3f0060200 2023-12-01T22:12:59.744 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:59.742+0000 7fe3effff700 1 --2- 172.21.15.73:0/3609239530 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fe3e002ff60 0x7fe3e0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:12:59.744 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:59.743+0000 7fe3effff700 1 --2- 172.21.15.73:0/3609239530 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fe3e002ff60 0x7fe3e0032420 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7fe3dc0096a0 tx=0x7fe3dc006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:12:59.747 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:59.747+0000 7fe3ed7fa700 1 -- 172.21.15.73:0/3609239530 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe3e4014420 con 0x7fe3f0060200 2023-12-01T22:12:59.956 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:59.955+0000 7fe3f6d98700 1 -- 172.21.15.73:0/3609239530 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe3f010a300 con 0x7fe3f0060200 2023-12-01T22:12:59.956 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:59.956+0000 7fe3ed7fa700 1 -- 172.21.15.73:0/3609239530 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe3e4010230 con 0x7fe3f0060200 2023-12-01T22:12:59.957 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:12:59.957 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:12:59.958 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:59.957+0000 7fe3f6d98700 1 -- 172.21.15.73:0/3609239530 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fe3e002ff60 msgr2=0x7fe3e0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:12:59.958 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:59.957+0000 7fe3f6d98700 1 --2- 172.21.15.73:0/3609239530 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fe3e002ff60 0x7fe3e0032420 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7fe3dc0096a0 tx=0x7fe3dc006b40).stop 2023-12-01T22:12:59.958 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:59.957+0000 7fe3f6d98700 1 -- 172.21.15.73:0/3609239530 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3f0060200 msgr2=0x7fe3f01088d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:12:59.958 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:59.957+0000 7fe3f6d98700 1 --2- 172.21.15.73:0/3609239530 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3f0060200 0x7fe3f01088d0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7fe3e4013510 tx=0x7fe3e400b040).stop 2023-12-01T22:12:59.959 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:59.957+0000 7fe3f6d98700 1 -- 172.21.15.73:0/3609239530 shutdown_connections 2023-12-01T22:12:59.959 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:59.957+0000 7fe3f6d98700 1 --2- 172.21.15.73:0/3609239530 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fe3e002ff60 0x7fe3e0032420 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:12:59.959 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:59.957+0000 7fe3f6d98700 1 --2- 172.21.15.73:0/3609239530 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3f0060200 0x7fe3f01088d0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:12:59.959 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:59.958+0000 7fe3f6d98700 1 -- 172.21.15.73:0/3609239530 >> 172.21.15.73:0/3609239530 conn(0x7fe3f00f0b70 msgr2=0x7fe3f00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:12:59.959 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:59.958+0000 7fe3f6d98700 1 -- 172.21.15.73:0/3609239530 shutdown_connections 2023-12-01T22:12:59.959 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:12:59.958+0000 7fe3f6d98700 1 -- 172.21.15.73:0/3609239530 wait complete. 2023-12-01T22:12:59.959 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:13:00.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:12:59 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:12:58.633373+0000 mgr.smithi029.loxrfu (mgr.14164) 54 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:13:01.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:13:00 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:12:59.956183+0000 mon.smithi029 (mon.0) 147 : audit [DBG] from='client.? 172.21.15.73:0/3609239530' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:13:01.350 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:13:01.350 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:13:02.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:13:01 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:13:00.633747+0000 mgr.smithi029.loxrfu (mgr.14164) 55 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:13:02.510 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:02.508+0000 7f816e3a0700 1 -- 172.21.15.73:0/3405158312 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81680f51a0 msgr2=0x7f81680f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:13:02.510 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:02.508+0000 7f816e3a0700 1 --2- 172.21.15.73:0/3405158312 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81680f51a0 0x7f81680f55c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f8164009fb0 tx=0x7f8164009b30).stop 2023-12-01T22:13:02.510 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:02.508+0000 7f816e3a0700 1 -- 172.21.15.73:0/3405158312 shutdown_connections 2023-12-01T22:13:02.510 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:02.508+0000 7f816e3a0700 1 --2- 172.21.15.73:0/3405158312 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81680f51a0 0x7f81680f55c0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:13:02.510 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:02.508+0000 7f816e3a0700 1 -- 172.21.15.73:0/3405158312 >> 172.21.15.73:0/3405158312 conn(0x7f81680f09a0 msgr2=0x7f81680f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:13:02.511 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:02.508+0000 7f816e3a0700 1 -- 172.21.15.73:0/3405158312 shutdown_connections 2023-12-01T22:13:02.511 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:02.508+0000 7f816e3a0700 1 -- 172.21.15.73:0/3405158312 wait complete. 2023-12-01T22:13:02.511 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:02.508+0000 7f816e3a0700 1 Processor -- start 2023-12-01T22:13:02.511 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:02.508+0000 7f816e3a0700 1 -- start start 2023-12-01T22:13:02.511 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:02.509+0000 7f816e3a0700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81680f51a0 0x7f81681053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:13:02.511 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:02.509+0000 7f816e3a0700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81681058a0 con 0x7f81680f51a0 2023-12-01T22:13:02.511 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:02.509+0000 7f816d39e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81680f51a0 0x7f81681053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:13:02.511 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:02.509+0000 7f816d39e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81680f51a0 0x7f81681053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:57630/0 (socket says 172.21.15.73:57630) 2023-12-01T22:13:02.511 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:02.509+0000 7f816d39e700 1 -- 172.21.15.73:0/374989794 learned_addr learned my addr 172.21.15.73:0/374989794 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:13:02.511 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:02.509+0000 7f816d39e700 1 -- 172.21.15.73:0/374989794 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8164005040 con 0x7f81680f51a0 2023-12-01T22:13:02.511 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:02.509+0000 7f816d39e700 1 --2- 172.21.15.73:0/374989794 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81680f51a0 0x7f81681053b0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f8164013510 tx=0x7f816400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:13:02.511 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:02.509+0000 7f815dffb700 1 -- 172.21.15.73:0/374989794 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8164009320 con 0x7f81680f51a0 2023-12-01T22:13:02.511 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:02.509+0000 7f816e3a0700 1 -- 172.21.15.73:0/374989794 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8168105aa0 con 0x7f81680f51a0 2023-12-01T22:13:02.511 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:02.509+0000 7f815dffb700 1 -- 172.21.15.73:0/374989794 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8164013f00 con 0x7f81680f51a0 2023-12-01T22:13:02.512 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:02.509+0000 7f815dffb700 1 -- 172.21.15.73:0/374989794 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8164014dc0 con 0x7f81680f51a0 2023-12-01T22:13:02.512 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:02.510+0000 7f816e3a0700 1 -- 172.21.15.73:0/374989794 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8168105f30 con 0x7f81680f51a0 2023-12-01T22:13:02.512 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:02.510+0000 7f815dffb700 1 -- 172.21.15.73:0/374989794 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f81640095f0 con 0x7f81680f51a0 2023-12-01T22:13:02.513 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:02.510+0000 7f816e3a0700 1 -- 172.21.15.73:0/374989794 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8168045ad0 con 0x7f81680f51a0 2023-12-01T22:13:02.513 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:02.511+0000 7f815dffb700 1 --2- 172.21.15.73:0/374989794 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f815402ffb0 0x7f8154032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:13:02.513 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:02.511+0000 7f815dffb700 1 -- 172.21.15.73:0/374989794 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8164042e90 con 0x7f81680f51a0 2023-12-01T22:13:02.513 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:02.511+0000 7f816cb9d700 1 --2- 172.21.15.73:0/374989794 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f815402ffb0 0x7f8154032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:13:02.513 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:02.511+0000 7f816cb9d700 1 --2- 172.21.15.73:0/374989794 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f815402ffb0 0x7f8154032470 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f81580096a0 tx=0x7f8158006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:13:02.516 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:02.516+0000 7f815dffb700 1 -- 172.21.15.73:0/374989794 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8164014420 con 0x7f81680f51a0 2023-12-01T22:13:02.724 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:02.724+0000 7f816e3a0700 1 -- 172.21.15.73:0/374989794 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8168106d80 con 0x7f81680f51a0 2023-12-01T22:13:02.725 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:02.724+0000 7f815dffb700 1 -- 172.21.15.73:0/374989794 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f816401f980 con 0x7f81680f51a0 2023-12-01T22:13:02.725 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:13:02.725 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:13:02.726 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:02.726+0000 7f816e3a0700 1 -- 172.21.15.73:0/374989794 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f815402ffb0 msgr2=0x7f8154032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:13:02.727 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:02.726+0000 7f816e3a0700 1 --2- 172.21.15.73:0/374989794 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f815402ffb0 0x7f8154032470 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f81580096a0 tx=0x7f8158006b40).stop 2023-12-01T22:13:02.727 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:02.726+0000 7f816e3a0700 1 -- 172.21.15.73:0/374989794 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81680f51a0 msgr2=0x7f81681053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:13:02.727 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:02.726+0000 7f816e3a0700 1 --2- 172.21.15.73:0/374989794 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81680f51a0 0x7f81681053b0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f8164013510 tx=0x7f816400b040).stop 2023-12-01T22:13:02.727 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:02.726+0000 7f816e3a0700 1 -- 172.21.15.73:0/374989794 shutdown_connections 2023-12-01T22:13:02.727 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:02.726+0000 7f816e3a0700 1 --2- 172.21.15.73:0/374989794 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f815402ffb0 0x7f8154032470 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:13:02.727 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:02.726+0000 7f816e3a0700 1 --2- 172.21.15.73:0/374989794 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81680f51a0 0x7f81681053b0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:13:02.727 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:02.726+0000 7f816e3a0700 1 -- 172.21.15.73:0/374989794 >> 172.21.15.73:0/374989794 conn(0x7f81680f09a0 msgr2=0x7f81680f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:13:02.727 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:02.726+0000 7f816e3a0700 1 -- 172.21.15.73:0/374989794 shutdown_connections 2023-12-01T22:13:02.727 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:02.726+0000 7f816e3a0700 1 -- 172.21.15.73:0/374989794 wait complete. 2023-12-01T22:13:02.727 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:13:04.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:13:03 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:13:02.634123+0000 mgr.smithi029.loxrfu (mgr.14164) 56 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:13:04.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:13:03 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:13:02.724737+0000 mon.smithi029 (mon.0) 148 : audit [DBG] from='client.? 172.21.15.73:0/374989794' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:13:04.108 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:13:04.108 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:13:05.261 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:05.259+0000 7fa2a663b700 1 -- 172.21.15.73:0/2670067253 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2a0060280 msgr2=0x7fa2a00f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:13:05.262 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:05.259+0000 7fa2a663b700 1 --2- 172.21.15.73:0/2670067253 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2a0060280 0x7fa2a00f90f0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7fa290009fb0 tx=0x7fa290009b30).stop 2023-12-01T22:13:05.262 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:05.259+0000 7fa2a663b700 1 -- 172.21.15.73:0/2670067253 shutdown_connections 2023-12-01T22:13:05.262 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:05.259+0000 7fa2a663b700 1 --2- 172.21.15.73:0/2670067253 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2a0060280 0x7fa2a00f90f0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:13:05.262 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:05.259+0000 7fa2a663b700 1 -- 172.21.15.73:0/2670067253 >> 172.21.15.73:0/2670067253 conn(0x7fa2a00f0b70 msgr2=0x7fa2a00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:13:05.262 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:05.259+0000 7fa2a663b700 1 -- 172.21.15.73:0/2670067253 shutdown_connections 2023-12-01T22:13:05.262 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:05.259+0000 7fa2a663b700 1 -- 172.21.15.73:0/2670067253 wait complete. 2023-12-01T22:13:05.262 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:05.260+0000 7fa2a663b700 1 Processor -- start 2023-12-01T22:13:05.262 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:05.260+0000 7fa2a663b700 1 -- start start 2023-12-01T22:13:05.262 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:05.260+0000 7fa2a663b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2a0060280 0x7fa2a01066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:13:05.262 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:05.260+0000 7fa2a663b700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2a0106bd0 con 0x7fa2a0060280 2023-12-01T22:13:05.262 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:05.260+0000 7fa29ffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2a0060280 0x7fa2a01066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:13:05.263 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:05.260+0000 7fa29ffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2a0060280 0x7fa2a01066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:53894/0 (socket says 172.21.15.73:53894) 2023-12-01T22:13:05.263 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:05.260+0000 7fa29ffff700 1 -- 172.21.15.73:0/1018469155 learned_addr learned my addr 172.21.15.73:0/1018469155 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:13:05.263 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:05.261+0000 7fa29ffff700 1 -- 172.21.15.73:0/1018469155 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa290005040 con 0x7fa2a0060280 2023-12-01T22:13:05.263 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:05.261+0000 7fa29ffff700 1 --2- 172.21.15.73:0/1018469155 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2a0060280 0x7fa2a01066e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7fa290013510 tx=0x7fa29000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:13:05.263 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:05.261+0000 7fa29cff9700 1 -- 172.21.15.73:0/1018469155 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa290009320 con 0x7fa2a0060280 2023-12-01T22:13:05.263 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:05.261+0000 7fa2a663b700 1 -- 172.21.15.73:0/1018469155 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2a0106dd0 con 0x7fa2a0060280 2023-12-01T22:13:05.263 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:05.261+0000 7fa29cff9700 1 -- 172.21.15.73:0/1018469155 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa290013f00 con 0x7fa2a0060280 2023-12-01T22:13:05.263 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:05.261+0000 7fa29cff9700 1 -- 172.21.15.73:0/1018469155 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa290014dc0 con 0x7fa2a0060280 2023-12-01T22:13:05.263 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:05.261+0000 7fa2a663b700 1 -- 172.21.15.73:0/1018469155 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2a0107260 con 0x7fa2a0060280 2023-12-01T22:13:05.264 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:05.262+0000 7fa29cff9700 1 -- 172.21.15.73:0/1018469155 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa29001d3d0 con 0x7fa2a0060280 2023-12-01T22:13:05.264 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:05.262+0000 7fa2a663b700 1 -- 172.21.15.73:0/1018469155 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2a0045ad0 con 0x7fa2a0060280 2023-12-01T22:13:05.264 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:05.262+0000 7fa29cff9700 1 --2- 172.21.15.73:0/1018469155 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fa28802ff60 0x7fa288032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:13:05.264 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:05.262+0000 7fa29cff9700 1 -- 172.21.15.73:0/1018469155 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa290041300 con 0x7fa2a0060280 2023-12-01T22:13:05.264 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:05.262+0000 7fa29f7fe700 1 --2- 172.21.15.73:0/1018469155 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fa28802ff60 0x7fa288032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:13:05.264 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:05.262+0000 7fa29f7fe700 1 --2- 172.21.15.73:0/1018469155 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fa28802ff60 0x7fa288032420 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7fa2940096a0 tx=0x7fa294006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:13:05.267 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:05.267+0000 7fa29cff9700 1 -- 172.21.15.73:0/1018469155 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa290012330 con 0x7fa2a0060280 2023-12-01T22:13:05.476 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:05.474+0000 7fa2a663b700 1 -- 172.21.15.73:0/1018469155 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa2a0108110 con 0x7fa2a0060280 2023-12-01T22:13:05.476 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:05.475+0000 7fa29cff9700 1 -- 172.21.15.73:0/1018469155 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa290012330 con 0x7fa2a0060280 2023-12-01T22:13:05.477 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:13:05.477 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:13:05.478 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:05.476+0000 7fa2a663b700 1 -- 172.21.15.73:0/1018469155 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fa28802ff60 msgr2=0x7fa288032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:13:05.478 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:05.476+0000 7fa2a663b700 1 --2- 172.21.15.73:0/1018469155 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fa28802ff60 0x7fa288032420 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7fa2940096a0 tx=0x7fa294006b40).stop 2023-12-01T22:13:05.479 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:05.477+0000 7fa2a663b700 1 -- 172.21.15.73:0/1018469155 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2a0060280 msgr2=0x7fa2a01066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:13:05.479 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:05.477+0000 7fa2a663b700 1 --2- 172.21.15.73:0/1018469155 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2a0060280 0x7fa2a01066e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7fa290013510 tx=0x7fa29000b040).stop 2023-12-01T22:13:05.479 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:05.477+0000 7fa2a663b700 1 -- 172.21.15.73:0/1018469155 shutdown_connections 2023-12-01T22:13:05.479 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:05.477+0000 7fa2a663b700 1 --2- 172.21.15.73:0/1018469155 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fa28802ff60 0x7fa288032420 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:13:05.479 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:05.477+0000 7fa2a663b700 1 --2- 172.21.15.73:0/1018469155 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2a0060280 0x7fa2a01066e0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:13:05.479 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:05.477+0000 7fa2a663b700 1 -- 172.21.15.73:0/1018469155 >> 172.21.15.73:0/1018469155 conn(0x7fa2a00f0b70 msgr2=0x7fa2a00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:13:05.479 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:05.477+0000 7fa2a663b700 1 -- 172.21.15.73:0/1018469155 shutdown_connections 2023-12-01T22:13:05.479 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:05.477+0000 7fa2a663b700 1 -- 172.21.15.73:0/1018469155 wait complete. 2023-12-01T22:13:05.479 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:13:06.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:13:05 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:13:04.634542+0000 mgr.smithi029.loxrfu (mgr.14164) 57 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:13:06.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:13:05 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:13:05.476351+0000 mon.smithi029 (mon.0) 149 : audit [DBG] from='client.? 172.21.15.73:0/1018469155' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:13:06.846 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:13:06.846 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:13:08.027 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:08.024+0000 7feb19ee0700 1 -- 172.21.15.73:0/710089334 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb140f5410 msgr2=0x7feb140f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:13:08.027 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:08.024+0000 7feb19ee0700 1 --2- 172.21.15.73:0/710089334 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb140f5410 0x7feb140f5830 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7feb08009fb0 tx=0x7feb08009b30).stop 2023-12-01T22:13:08.027 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:08.024+0000 7feb19ee0700 1 -- 172.21.15.73:0/710089334 shutdown_connections 2023-12-01T22:13:08.027 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:08.024+0000 7feb19ee0700 1 --2- 172.21.15.73:0/710089334 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb140f5410 0x7feb140f5830 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:13:08.027 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:08.024+0000 7feb19ee0700 1 -- 172.21.15.73:0/710089334 >> 172.21.15.73:0/710089334 conn(0x7feb140f0b70 msgr2=0x7feb140f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:13:08.027 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:08.024+0000 7feb19ee0700 1 -- 172.21.15.73:0/710089334 shutdown_connections 2023-12-01T22:13:08.027 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:08.024+0000 7feb19ee0700 1 -- 172.21.15.73:0/710089334 wait complete. 2023-12-01T22:13:08.027 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:08.025+0000 7feb19ee0700 1 Processor -- start 2023-12-01T22:13:08.027 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:08.025+0000 7feb19ee0700 1 -- start start 2023-12-01T22:13:08.028 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:08.025+0000 7feb19ee0700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb140f5410 0x7feb14109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:13:08.028 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:08.025+0000 7feb19ee0700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb14060240 con 0x7feb140f5410 2023-12-01T22:13:08.028 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:08.025+0000 7feb137fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb140f5410 0x7feb14109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:13:08.028 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:08.025+0000 7feb137fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb140f5410 0x7feb14109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:53912/0 (socket says 172.21.15.73:53912) 2023-12-01T22:13:08.028 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:08.025+0000 7feb137fe700 1 -- 172.21.15.73:0/389676746 learned_addr learned my addr 172.21.15.73:0/389676746 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:13:08.028 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:13:07 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:13:06.634986+0000 mgr.smithi029.loxrfu (mgr.14164) 58 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:13:08.029 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:08.026+0000 7feb137fe700 1 -- 172.21.15.73:0/389676746 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb08005040 con 0x7feb140f5410 2023-12-01T22:13:08.030 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:08.026+0000 7feb137fe700 1 --2- 172.21.15.73:0/389676746 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb140f5410 0x7feb14109430 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7feb08013510 tx=0x7feb0800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:13:08.030 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:08.026+0000 7feafbfff700 1 -- 172.21.15.73:0/389676746 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feb08009320 con 0x7feb140f5410 2023-12-01T22:13:08.030 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:08.026+0000 7feafbfff700 1 -- 172.21.15.73:0/389676746 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7feb08013f00 con 0x7feb140f5410 2023-12-01T22:13:08.030 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:08.026+0000 7feb19ee0700 1 -- 172.21.15.73:0/389676746 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb14060440 con 0x7feb140f5410 2023-12-01T22:13:08.030 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:08.026+0000 7feafbfff700 1 -- 172.21.15.73:0/389676746 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feb08014d90 con 0x7feb140f5410 2023-12-01T22:13:08.030 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:08.026+0000 7feb19ee0700 1 -- 172.21.15.73:0/389676746 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb14109ae0 con 0x7feb140f5410 2023-12-01T22:13:08.030 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:08.027+0000 7feafbfff700 1 -- 172.21.15.73:0/389676746 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7feb080263a0 con 0x7feb140f5410 2023-12-01T22:13:08.030 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:08.027+0000 7feafbfff700 1 --2- 172.21.15.73:0/389676746 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7feafc02ff60 0x7feafc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:13:08.030 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:08.027+0000 7feafbfff700 1 -- 172.21.15.73:0/389676746 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7feb08041020 con 0x7feb140f5410 2023-12-01T22:13:08.030 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:08.027+0000 7feb19ee0700 1 -- 172.21.15.73:0/389676746 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb14045ad0 con 0x7feb140f5410 2023-12-01T22:13:08.030 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:08.027+0000 7feb12ffd700 1 --2- 172.21.15.73:0/389676746 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7feafc02ff60 0x7feafc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:13:08.030 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:08.028+0000 7feb12ffd700 1 --2- 172.21.15.73:0/389676746 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7feafc02ff60 0x7feafc032420 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7feb040096a0 tx=0x7feb04006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:13:08.033 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:08.032+0000 7feafbfff700 1 -- 172.21.15.73:0/389676746 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7feb08012920 con 0x7feb140f5410 2023-12-01T22:13:08.239 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:08.237+0000 7feb19ee0700 1 -- 172.21.15.73:0/389676746 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7feb1410a1f0 con 0x7feb140f5410 2023-12-01T22:13:08.240 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:08.238+0000 7feafbfff700 1 -- 172.21.15.73:0/389676746 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7feb0802b020 con 0x7feb140f5410 2023-12-01T22:13:08.240 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:13:08.240 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:13:08.241 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:08.239+0000 7feb19ee0700 1 -- 172.21.15.73:0/389676746 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7feafc02ff60 msgr2=0x7feafc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:13:08.241 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:08.239+0000 7feb19ee0700 1 --2- 172.21.15.73:0/389676746 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7feafc02ff60 0x7feafc032420 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7feb040096a0 tx=0x7feb04006b40).stop 2023-12-01T22:13:08.241 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:08.239+0000 7feb19ee0700 1 -- 172.21.15.73:0/389676746 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb140f5410 msgr2=0x7feb14109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:13:08.241 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:08.239+0000 7feb19ee0700 1 --2- 172.21.15.73:0/389676746 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb140f5410 0x7feb14109430 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7feb08013510 tx=0x7feb0800b040).stop 2023-12-01T22:13:08.242 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:08.240+0000 7feb19ee0700 1 -- 172.21.15.73:0/389676746 shutdown_connections 2023-12-01T22:13:08.242 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:08.240+0000 7feb19ee0700 1 --2- 172.21.15.73:0/389676746 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7feafc02ff60 0x7feafc032420 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:13:08.242 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:08.240+0000 7feb19ee0700 1 --2- 172.21.15.73:0/389676746 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb140f5410 0x7feb14109430 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:13:08.242 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:08.240+0000 7feb19ee0700 1 -- 172.21.15.73:0/389676746 >> 172.21.15.73:0/389676746 conn(0x7feb140f0b70 msgr2=0x7feb140f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:13:08.242 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:08.240+0000 7feb19ee0700 1 -- 172.21.15.73:0/389676746 shutdown_connections 2023-12-01T22:13:08.242 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:08.240+0000 7feb19ee0700 1 -- 172.21.15.73:0/389676746 wait complete. 2023-12-01T22:13:08.242 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:13:09.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:13:08 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:13:08.239147+0000 mon.smithi029 (mon.0) 150 : audit [DBG] from='client.? 172.21.15.73:0/389676746' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:13:09.621 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:13:09.621 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:13:10.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:13:09 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:13:08.635400+0000 mgr.smithi029.loxrfu (mgr.14164) 59 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:13:10.788 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:10.785+0000 7ff54e61a700 1 -- 172.21.15.73:0/1284564945 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff548060220 msgr2=0x7ff5480f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:13:10.788 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:10.785+0000 7ff54e61a700 1 --2- 172.21.15.73:0/1284564945 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff548060220 0x7ff5480f9150 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7ff538009fb0 tx=0x7ff538009b30).stop 2023-12-01T22:13:10.788 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:10.785+0000 7ff54e61a700 1 -- 172.21.15.73:0/1284564945 shutdown_connections 2023-12-01T22:13:10.788 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:10.785+0000 7ff54e61a700 1 --2- 172.21.15.73:0/1284564945 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff548060220 0x7ff5480f9150 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:13:10.788 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:10.785+0000 7ff54e61a700 1 -- 172.21.15.73:0/1284564945 >> 172.21.15.73:0/1284564945 conn(0x7ff5480f0b70 msgr2=0x7ff5480f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:13:10.788 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:10.785+0000 7ff54e61a700 1 -- 172.21.15.73:0/1284564945 shutdown_connections 2023-12-01T22:13:10.789 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:10.785+0000 7ff54e61a700 1 -- 172.21.15.73:0/1284564945 wait complete. 2023-12-01T22:13:10.789 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:10.786+0000 7ff54e61a700 1 Processor -- start 2023-12-01T22:13:10.789 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:10.786+0000 7ff54e61a700 1 -- start start 2023-12-01T22:13:10.789 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:10.786+0000 7ff54e61a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff548060220 0x7ff5480f6bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:13:10.789 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:10.786+0000 7ff54e61a700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5480f5050 con 0x7ff548060220 2023-12-01T22:13:10.789 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:10.786+0000 7ff547fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff548060220 0x7ff5480f6bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:13:10.789 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:10.786+0000 7ff547fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff548060220 0x7ff5480f6bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:53944/0 (socket says 172.21.15.73:53944) 2023-12-01T22:13:10.789 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:10.786+0000 7ff547fff700 1 -- 172.21.15.73:0/1269413915 learned_addr learned my addr 172.21.15.73:0/1269413915 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:13:10.789 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:10.786+0000 7ff547fff700 1 -- 172.21.15.73:0/1269413915 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff538005040 con 0x7ff548060220 2023-12-01T22:13:10.789 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:10.787+0000 7ff547fff700 1 --2- 172.21.15.73:0/1269413915 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff548060220 0x7ff5480f6bd0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7ff538013510 tx=0x7ff53800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:13:10.790 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:10.787+0000 7ff544ff9700 1 -- 172.21.15.73:0/1269413915 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff538009320 con 0x7ff548060220 2023-12-01T22:13:10.791 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:10.787+0000 7ff54e61a700 1 -- 172.21.15.73:0/1269413915 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5480f51f0 con 0x7ff548060220 2023-12-01T22:13:10.791 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:10.787+0000 7ff544ff9700 1 -- 172.21.15.73:0/1269413915 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff538013f00 con 0x7ff548060220 2023-12-01T22:13:10.791 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:10.787+0000 7ff544ff9700 1 -- 172.21.15.73:0/1269413915 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff538014dc0 con 0x7ff548060220 2023-12-01T22:13:10.791 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:10.787+0000 7ff54e61a700 1 -- 172.21.15.73:0/1269413915 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5480f5680 con 0x7ff548060220 2023-12-01T22:13:10.791 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:10.788+0000 7ff544ff9700 1 -- 172.21.15.73:0/1269413915 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff53801d3d0 con 0x7ff548060220 2023-12-01T22:13:10.791 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:10.788+0000 7ff544ff9700 1 --2- 172.21.15.73:0/1269413915 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7ff53002ff60 0x7ff530032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:13:10.791 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:10.788+0000 7ff544ff9700 1 -- 172.21.15.73:0/1269413915 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff538041000 con 0x7ff548060220 2023-12-01T22:13:10.791 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:10.788+0000 7ff54e61a700 1 -- 172.21.15.73:0/1269413915 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff548059100 con 0x7ff548060220 2023-12-01T22:13:10.791 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:10.788+0000 7ff5477fe700 1 --2- 172.21.15.73:0/1269413915 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7ff53002ff60 0x7ff530032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:13:10.791 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:10.788+0000 7ff5477fe700 1 --2- 172.21.15.73:0/1269413915 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7ff53002ff60 0x7ff530032420 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7ff53c0096a0 tx=0x7ff53c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:13:10.794 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:10.793+0000 7ff544ff9700 1 -- 172.21.15.73:0/1269413915 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff538012900 con 0x7ff548060220 2023-12-01T22:13:11.003 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:11.000+0000 7ff54e61a700 1 -- 172.21.15.73:0/1269413915 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff5480f5d80 con 0x7ff548060220 2023-12-01T22:13:11.004 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:11.000+0000 7ff544ff9700 1 -- 172.21.15.73:0/1269413915 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff53802b020 con 0x7ff548060220 2023-12-01T22:13:11.004 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:13:11.005 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:13:11.006 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:11.002+0000 7ff54e61a700 1 -- 172.21.15.73:0/1269413915 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7ff53002ff60 msgr2=0x7ff530032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:13:11.006 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:11.002+0000 7ff54e61a700 1 --2- 172.21.15.73:0/1269413915 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7ff53002ff60 0x7ff530032420 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7ff53c0096a0 tx=0x7ff53c006b40).stop 2023-12-01T22:13:11.007 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:11.002+0000 7ff54e61a700 1 -- 172.21.15.73:0/1269413915 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff548060220 msgr2=0x7ff5480f6bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:13:11.007 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:11.002+0000 7ff54e61a700 1 --2- 172.21.15.73:0/1269413915 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff548060220 0x7ff5480f6bd0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7ff538013510 tx=0x7ff53800b040).stop 2023-12-01T22:13:11.007 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:11.002+0000 7ff54e61a700 1 -- 172.21.15.73:0/1269413915 shutdown_connections 2023-12-01T22:13:11.007 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:11.002+0000 7ff54e61a700 1 --2- 172.21.15.73:0/1269413915 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7ff53002ff60 0x7ff530032420 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:13:11.007 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:11.002+0000 7ff54e61a700 1 --2- 172.21.15.73:0/1269413915 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff548060220 0x7ff5480f6bd0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:13:11.008 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:11.002+0000 7ff54e61a700 1 -- 172.21.15.73:0/1269413915 >> 172.21.15.73:0/1269413915 conn(0x7ff5480f0b70 msgr2=0x7ff5480f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:13:11.008 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:11.002+0000 7ff54e61a700 1 -- 172.21.15.73:0/1269413915 shutdown_connections 2023-12-01T22:13:11.008 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:11.002+0000 7ff54e61a700 1 -- 172.21.15.73:0/1269413915 wait complete. 2023-12-01T22:13:11.008 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:13:11.777 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:13:11 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:13:10.635685+0000 mgr.smithi029.loxrfu (mgr.14164) 60 : cluster [DBG] pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:13:11.777 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:13:11 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:13:11.001487+0000 mon.smithi029 (mon.0) 151 : audit [DBG] from='client.? 172.21.15.73:0/1269413915' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:13:12.390 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:13:12.390 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:13:13.505 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:13.503+0000 7f94283bc700 1 -- 172.21.15.73:0/4124694516 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9420060220 msgr2=0x7f94200f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:13:13.506 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:13.503+0000 7f94283bc700 1 --2- 172.21.15.73:0/4124694516 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9420060220 0x7f94200f9150 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f9410009fb0 tx=0x7f9410009b30).stop 2023-12-01T22:13:13.506 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:13.503+0000 7f94283bc700 1 -- 172.21.15.73:0/4124694516 shutdown_connections 2023-12-01T22:13:13.506 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:13.503+0000 7f94283bc700 1 --2- 172.21.15.73:0/4124694516 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9420060220 0x7f94200f9150 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:13:13.506 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:13.503+0000 7f94283bc700 1 -- 172.21.15.73:0/4124694516 >> 172.21.15.73:0/4124694516 conn(0x7f94200f0b70 msgr2=0x7f94200f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:13:13.506 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:13.503+0000 7f94283bc700 1 -- 172.21.15.73:0/4124694516 shutdown_connections 2023-12-01T22:13:13.506 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:13.503+0000 7f94283bc700 1 -- 172.21.15.73:0/4124694516 wait complete. 2023-12-01T22:13:13.506 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:13.503+0000 7f94283bc700 1 Processor -- start 2023-12-01T22:13:13.506 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:13.503+0000 7f94283bc700 1 -- start start 2023-12-01T22:13:13.506 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:13.504+0000 7f94283bc700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9420060220 0x7f94201066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:13:13.506 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:13.504+0000 7f94283bc700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9420106bd0 con 0x7f9420060220 2023-12-01T22:13:13.506 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:13.504+0000 7f9426158700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9420060220 0x7f94201066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:13:13.506 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:13.504+0000 7f9426158700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9420060220 0x7f94201066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:53952/0 (socket says 172.21.15.73:53952) 2023-12-01T22:13:13.507 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:13.504+0000 7f9426158700 1 -- 172.21.15.73:0/3150220769 learned_addr learned my addr 172.21.15.73:0/3150220769 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:13:13.508 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:13.504+0000 7f9426158700 1 -- 172.21.15.73:0/3150220769 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9410005040 con 0x7f9420060220 2023-12-01T22:13:13.508 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:13.504+0000 7f9426158700 1 --2- 172.21.15.73:0/3150220769 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9420060220 0x7f94201066e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f9410013510 tx=0x7f941000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:13:13.508 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:13.505+0000 7f9416ffd700 1 -- 172.21.15.73:0/3150220769 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9410009320 con 0x7f9420060220 2023-12-01T22:13:13.508 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:13.505+0000 7f94283bc700 1 -- 172.21.15.73:0/3150220769 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9420106dd0 con 0x7f9420060220 2023-12-01T22:13:13.508 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:13.505+0000 7f9416ffd700 1 -- 172.21.15.73:0/3150220769 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9410013f00 con 0x7f9420060220 2023-12-01T22:13:13.508 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:13.505+0000 7f9416ffd700 1 -- 172.21.15.73:0/3150220769 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9410014dc0 con 0x7f9420060220 2023-12-01T22:13:13.508 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:13.505+0000 7f94283bc700 1 -- 172.21.15.73:0/3150220769 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9420107260 con 0x7f9420060220 2023-12-01T22:13:13.508 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:13.505+0000 7f9416ffd700 1 -- 172.21.15.73:0/3150220769 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f94100095f0 con 0x7f9420060220 2023-12-01T22:13:13.508 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:13.505+0000 7f94283bc700 1 -- 172.21.15.73:0/3150220769 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9420045ad0 con 0x7f9420060220 2023-12-01T22:13:13.509 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:13.506+0000 7f9416ffd700 1 --2- 172.21.15.73:0/3150220769 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f940c02ff60 0x7f940c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:13:13.509 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:13.506+0000 7f9416ffd700 1 -- 172.21.15.73:0/3150220769 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f94100410e0 con 0x7f9420060220 2023-12-01T22:13:13.509 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:13.506+0000 7f9425957700 1 --2- 172.21.15.73:0/3150220769 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f940c02ff60 0x7f940c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:13:13.509 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:13.506+0000 7f9425957700 1 --2- 172.21.15.73:0/3150220769 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f940c02ff60 0x7f940c032420 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f941c0096a0 tx=0x7f941c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:13:13.513 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:13.511+0000 7f9416ffd700 1 -- 172.21.15.73:0/3150220769 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9410014420 con 0x7f9420060220 2023-12-01T22:13:13.718 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:13.717+0000 7f94283bc700 1 -- 172.21.15.73:0/3150220769 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f94201080b0 con 0x7f9420060220 2023-12-01T22:13:13.719 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:13.718+0000 7f9416ffd700 1 -- 172.21.15.73:0/3150220769 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f941001c920 con 0x7f9420060220 2023-12-01T22:13:13.719 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:13:13.719 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:13:13.721 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:13.719+0000 7f94283bc700 1 -- 172.21.15.73:0/3150220769 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f940c02ff60 msgr2=0x7f940c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:13:13.721 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:13.719+0000 7f94283bc700 1 --2- 172.21.15.73:0/3150220769 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f940c02ff60 0x7f940c032420 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f941c0096a0 tx=0x7f941c006b40).stop 2023-12-01T22:13:13.721 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:13.719+0000 7f94283bc700 1 -- 172.21.15.73:0/3150220769 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9420060220 msgr2=0x7f94201066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:13:13.721 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:13.719+0000 7f94283bc700 1 --2- 172.21.15.73:0/3150220769 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9420060220 0x7f94201066e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f9410013510 tx=0x7f941000b040).stop 2023-12-01T22:13:13.721 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:13.719+0000 7f94283bc700 1 -- 172.21.15.73:0/3150220769 shutdown_connections 2023-12-01T22:13:13.721 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:13.719+0000 7f94283bc700 1 --2- 172.21.15.73:0/3150220769 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f940c02ff60 0x7f940c032420 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:13:13.721 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:13.719+0000 7f94283bc700 1 --2- 172.21.15.73:0/3150220769 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9420060220 0x7f94201066e0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:13:13.722 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:13.719+0000 7f94283bc700 1 -- 172.21.15.73:0/3150220769 >> 172.21.15.73:0/3150220769 conn(0x7f94200f0b70 msgr2=0x7f94200f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:13:13.722 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:13.719+0000 7f94283bc700 1 -- 172.21.15.73:0/3150220769 shutdown_connections 2023-12-01T22:13:13.722 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:13.719+0000 7f94283bc700 1 -- 172.21.15.73:0/3150220769 wait complete. 2023-12-01T22:13:13.722 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:13:14.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:13:13 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:13:12.635933+0000 mgr.smithi029.loxrfu (mgr.14164) 61 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:13:14.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:13:13 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:13:12.777369+0000 mon.smithi029 (mon.0) 152 : audit [INF] from='mgr.14164 172.21.15.29:0/3195927111' entity='mgr.smithi029.loxrfu' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi029.loxrfu/mirror_snapshot_schedule"}]: dispatch 2023-12-01T22:13:14.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:13:13 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:13:12.781339+0000 mon.smithi029 (mon.0) 153 : audit [INF] from='mgr.14164 172.21.15.29:0/3195927111' entity='mgr.smithi029.loxrfu' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi029.loxrfu/trash_purge_schedule"}]: dispatch 2023-12-01T22:13:15.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:13:14 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:13:13.718877+0000 mon.smithi029 (mon.0) 154 : audit [DBG] from='client.? 172.21.15.73:0/3150220769' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:13:15.116 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:13:15.116 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:13:16.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:13:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:13:14.636219+0000 mgr.smithi029.loxrfu (mgr.14164) 62 : cluster [DBG] pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:13:16.249 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:16.247+0000 7f86b3fff700 1 -- 172.21.15.73:0/3529625359 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86b4060160 msgr2=0x7f86b40f8fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:13:16.249 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:16.247+0000 7f86b3fff700 1 --2- 172.21.15.73:0/3529625359 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86b4060160 0x7f86b40f8fa0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f86a4009fb0 tx=0x7f86a4009b30).stop 2023-12-01T22:13:16.249 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:16.247+0000 7f86b3fff700 1 -- 172.21.15.73:0/3529625359 shutdown_connections 2023-12-01T22:13:16.249 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:16.247+0000 7f86b3fff700 1 --2- 172.21.15.73:0/3529625359 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86b4060160 0x7f86b40f8fa0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:13:16.249 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:16.247+0000 7f86b3fff700 1 -- 172.21.15.73:0/3529625359 >> 172.21.15.73:0/3529625359 conn(0x7f86b40f09a0 msgr2=0x7f86b40f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:13:16.249 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:16.247+0000 7f86b3fff700 1 -- 172.21.15.73:0/3529625359 shutdown_connections 2023-12-01T22:13:16.249 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:16.247+0000 7f86b3fff700 1 -- 172.21.15.73:0/3529625359 wait complete. 2023-12-01T22:13:16.251 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:16.248+0000 7f86b3fff700 1 Processor -- start 2023-12-01T22:13:16.251 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:16.248+0000 7f86b3fff700 1 -- start start 2023-12-01T22:13:16.251 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:16.248+0000 7f86b3fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86b4060160 0x7f86b4106530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:13:16.251 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:16.248+0000 7f86b3fff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86b4106a20 con 0x7f86b4060160 2023-12-01T22:13:16.251 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:16.248+0000 7f86b2ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86b4060160 0x7f86b4106530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:13:16.251 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:16.248+0000 7f86b2ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86b4060160 0x7f86b4106530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:49694/0 (socket says 172.21.15.73:49694) 2023-12-01T22:13:16.251 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:16.248+0000 7f86b2ffd700 1 -- 172.21.15.73:0/1176157378 learned_addr learned my addr 172.21.15.73:0/1176157378 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:13:16.251 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:16.248+0000 7f86b2ffd700 1 -- 172.21.15.73:0/1176157378 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86a4005040 con 0x7f86b4060160 2023-12-01T22:13:16.252 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:16.249+0000 7f86b2ffd700 1 --2- 172.21.15.73:0/1176157378 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86b4060160 0x7f86b4106530 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f86a4013510 tx=0x7f86a400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:13:16.252 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:16.249+0000 7f869b7fe700 1 -- 172.21.15.73:0/1176157378 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f86a4009320 con 0x7f86b4060160 2023-12-01T22:13:16.252 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:16.249+0000 7f86b3fff700 1 -- 172.21.15.73:0/1176157378 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86b4106c20 con 0x7f86b4060160 2023-12-01T22:13:16.252 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:16.249+0000 7f869b7fe700 1 -- 172.21.15.73:0/1176157378 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f86a4013f00 con 0x7f86b4060160 2023-12-01T22:13:16.252 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:16.249+0000 7f869b7fe700 1 -- 172.21.15.73:0/1176157378 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f86a4014d90 con 0x7f86b4060160 2023-12-01T22:13:16.252 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:16.249+0000 7f86b3fff700 1 -- 172.21.15.73:0/1176157378 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86b41070b0 con 0x7f86b4060160 2023-12-01T22:13:16.252 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:16.250+0000 7f869b7fe700 1 -- 172.21.15.73:0/1176157378 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f86a4014420 con 0x7f86b4060160 2023-12-01T22:13:16.252 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:16.250+0000 7f86b3fff700 1 -- 172.21.15.73:0/1176157378 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86b4045ad0 con 0x7f86b4060160 2023-12-01T22:13:16.253 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:16.250+0000 7f869b7fe700 1 --2- 172.21.15.73:0/1176157378 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f869c02ff60 0x7f869c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:13:16.253 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:16.250+0000 7f869b7fe700 1 -- 172.21.15.73:0/1176157378 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f86a40410c0 con 0x7f86b4060160 2023-12-01T22:13:16.253 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:16.250+0000 7f86b27fc700 1 --2- 172.21.15.73:0/1176157378 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f869c02ff60 0x7f869c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:13:16.253 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:16.250+0000 7f86b27fc700 1 --2- 172.21.15.73:0/1176157378 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f869c02ff60 0x7f869c032420 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f86a80096a0 tx=0x7f86a8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:13:16.256 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:16.255+0000 7f869b7fe700 1 -- 172.21.15.73:0/1176157378 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f86a40095f0 con 0x7f86b4060160 2023-12-01T22:13:16.465 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:16.464+0000 7f86b3fff700 1 -- 172.21.15.73:0/1176157378 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f86b4107f60 con 0x7f86b4060160 2023-12-01T22:13:16.466 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:16.464+0000 7f869b7fe700 1 -- 172.21.15.73:0/1176157378 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f86a401db70 con 0x7f86b4060160 2023-12-01T22:13:16.466 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:13:16.466 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:13:16.468 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:16.466+0000 7f86b3fff700 1 -- 172.21.15.73:0/1176157378 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f869c02ff60 msgr2=0x7f869c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:13:16.468 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:16.466+0000 7f86b3fff700 1 --2- 172.21.15.73:0/1176157378 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f869c02ff60 0x7f869c032420 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f86a80096a0 tx=0x7f86a8006b40).stop 2023-12-01T22:13:16.468 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:16.466+0000 7f86b3fff700 1 -- 172.21.15.73:0/1176157378 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86b4060160 msgr2=0x7f86b4106530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:13:16.468 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:16.466+0000 7f86b3fff700 1 --2- 172.21.15.73:0/1176157378 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86b4060160 0x7f86b4106530 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f86a4013510 tx=0x7f86a400b040).stop 2023-12-01T22:13:16.468 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:16.466+0000 7f86b3fff700 1 -- 172.21.15.73:0/1176157378 shutdown_connections 2023-12-01T22:13:16.468 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:16.466+0000 7f86b3fff700 1 --2- 172.21.15.73:0/1176157378 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f869c02ff60 0x7f869c032420 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:13:16.468 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:16.466+0000 7f86b3fff700 1 --2- 172.21.15.73:0/1176157378 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86b4060160 0x7f86b4106530 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:13:16.468 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:16.466+0000 7f86b3fff700 1 -- 172.21.15.73:0/1176157378 >> 172.21.15.73:0/1176157378 conn(0x7f86b40f09a0 msgr2=0x7f86b40f14a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:13:16.468 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:16.466+0000 7f86b3fff700 1 -- 172.21.15.73:0/1176157378 shutdown_connections 2023-12-01T22:13:16.468 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:16.466+0000 7f86b3fff700 1 -- 172.21.15.73:0/1176157378 wait complete. 2023-12-01T22:13:16.469 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:13:17.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:13:16 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:13:16.465743+0000 mon.smithi029 (mon.0) 155 : audit [DBG] from='client.? 172.21.15.73:0/1176157378' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:13:17.904 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:13:17.905 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:13:18.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:13:17 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:13:16.636503+0000 mgr.smithi029.loxrfu (mgr.14164) 63 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:13:19.071 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:19.067+0000 7f685b59e700 1 -- 172.21.15.73:0/4198713106 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f685c0f51a0 msgr2=0x7f685c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:13:19.071 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:19.067+0000 7f685b59e700 1 --2- 172.21.15.73:0/4198713106 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f685c0f51a0 0x7f685c0f55c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f684c009fb0 tx=0x7f684c009b30).stop 2023-12-01T22:13:19.071 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:19.068+0000 7f685b59e700 1 -- 172.21.15.73:0/4198713106 shutdown_connections 2023-12-01T22:13:19.071 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:19.068+0000 7f685b59e700 1 --2- 172.21.15.73:0/4198713106 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f685c0f51a0 0x7f685c0f55c0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:13:19.071 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:19.068+0000 7f685b59e700 1 -- 172.21.15.73:0/4198713106 >> 172.21.15.73:0/4198713106 conn(0x7f685c0f09a0 msgr2=0x7f685c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:13:19.071 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:19.068+0000 7f685b59e700 1 -- 172.21.15.73:0/4198713106 shutdown_connections 2023-12-01T22:13:19.071 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:19.068+0000 7f685b59e700 1 -- 172.21.15.73:0/4198713106 wait complete. 2023-12-01T22:13:19.071 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:19.068+0000 7f685b59e700 1 Processor -- start 2023-12-01T22:13:19.071 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:19.068+0000 7f685b59e700 1 -- start start 2023-12-01T22:13:19.071 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:19.068+0000 7f685b59e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f685c0f51a0 0x7f685c1064c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:13:19.072 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:19.068+0000 7f685b59e700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f685c1069b0 con 0x7f685c0f51a0 2023-12-01T22:13:19.072 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:19.068+0000 7f685a59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f685c0f51a0 0x7f685c1064c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:13:19.072 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:19.068+0000 7f685a59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f685c0f51a0 0x7f685c1064c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:49718/0 (socket says 172.21.15.73:49718) 2023-12-01T22:13:19.072 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:19.068+0000 7f685a59c700 1 -- 172.21.15.73:0/3533926227 learned_addr learned my addr 172.21.15.73:0/3533926227 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:13:19.072 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:19.069+0000 7f685a59c700 1 -- 172.21.15.73:0/3533926227 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f684c005040 con 0x7f685c0f51a0 2023-12-01T22:13:19.072 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:19.069+0000 7f685a59c700 1 --2- 172.21.15.73:0/3533926227 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f685c0f51a0 0x7f685c1064c0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f684c013510 tx=0x7f684c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:13:19.072 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:19.069+0000 7f684affd700 1 -- 172.21.15.73:0/3533926227 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f684c009320 con 0x7f685c0f51a0 2023-12-01T22:13:19.072 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:19.069+0000 7f685b59e700 1 -- 172.21.15.73:0/3533926227 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f685c106bb0 con 0x7f685c0f51a0 2023-12-01T22:13:19.072 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:19.069+0000 7f684affd700 1 -- 172.21.15.73:0/3533926227 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f684c013f00 con 0x7f685c0f51a0 2023-12-01T22:13:19.072 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:19.069+0000 7f684affd700 1 -- 172.21.15.73:0/3533926227 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f684c014dc0 con 0x7f685c0f51a0 2023-12-01T22:13:19.072 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:19.069+0000 7f685b59e700 1 -- 172.21.15.73:0/3533926227 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f685c107040 con 0x7f685c0f51a0 2023-12-01T22:13:19.073 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:19.070+0000 7f684affd700 1 -- 172.21.15.73:0/3533926227 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f684c0095f0 con 0x7f685c0f51a0 2023-12-01T22:13:19.073 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:19.070+0000 7f685b59e700 1 -- 172.21.15.73:0/3533926227 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f685c045ad0 con 0x7f685c0f51a0 2023-12-01T22:13:19.073 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:19.070+0000 7f684affd700 1 --2- 172.21.15.73:0/3533926227 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f684402ff60 0x7f6844032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:13:19.073 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:19.070+0000 7f684affd700 1 -- 172.21.15.73:0/3533926227 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f684c0410c0 con 0x7f685c0f51a0 2023-12-01T22:13:19.073 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:19.070+0000 7f6859d9b700 1 --2- 172.21.15.73:0/3533926227 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f684402ff60 0x7f6844032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:13:19.074 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:19.071+0000 7f6859d9b700 1 --2- 172.21.15.73:0/3533926227 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f684402ff60 0x7f6844032420 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f68500096a0 tx=0x7f6850006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:13:19.076 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:19.075+0000 7f684affd700 1 -- 172.21.15.73:0/3533926227 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f684c014420 con 0x7f685c0f51a0 2023-12-01T22:13:19.281 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:19.280+0000 7f685b59e700 1 -- 172.21.15.73:0/3533926227 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f685c107ef0 con 0x7f685c0f51a0 2023-12-01T22:13:19.282 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:19.280+0000 7f684affd700 1 -- 172.21.15.73:0/3533926227 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f684c010230 con 0x7f685c0f51a0 2023-12-01T22:13:19.282 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:13:19.282 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:13:19.284 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:19.282+0000 7f685b59e700 1 -- 172.21.15.73:0/3533926227 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f684402ff60 msgr2=0x7f6844032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:13:19.284 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:19.282+0000 7f685b59e700 1 --2- 172.21.15.73:0/3533926227 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f684402ff60 0x7f6844032420 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f68500096a0 tx=0x7f6850006b40).stop 2023-12-01T22:13:19.284 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:19.282+0000 7f685b59e700 1 -- 172.21.15.73:0/3533926227 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f685c0f51a0 msgr2=0x7f685c1064c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:13:19.284 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:19.282+0000 7f685b59e700 1 --2- 172.21.15.73:0/3533926227 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f685c0f51a0 0x7f685c1064c0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f684c013510 tx=0x7f684c00b040).stop 2023-12-01T22:13:19.284 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:19.282+0000 7f685b59e700 1 -- 172.21.15.73:0/3533926227 shutdown_connections 2023-12-01T22:13:19.284 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:19.282+0000 7f685b59e700 1 --2- 172.21.15.73:0/3533926227 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f684402ff60 0x7f6844032420 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:13:19.284 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:19.282+0000 7f685b59e700 1 --2- 172.21.15.73:0/3533926227 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f685c0f51a0 0x7f685c1064c0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:13:19.284 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:19.282+0000 7f685b59e700 1 -- 172.21.15.73:0/3533926227 >> 172.21.15.73:0/3533926227 conn(0x7f685c0f09a0 msgr2=0x7f685c0f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:13:19.284 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:19.282+0000 7f685b59e700 1 -- 172.21.15.73:0/3533926227 shutdown_connections 2023-12-01T22:13:19.284 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:19.282+0000 7f685b59e700 1 -- 172.21.15.73:0/3533926227 wait complete. 2023-12-01T22:13:19.285 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:13:20.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:13:19 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:13:18.636854+0000 mgr.smithi029.loxrfu (mgr.14164) 64 : cluster [DBG] pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:13:20.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:13:19 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:13:19.281571+0000 mon.smithi029 (mon.0) 156 : audit [DBG] from='client.? 172.21.15.73:0/3533926227' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:13:20.715 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:13:20.716 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:13:21.776 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:13:21 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:13:20.637249+0000 mgr.smithi029.loxrfu (mgr.14164) 65 : cluster [DBG] pgmap v58: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:13:21.847 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:21.845+0000 7f1fb84e2700 1 -- 172.21.15.73:0/2253938025 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1fb00f5410 msgr2=0x7f1fb00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:13:21.847 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:21.845+0000 7f1fb84e2700 1 --2- 172.21.15.73:0/2253938025 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1fb00f5410 0x7f1fb00f5830 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f1fa0009fb0 tx=0x7f1fa0009b30).stop 2023-12-01T22:13:21.847 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:21.845+0000 7f1fb84e2700 1 -- 172.21.15.73:0/2253938025 shutdown_connections 2023-12-01T22:13:21.847 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:21.845+0000 7f1fb84e2700 1 --2- 172.21.15.73:0/2253938025 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1fb00f5410 0x7f1fb00f5830 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:13:21.847 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:21.845+0000 7f1fb84e2700 1 -- 172.21.15.73:0/2253938025 >> 172.21.15.73:0/2253938025 conn(0x7f1fb00f0b70 msgr2=0x7f1fb00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:13:21.847 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:21.845+0000 7f1fb84e2700 1 -- 172.21.15.73:0/2253938025 shutdown_connections 2023-12-01T22:13:21.847 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:21.845+0000 7f1fb84e2700 1 -- 172.21.15.73:0/2253938025 wait complete. 2023-12-01T22:13:21.849 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:21.846+0000 7f1fb84e2700 1 Processor -- start 2023-12-01T22:13:21.849 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:21.846+0000 7f1fb84e2700 1 -- start start 2023-12-01T22:13:21.849 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:21.846+0000 7f1fb84e2700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1fb00f5410 0x7f1fb01093f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:13:21.849 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:21.846+0000 7f1fb84e2700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fb0060280 con 0x7f1fb00f5410 2023-12-01T22:13:21.849 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:21.846+0000 7f1fb627e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1fb00f5410 0x7f1fb01093f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:13:21.849 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:21.846+0000 7f1fb627e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1fb00f5410 0x7f1fb01093f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:49742/0 (socket says 172.21.15.73:49742) 2023-12-01T22:13:21.849 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:21.846+0000 7f1fb627e700 1 -- 172.21.15.73:0/599554023 learned_addr learned my addr 172.21.15.73:0/599554023 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:13:21.849 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:21.847+0000 7f1fb627e700 1 -- 172.21.15.73:0/599554023 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fa0005040 con 0x7f1fb00f5410 2023-12-01T22:13:21.849 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:21.847+0000 7f1fb627e700 1 --2- 172.21.15.73:0/599554023 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1fb00f5410 0x7f1fb01093f0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f1fa0013510 tx=0x7f1fa000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:13:21.849 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:21.847+0000 7f1fa6ffd700 1 -- 172.21.15.73:0/599554023 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1fa0009320 con 0x7f1fb00f5410 2023-12-01T22:13:21.850 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:21.847+0000 7f1fb84e2700 1 -- 172.21.15.73:0/599554023 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fb0060480 con 0x7f1fb00f5410 2023-12-01T22:13:21.850 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:21.847+0000 7f1fa6ffd700 1 -- 172.21.15.73:0/599554023 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1fa0013f00 con 0x7f1fb00f5410 2023-12-01T22:13:21.850 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:21.847+0000 7f1fa6ffd700 1 -- 172.21.15.73:0/599554023 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1fa0014d90 con 0x7f1fb00f5410 2023-12-01T22:13:21.850 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:21.847+0000 7f1fb84e2700 1 -- 172.21.15.73:0/599554023 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fb0109b20 con 0x7f1fb00f5410 2023-12-01T22:13:21.850 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:21.848+0000 7f1fa6ffd700 1 -- 172.21.15.73:0/599554023 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1fa00263a0 con 0x7f1fb00f5410 2023-12-01T22:13:21.851 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:21.848+0000 7f1fa6ffd700 1 --2- 172.21.15.73:0/599554023 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f1f9c02ff60 0x7f1f9c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:13:21.851 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:21.848+0000 7f1fb84e2700 1 -- 172.21.15.73:0/599554023 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fb0045ad0 con 0x7f1fb00f5410 2023-12-01T22:13:21.851 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:21.848+0000 7f1fa6ffd700 1 -- 172.21.15.73:0/599554023 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1fa00410e0 con 0x7f1fb00f5410 2023-12-01T22:13:21.851 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:21.848+0000 7f1fb5a7d700 1 --2- 172.21.15.73:0/599554023 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f1f9c02ff60 0x7f1f9c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:13:21.851 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:21.849+0000 7f1fb5a7d700 1 --2- 172.21.15.73:0/599554023 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f1f9c02ff60 0x7f1f9c032420 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f1fac0096a0 tx=0x7f1fac006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:13:21.855 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:21.853+0000 7f1fa6ffd700 1 -- 172.21.15.73:0/599554023 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1fa0012920 con 0x7f1fb00f5410 2023-12-01T22:13:22.066 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:22.063+0000 7f1fb84e2700 1 -- 172.21.15.73:0/599554023 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1fb010a340 con 0x7f1fb00f5410 2023-12-01T22:13:22.066 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:22.064+0000 7f1fa6ffd700 1 -- 172.21.15.73:0/599554023 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1fa002b020 con 0x7f1fb00f5410 2023-12-01T22:13:22.069 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:13:22.069 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:13:22.070 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:22.066+0000 7f1fb84e2700 1 -- 172.21.15.73:0/599554023 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f1f9c02ff60 msgr2=0x7f1f9c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:13:22.070 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:22.066+0000 7f1fb84e2700 1 --2- 172.21.15.73:0/599554023 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f1f9c02ff60 0x7f1f9c032420 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f1fac0096a0 tx=0x7f1fac006b40).stop 2023-12-01T22:13:22.071 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:22.066+0000 7f1fb84e2700 1 -- 172.21.15.73:0/599554023 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1fb00f5410 msgr2=0x7f1fb01093f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:13:22.071 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:22.066+0000 7f1fb84e2700 1 --2- 172.21.15.73:0/599554023 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1fb00f5410 0x7f1fb01093f0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f1fa0013510 tx=0x7f1fa000b040).stop 2023-12-01T22:13:22.071 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:22.066+0000 7f1fb84e2700 1 -- 172.21.15.73:0/599554023 shutdown_connections 2023-12-01T22:13:22.071 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:22.066+0000 7f1fb84e2700 1 --2- 172.21.15.73:0/599554023 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f1f9c02ff60 0x7f1f9c032420 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:13:22.071 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:22.066+0000 7f1fb84e2700 1 --2- 172.21.15.73:0/599554023 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1fb00f5410 0x7f1fb01093f0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:13:22.071 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:22.066+0000 7f1fb84e2700 1 -- 172.21.15.73:0/599554023 >> 172.21.15.73:0/599554023 conn(0x7f1fb00f0b70 msgr2=0x7f1fb00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:13:22.071 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:22.066+0000 7f1fb84e2700 1 -- 172.21.15.73:0/599554023 shutdown_connections 2023-12-01T22:13:22.071 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:22.066+0000 7f1fb84e2700 1 -- 172.21.15.73:0/599554023 wait complete. 2023-12-01T22:13:22.071 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:13:22.777 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:13:22 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:13:22.065440+0000 mon.smithi029 (mon.0) 157 : audit [DBG] from='client.? 172.21.15.73:0/599554023' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:13:23.492 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:13:23.493 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:13:23.777 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:13:23 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:13:22.637611+0000 mgr.smithi029.loxrfu (mgr.14164) 66 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:13:24.610 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:24.608+0000 7f1dc24af700 1 -- 172.21.15.73:0/136956462 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1dbc0f6510 msgr2=0x7f1dbc0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:13:24.610 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:24.608+0000 7f1dc24af700 1 --2- 172.21.15.73:0/136956462 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1dbc0f6510 0x7f1dbc0f6930 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f1db0009fb0 tx=0x7f1db0009b30).stop 2023-12-01T22:13:24.610 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:24.608+0000 7f1dc24af700 1 -- 172.21.15.73:0/136956462 shutdown_connections 2023-12-01T22:13:24.611 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:24.608+0000 7f1dc24af700 1 --2- 172.21.15.73:0/136956462 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1dbc0f6510 0x7f1dbc0f6930 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:13:24.611 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:24.608+0000 7f1dc24af700 1 -- 172.21.15.73:0/136956462 >> 172.21.15.73:0/136956462 conn(0x7f1dbc0f1c70 msgr2=0x7f1dbc0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:13:24.611 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:24.608+0000 7f1dc24af700 1 -- 172.21.15.73:0/136956462 shutdown_connections 2023-12-01T22:13:24.611 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:24.608+0000 7f1dc24af700 1 -- 172.21.15.73:0/136956462 wait complete. 2023-12-01T22:13:24.611 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:24.608+0000 7f1dc24af700 1 Processor -- start 2023-12-01T22:13:24.611 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:24.609+0000 7f1dc24af700 1 -- start start 2023-12-01T22:13:24.611 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:24.609+0000 7f1dc24af700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1dbc0f6510 0x7f1dbc106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:13:24.611 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:24.609+0000 7f1dc24af700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dbc106c20 con 0x7f1dbc0f6510 2023-12-01T22:13:24.613 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:24.609+0000 7f1dbbfff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1dbc0f6510 0x7f1dbc106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:13:24.613 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:24.609+0000 7f1dbbfff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1dbc0f6510 0x7f1dbc106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:42012/0 (socket says 172.21.15.73:42012) 2023-12-01T22:13:24.613 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:24.609+0000 7f1dbbfff700 1 -- 172.21.15.73:0/3106857818 learned_addr learned my addr 172.21.15.73:0/3106857818 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:13:24.613 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:24.609+0000 7f1dbbfff700 1 -- 172.21.15.73:0/3106857818 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1db0005040 con 0x7f1dbc0f6510 2023-12-01T22:13:24.613 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:24.609+0000 7f1dbbfff700 1 --2- 172.21.15.73:0/3106857818 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1dbc0f6510 0x7f1dbc106730 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f1db0013510 tx=0x7f1db000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:13:24.613 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:24.610+0000 7f1db8ff9700 1 -- 172.21.15.73:0/3106857818 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1db0009320 con 0x7f1dbc0f6510 2023-12-01T22:13:24.613 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:24.610+0000 7f1dc24af700 1 -- 172.21.15.73:0/3106857818 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1dbc106e20 con 0x7f1dbc0f6510 2023-12-01T22:13:24.613 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:24.610+0000 7f1db8ff9700 1 -- 172.21.15.73:0/3106857818 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1db0013f00 con 0x7f1dbc0f6510 2023-12-01T22:13:24.613 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:24.610+0000 7f1db8ff9700 1 -- 172.21.15.73:0/3106857818 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1db0014d90 con 0x7f1dbc0f6510 2023-12-01T22:13:24.613 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:24.610+0000 7f1dc24af700 1 -- 172.21.15.73:0/3106857818 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1dbc1072b0 con 0x7f1dbc0f6510 2023-12-01T22:13:24.613 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:24.610+0000 7f1db8ff9700 1 -- 172.21.15.73:0/3106857818 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1db00263a0 con 0x7f1dbc0f6510 2023-12-01T22:13:24.614 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:24.610+0000 7f1dc24af700 1 -- 172.21.15.73:0/3106857818 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1dbc045ad0 con 0x7f1dbc0f6510 2023-12-01T22:13:24.614 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:24.611+0000 7f1db8ff9700 1 --2- 172.21.15.73:0/3106857818 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f1da402ff60 0x7f1da4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:13:24.614 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:24.611+0000 7f1db8ff9700 1 -- 172.21.15.73:0/3106857818 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1db0041400 con 0x7f1dbc0f6510 2023-12-01T22:13:24.614 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:24.611+0000 7f1dbb7fe700 1 --2- 172.21.15.73:0/3106857818 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f1da402ff60 0x7f1da4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:13:24.614 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:24.611+0000 7f1dbb7fe700 1 --2- 172.21.15.73:0/3106857818 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f1da402ff60 0x7f1da4032420 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f1dac0096a0 tx=0x7f1dac006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:13:24.617 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:24.615+0000 7f1db8ff9700 1 -- 172.21.15.73:0/3106857818 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1db00126f0 con 0x7f1dbc0f6510 2023-12-01T22:13:24.821 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:24.820+0000 7f1dc24af700 1 -- 172.21.15.73:0/3106857818 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1dbc0fbf60 con 0x7f1dbc0f6510 2023-12-01T22:13:24.822 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:24.820+0000 7f1db8ff9700 1 -- 172.21.15.73:0/3106857818 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1db0012330 con 0x7f1dbc0f6510 2023-12-01T22:13:24.822 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:13:24.822 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:13:24.824 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:24.822+0000 7f1dc24af700 1 -- 172.21.15.73:0/3106857818 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f1da402ff60 msgr2=0x7f1da4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:13:24.824 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:24.822+0000 7f1dc24af700 1 --2- 172.21.15.73:0/3106857818 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f1da402ff60 0x7f1da4032420 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f1dac0096a0 tx=0x7f1dac006b40).stop 2023-12-01T22:13:24.824 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:24.822+0000 7f1dc24af700 1 -- 172.21.15.73:0/3106857818 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1dbc0f6510 msgr2=0x7f1dbc106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:13:24.824 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:24.822+0000 7f1dc24af700 1 --2- 172.21.15.73:0/3106857818 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1dbc0f6510 0x7f1dbc106730 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f1db0013510 tx=0x7f1db000b040).stop 2023-12-01T22:13:24.824 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:24.822+0000 7f1dc24af700 1 -- 172.21.15.73:0/3106857818 shutdown_connections 2023-12-01T22:13:24.824 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:24.822+0000 7f1dc24af700 1 --2- 172.21.15.73:0/3106857818 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f1da402ff60 0x7f1da4032420 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:13:24.824 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:24.822+0000 7f1dc24af700 1 --2- 172.21.15.73:0/3106857818 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1dbc0f6510 0x7f1dbc106730 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:13:24.824 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:24.822+0000 7f1dc24af700 1 -- 172.21.15.73:0/3106857818 >> 172.21.15.73:0/3106857818 conn(0x7f1dbc0f1c70 msgr2=0x7f1dbc0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:13:24.824 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:24.822+0000 7f1dc24af700 1 -- 172.21.15.73:0/3106857818 shutdown_connections 2023-12-01T22:13:24.824 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:24.822+0000 7f1dc24af700 1 -- 172.21.15.73:0/3106857818 wait complete. 2023-12-01T22:13:24.825 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:13:26.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:13:25 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:13:24.637948+0000 mgr.smithi029.loxrfu (mgr.14164) 67 : cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:13:26.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:13:25 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:13:24.821693+0000 mon.smithi029 (mon.0) 158 : audit [DBG] from='client.? 172.21.15.73:0/3106857818' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:13:26.217 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:13:26.217 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:13:27.382 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:27.378+0000 7f045fb00700 1 -- 172.21.15.73:0/574423230 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f04580f6510 msgr2=0x7f04580f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:13:27.382 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:27.378+0000 7f045fb00700 1 --2- 172.21.15.73:0/574423230 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f04580f6510 0x7f04580f6930 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f044c009fb0 tx=0x7f044c009b30).stop 2023-12-01T22:13:27.382 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:27.378+0000 7f045fb00700 1 -- 172.21.15.73:0/574423230 shutdown_connections 2023-12-01T22:13:27.382 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:27.378+0000 7f045fb00700 1 --2- 172.21.15.73:0/574423230 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f04580f6510 0x7f04580f6930 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:13:27.382 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:27.378+0000 7f045fb00700 1 -- 172.21.15.73:0/574423230 >> 172.21.15.73:0/574423230 conn(0x7f04580f1c70 msgr2=0x7f04580f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:13:27.382 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:27.378+0000 7f045fb00700 1 -- 172.21.15.73:0/574423230 shutdown_connections 2023-12-01T22:13:27.382 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:27.379+0000 7f045fb00700 1 -- 172.21.15.73:0/574423230 wait complete. 2023-12-01T22:13:27.382 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:27.379+0000 7f045fb00700 1 Processor -- start 2023-12-01T22:13:27.382 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:27.379+0000 7f045fb00700 1 -- start start 2023-12-01T22:13:27.383 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:27.379+0000 7f045fb00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f04580f6510 0x7f0458106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:13:27.383 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:27.379+0000 7f045fb00700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0458106c20 con 0x7f04580f6510 2023-12-01T22:13:27.383 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:27.379+0000 7f045d89c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f04580f6510 0x7f0458106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:13:27.383 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:27.379+0000 7f045d89c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f04580f6510 0x7f0458106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:42026/0 (socket says 172.21.15.73:42026) 2023-12-01T22:13:27.383 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:27.379+0000 7f045d89c700 1 -- 172.21.15.73:0/1056638147 learned_addr learned my addr 172.21.15.73:0/1056638147 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:13:27.383 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:27.380+0000 7f045d89c700 1 -- 172.21.15.73:0/1056638147 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f044c005040 con 0x7f04580f6510 2023-12-01T22:13:27.383 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:27.380+0000 7f045d89c700 1 --2- 172.21.15.73:0/1056638147 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f04580f6510 0x7f0458106730 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f044c013510 tx=0x7f044c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:13:27.383 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:27.380+0000 7f044a7fc700 1 -- 172.21.15.73:0/1056638147 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f044c009320 con 0x7f04580f6510 2023-12-01T22:13:27.383 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:27.380+0000 7f045fb00700 1 -- 172.21.15.73:0/1056638147 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0458106e20 con 0x7f04580f6510 2023-12-01T22:13:27.383 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:27.380+0000 7f044a7fc700 1 -- 172.21.15.73:0/1056638147 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f044c013f00 con 0x7f04580f6510 2023-12-01T22:13:27.383 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:27.380+0000 7f044a7fc700 1 -- 172.21.15.73:0/1056638147 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f044c014d90 con 0x7f04580f6510 2023-12-01T22:13:27.383 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:27.380+0000 7f045fb00700 1 -- 172.21.15.73:0/1056638147 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04581072b0 con 0x7f04580f6510 2023-12-01T22:13:27.384 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:27.381+0000 7f044a7fc700 1 -- 172.21.15.73:0/1056638147 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f044c0095f0 con 0x7f04580f6510 2023-12-01T22:13:27.384 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:27.381+0000 7f045fb00700 1 -- 172.21.15.73:0/1056638147 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0458045ad0 con 0x7f04580f6510 2023-12-01T22:13:27.384 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:27.381+0000 7f044a7fc700 1 --2- 172.21.15.73:0/1056638147 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f044402ff60 0x7f0444032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:13:27.384 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:27.381+0000 7f044a7fc700 1 -- 172.21.15.73:0/1056638147 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f044c041000 con 0x7f04580f6510 2023-12-01T22:13:27.384 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:27.381+0000 7f045d09b700 1 --2- 172.21.15.73:0/1056638147 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f044402ff60 0x7f0444032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:13:27.384 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:27.382+0000 7f045d09b700 1 --2- 172.21.15.73:0/1056638147 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f044402ff60 0x7f0444032420 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f04540096a0 tx=0x7f0454006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:13:27.387 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:27.386+0000 7f044a7fc700 1 -- 172.21.15.73:0/1056638147 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f044c014420 con 0x7f04580f6510 2023-12-01T22:13:27.595 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:27.593+0000 7f045fb00700 1 -- 172.21.15.73:0/1056638147 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f04580fbf60 con 0x7f04580f6510 2023-12-01T22:13:27.595 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:27.593+0000 7f044a7fc700 1 -- 172.21.15.73:0/1056638147 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f044c010230 con 0x7f04580f6510 2023-12-01T22:13:27.596 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:13:27.596 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:13:27.597 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:27.595+0000 7f045fb00700 1 -- 172.21.15.73:0/1056638147 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f044402ff60 msgr2=0x7f0444032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:13:27.597 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:27.595+0000 7f045fb00700 1 --2- 172.21.15.73:0/1056638147 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f044402ff60 0x7f0444032420 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f04540096a0 tx=0x7f0454006b40).stop 2023-12-01T22:13:27.597 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:27.595+0000 7f045fb00700 1 -- 172.21.15.73:0/1056638147 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f04580f6510 msgr2=0x7f0458106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:13:27.597 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:27.595+0000 7f045fb00700 1 --2- 172.21.15.73:0/1056638147 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f04580f6510 0x7f0458106730 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f044c013510 tx=0x7f044c00b040).stop 2023-12-01T22:13:27.597 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:27.595+0000 7f045fb00700 1 -- 172.21.15.73:0/1056638147 shutdown_connections 2023-12-01T22:13:27.597 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:27.595+0000 7f045fb00700 1 --2- 172.21.15.73:0/1056638147 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f044402ff60 0x7f0444032420 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:13:27.597 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:27.595+0000 7f045fb00700 1 --2- 172.21.15.73:0/1056638147 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f04580f6510 0x7f0458106730 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:13:27.597 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:27.595+0000 7f045fb00700 1 -- 172.21.15.73:0/1056638147 >> 172.21.15.73:0/1056638147 conn(0x7f04580f1c70 msgr2=0x7f04580f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:13:27.597 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:27.595+0000 7f045fb00700 1 -- 172.21.15.73:0/1056638147 shutdown_connections 2023-12-01T22:13:27.597 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:27.595+0000 7f045fb00700 1 -- 172.21.15.73:0/1056638147 wait complete. 2023-12-01T22:13:27.598 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:13:28.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:13:27 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:13:26.638298+0000 mgr.smithi029.loxrfu (mgr.14164) 68 : cluster [DBG] pgmap v61: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:13:28.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:13:27 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:13:27.594824+0000 mon.smithi029 (mon.0) 159 : audit [DBG] from='client.? 172.21.15.73:0/1056638147' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:13:29.022 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:13:29.023 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:13:30.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:13:29 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:13:28.638653+0000 mgr.smithi029.loxrfu (mgr.14164) 69 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:13:30.185 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:30.182+0000 7f5c5bba5700 1 -- 172.21.15.73:0/2120462030 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5c540f6510 msgr2=0x7f5c540f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:13:30.185 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:30.182+0000 7f5c5bba5700 1 --2- 172.21.15.73:0/2120462030 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5c540f6510 0x7f5c540f6930 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f5c44009fb0 tx=0x7f5c44009b30).stop 2023-12-01T22:13:30.185 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:30.182+0000 7f5c5bba5700 1 -- 172.21.15.73:0/2120462030 shutdown_connections 2023-12-01T22:13:30.185 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:30.183+0000 7f5c5bba5700 1 --2- 172.21.15.73:0/2120462030 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5c540f6510 0x7f5c540f6930 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:13:30.185 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:30.183+0000 7f5c5bba5700 1 -- 172.21.15.73:0/2120462030 >> 172.21.15.73:0/2120462030 conn(0x7f5c540f1c70 msgr2=0x7f5c540f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:13:30.185 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:30.183+0000 7f5c5bba5700 1 -- 172.21.15.73:0/2120462030 shutdown_connections 2023-12-01T22:13:30.185 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:30.183+0000 7f5c5bba5700 1 -- 172.21.15.73:0/2120462030 wait complete. 2023-12-01T22:13:30.186 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:30.183+0000 7f5c5bba5700 1 Processor -- start 2023-12-01T22:13:30.186 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:30.183+0000 7f5c5bba5700 1 -- start start 2023-12-01T22:13:30.187 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:30.184+0000 7f5c5bba5700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5c540f6510 0x7f5c5406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:13:30.187 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:30.184+0000 7f5c5bba5700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c54069e20 con 0x7f5c540f6510 2023-12-01T22:13:30.187 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:30.184+0000 7f5c59941700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5c540f6510 0x7f5c5406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:13:30.188 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:30.184+0000 7f5c59941700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5c540f6510 0x7f5c5406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:42032/0 (socket says 172.21.15.73:42032) 2023-12-01T22:13:30.188 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:30.184+0000 7f5c59941700 1 -- 172.21.15.73:0/3917935733 learned_addr learned my addr 172.21.15.73:0/3917935733 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:13:30.188 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:30.184+0000 7f5c59941700 1 -- 172.21.15.73:0/3917935733 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c44005040 con 0x7f5c540f6510 2023-12-01T22:13:30.188 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:30.184+0000 7f5c59941700 1 --2- 172.21.15.73:0/3917935733 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5c540f6510 0x7f5c5406b9a0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f5c44013510 tx=0x7f5c4400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:13:30.188 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:30.184+0000 7f5c4a7fc700 1 -- 172.21.15.73:0/3917935733 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5c44009320 con 0x7f5c540f6510 2023-12-01T22:13:30.188 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:30.185+0000 7f5c4a7fc700 1 -- 172.21.15.73:0/3917935733 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5c44013f00 con 0x7f5c540f6510 2023-12-01T22:13:30.188 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:30.185+0000 7f5c5bba5700 1 -- 172.21.15.73:0/3917935733 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c54069fc0 con 0x7f5c540f6510 2023-12-01T22:13:30.188 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:30.185+0000 7f5c4a7fc700 1 -- 172.21.15.73:0/3917935733 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5c44014d90 con 0x7f5c540f6510 2023-12-01T22:13:30.188 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:30.185+0000 7f5c5bba5700 1 -- 172.21.15.73:0/3917935733 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c5406a450 con 0x7f5c540f6510 2023-12-01T22:13:30.188 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:30.185+0000 7f5c4a7fc700 1 -- 172.21.15.73:0/3917935733 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5c44014420 con 0x7f5c540f6510 2023-12-01T22:13:30.188 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:30.186+0000 7f5c5bba5700 1 -- 172.21.15.73:0/3917935733 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c54059100 con 0x7f5c540f6510 2023-12-01T22:13:30.188 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:30.186+0000 7f5c4a7fc700 1 --2- 172.21.15.73:0/3917935733 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f5c4002ff60 0x7f5c40032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:13:30.188 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:30.186+0000 7f5c4a7fc700 1 -- 172.21.15.73:0/3917935733 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5c440410c0 con 0x7f5c540f6510 2023-12-01T22:13:30.188 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:30.186+0000 7f5c59140700 1 --2- 172.21.15.73:0/3917935733 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f5c4002ff60 0x7f5c40032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:13:30.189 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:30.186+0000 7f5c59140700 1 --2- 172.21.15.73:0/3917935733 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f5c4002ff60 0x7f5c40032420 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f5c500096a0 tx=0x7f5c50006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:13:30.192 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:30.190+0000 7f5c4a7fc700 1 -- 172.21.15.73:0/3917935733 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5c440095f0 con 0x7f5c540f6510 2023-12-01T22:13:30.396 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:30.395+0000 7f5c5bba5700 1 -- 172.21.15.73:0/3917935733 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5c540fbf60 con 0x7f5c540f6510 2023-12-01T22:13:30.397 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:30.395+0000 7f5c4a7fc700 1 -- 172.21.15.73:0/3917935733 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5c4401db70 con 0x7f5c540f6510 2023-12-01T22:13:30.397 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:13:30.397 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:13:30.398 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:30.397+0000 7f5c5bba5700 1 -- 172.21.15.73:0/3917935733 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f5c4002ff60 msgr2=0x7f5c40032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:13:30.399 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:30.397+0000 7f5c5bba5700 1 --2- 172.21.15.73:0/3917935733 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f5c4002ff60 0x7f5c40032420 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f5c500096a0 tx=0x7f5c50006b40).stop 2023-12-01T22:13:30.399 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:30.397+0000 7f5c5bba5700 1 -- 172.21.15.73:0/3917935733 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5c540f6510 msgr2=0x7f5c5406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:13:30.399 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:30.397+0000 7f5c5bba5700 1 --2- 172.21.15.73:0/3917935733 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5c540f6510 0x7f5c5406b9a0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f5c44013510 tx=0x7f5c4400b040).stop 2023-12-01T22:13:30.399 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:30.397+0000 7f5c5bba5700 1 -- 172.21.15.73:0/3917935733 shutdown_connections 2023-12-01T22:13:30.399 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:30.397+0000 7f5c5bba5700 1 --2- 172.21.15.73:0/3917935733 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f5c4002ff60 0x7f5c40032420 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:13:30.399 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:30.397+0000 7f5c5bba5700 1 --2- 172.21.15.73:0/3917935733 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5c540f6510 0x7f5c5406b9a0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:13:30.399 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:30.397+0000 7f5c5bba5700 1 -- 172.21.15.73:0/3917935733 >> 172.21.15.73:0/3917935733 conn(0x7f5c540f1c70 msgr2=0x7f5c540f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:13:30.399 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:30.397+0000 7f5c5bba5700 1 -- 172.21.15.73:0/3917935733 shutdown_connections 2023-12-01T22:13:30.399 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:30.397+0000 7f5c5bba5700 1 -- 172.21.15.73:0/3917935733 wait complete. 2023-12-01T22:13:30.400 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:13:31.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:13:30 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:13:30.396718+0000 mon.smithi029 (mon.0) 160 : audit [DBG] from='client.? 172.21.15.73:0/3917935733' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:13:31.786 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:13:31.787 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:13:32.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:13:31 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:13:30.639033+0000 mgr.smithi029.loxrfu (mgr.14164) 70 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:13:32.952 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:32.949+0000 7f5a45888700 1 -- 172.21.15.73:0/2200308326 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a400f5410 msgr2=0x7f5a400f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:13:32.953 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:32.949+0000 7f5a45888700 1 --2- 172.21.15.73:0/2200308326 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a400f5410 0x7f5a400f5830 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f5a28009fb0 tx=0x7f5a28009b30).stop 2023-12-01T22:13:32.953 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:32.949+0000 7f5a45888700 1 -- 172.21.15.73:0/2200308326 shutdown_connections 2023-12-01T22:13:32.953 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:32.949+0000 7f5a45888700 1 --2- 172.21.15.73:0/2200308326 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a400f5410 0x7f5a400f5830 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:13:32.953 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:32.949+0000 7f5a45888700 1 -- 172.21.15.73:0/2200308326 >> 172.21.15.73:0/2200308326 conn(0x7f5a400f0b70 msgr2=0x7f5a400f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:13:32.953 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:32.949+0000 7f5a45888700 1 -- 172.21.15.73:0/2200308326 shutdown_connections 2023-12-01T22:13:32.953 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:32.949+0000 7f5a45888700 1 -- 172.21.15.73:0/2200308326 wait complete. 2023-12-01T22:13:32.953 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:32.950+0000 7f5a45888700 1 Processor -- start 2023-12-01T22:13:32.953 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:32.950+0000 7f5a45888700 1 -- start start 2023-12-01T22:13:32.953 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:32.950+0000 7f5a45888700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a400f5410 0x7f5a400fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:13:32.953 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:32.950+0000 7f5a45888700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a400fb820 con 0x7f5a400f5410 2023-12-01T22:13:32.953 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:32.950+0000 7f5a3effd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a400f5410 0x7f5a400fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:13:32.953 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:32.950+0000 7f5a3effd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a400f5410 0x7f5a400fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:42040/0 (socket says 172.21.15.73:42040) 2023-12-01T22:13:32.954 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:32.950+0000 7f5a3effd700 1 -- 172.21.15.73:0/2328803354 learned_addr learned my addr 172.21.15.73:0/2328803354 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:13:32.954 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:32.950+0000 7f5a3effd700 1 -- 172.21.15.73:0/2328803354 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a28005040 con 0x7f5a400f5410 2023-12-01T22:13:32.954 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:32.951+0000 7f5a3effd700 1 --2- 172.21.15.73:0/2328803354 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a400f5410 0x7f5a400fd3a0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f5a28013510 tx=0x7f5a2800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:13:32.954 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:32.951+0000 7f5a377fe700 1 -- 172.21.15.73:0/2328803354 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5a28009320 con 0x7f5a400f5410 2023-12-01T22:13:32.954 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:32.951+0000 7f5a45888700 1 -- 172.21.15.73:0/2328803354 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a400fb9c0 con 0x7f5a400f5410 2023-12-01T22:13:32.954 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:32.951+0000 7f5a377fe700 1 -- 172.21.15.73:0/2328803354 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5a28013f00 con 0x7f5a400f5410 2023-12-01T22:13:32.954 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:32.951+0000 7f5a377fe700 1 -- 172.21.15.73:0/2328803354 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5a28014d90 con 0x7f5a400f5410 2023-12-01T22:13:32.954 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:32.951+0000 7f5a45888700 1 -- 172.21.15.73:0/2328803354 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a400fbe50 con 0x7f5a400f5410 2023-12-01T22:13:32.955 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:32.952+0000 7f5a377fe700 1 -- 172.21.15.73:0/2328803354 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5a280263a0 con 0x7f5a400f5410 2023-12-01T22:13:32.955 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:32.952+0000 7f5a377fe700 1 --2- 172.21.15.73:0/2328803354 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f5a2c02ff60 0x7f5a2c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:13:32.955 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:32.952+0000 7f5a377fe700 1 -- 172.21.15.73:0/2328803354 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5a28040ff0 con 0x7f5a400f5410 2023-12-01T22:13:32.955 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:32.952+0000 7f5a45888700 1 -- 172.21.15.73:0/2328803354 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a40059100 con 0x7f5a400f5410 2023-12-01T22:13:32.955 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:32.952+0000 7f5a3e7fc700 1 --2- 172.21.15.73:0/2328803354 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f5a2c02ff60 0x7f5a2c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:13:32.955 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:32.952+0000 7f5a3e7fc700 1 --2- 172.21.15.73:0/2328803354 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f5a2c02ff60 0x7f5a2c032420 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f5a300096a0 tx=0x7f5a30006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:13:32.958 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:32.957+0000 7f5a377fe700 1 -- 172.21.15.73:0/2328803354 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5a28012920 con 0x7f5a400f5410 2023-12-01T22:13:33.167 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:33.166+0000 7f5a45888700 1 -- 172.21.15.73:0/2328803354 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5a40045ad0 con 0x7f5a400f5410 2023-12-01T22:13:33.168 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:33.167+0000 7f5a377fe700 1 -- 172.21.15.73:0/2328803354 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5a2802b020 con 0x7f5a400f5410 2023-12-01T22:13:33.168 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:13:33.169 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:13:33.170 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:33.168+0000 7f5a45888700 1 -- 172.21.15.73:0/2328803354 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f5a2c02ff60 msgr2=0x7f5a2c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:13:33.170 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:33.168+0000 7f5a45888700 1 --2- 172.21.15.73:0/2328803354 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f5a2c02ff60 0x7f5a2c032420 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f5a300096a0 tx=0x7f5a30006b40).stop 2023-12-01T22:13:33.170 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:33.168+0000 7f5a45888700 1 -- 172.21.15.73:0/2328803354 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a400f5410 msgr2=0x7f5a400fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:13:33.170 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:33.168+0000 7f5a45888700 1 --2- 172.21.15.73:0/2328803354 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a400f5410 0x7f5a400fd3a0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f5a28013510 tx=0x7f5a2800b040).stop 2023-12-01T22:13:33.170 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:33.168+0000 7f5a45888700 1 -- 172.21.15.73:0/2328803354 shutdown_connections 2023-12-01T22:13:33.170 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:33.168+0000 7f5a45888700 1 --2- 172.21.15.73:0/2328803354 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f5a2c02ff60 0x7f5a2c032420 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:13:33.170 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:33.168+0000 7f5a45888700 1 --2- 172.21.15.73:0/2328803354 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a400f5410 0x7f5a400fd3a0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:13:33.171 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:33.168+0000 7f5a45888700 1 -- 172.21.15.73:0/2328803354 >> 172.21.15.73:0/2328803354 conn(0x7f5a400f0b70 msgr2=0x7f5a4005fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:13:33.171 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:33.168+0000 7f5a45888700 1 -- 172.21.15.73:0/2328803354 shutdown_connections 2023-12-01T22:13:33.171 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:33.168+0000 7f5a45888700 1 -- 172.21.15.73:0/2328803354 wait complete. 2023-12-01T22:13:33.171 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:13:34.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:13:33 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:13:32.639388+0000 mgr.smithi029.loxrfu (mgr.14164) 71 : cluster [DBG] pgmap v64: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:13:34.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:13:33 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:13:33.167688+0000 mon.smithi029 (mon.0) 161 : audit [DBG] from='client.? 172.21.15.73:0/2328803354' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:13:34.577 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:13:34.577 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:13:35.797 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:35.794+0000 7fe7b7790700 1 -- 172.21.15.73:0/1651079039 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe7b00f5410 msgr2=0x7fe7b00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:13:35.797 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:35.794+0000 7fe7b7790700 1 --2- 172.21.15.73:0/1651079039 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe7b00f5410 0x7fe7b00f5830 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7fe7a0009fb0 tx=0x7fe7a0009b30).stop 2023-12-01T22:13:35.798 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:35.794+0000 7fe7b7790700 1 -- 172.21.15.73:0/1651079039 shutdown_connections 2023-12-01T22:13:35.798 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:35.794+0000 7fe7b7790700 1 --2- 172.21.15.73:0/1651079039 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe7b00f5410 0x7fe7b00f5830 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:13:35.798 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:35.794+0000 7fe7b7790700 1 -- 172.21.15.73:0/1651079039 >> 172.21.15.73:0/1651079039 conn(0x7fe7b00f0b70 msgr2=0x7fe7b00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:13:35.798 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:35.794+0000 7fe7b7790700 1 -- 172.21.15.73:0/1651079039 shutdown_connections 2023-12-01T22:13:35.798 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:35.794+0000 7fe7b7790700 1 -- 172.21.15.73:0/1651079039 wait complete. 2023-12-01T22:13:35.798 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:35.795+0000 7fe7b7790700 1 Processor -- start 2023-12-01T22:13:35.798 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:35.795+0000 7fe7b7790700 1 -- start start 2023-12-01T22:13:35.798 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:35.795+0000 7fe7b7790700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe7b00f5410 0x7fe7b00fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:13:35.798 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:35.795+0000 7fe7b7790700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7b00fb820 con 0x7fe7b00f5410 2023-12-01T22:13:35.798 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:35.795+0000 7fe7b552c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe7b00f5410 0x7fe7b00fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:13:35.798 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:35.795+0000 7fe7b552c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe7b00f5410 0x7fe7b00fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:44742/0 (socket says 172.21.15.73:44742) 2023-12-01T22:13:35.799 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:35.795+0000 7fe7b552c700 1 -- 172.21.15.73:0/3076512855 learned_addr learned my addr 172.21.15.73:0/3076512855 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:13:35.799 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:35.796+0000 7fe7b552c700 1 -- 172.21.15.73:0/3076512855 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7a0005040 con 0x7fe7b00f5410 2023-12-01T22:13:35.799 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:35.796+0000 7fe7b552c700 1 --2- 172.21.15.73:0/3076512855 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe7b00f5410 0x7fe7b00fd3a0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7fe7a0014000 tx=0x7fe7a000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:13:35.800 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:35.796+0000 7fe7a5ffb700 1 -- 172.21.15.73:0/3076512855 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe7a00095f0 con 0x7fe7b00f5410 2023-12-01T22:13:35.800 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:35.796+0000 7fe7a5ffb700 1 -- 172.21.15.73:0/3076512855 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe7a0007e70 con 0x7fe7b00f5410 2023-12-01T22:13:35.800 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:35.796+0000 7fe7b7790700 1 -- 172.21.15.73:0/3076512855 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7b00fb9c0 con 0x7fe7b00f5410 2023-12-01T22:13:35.800 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:35.796+0000 7fe7a5ffb700 1 -- 172.21.15.73:0/3076512855 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe7a0014ea0 con 0x7fe7b00f5410 2023-12-01T22:13:35.800 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:35.796+0000 7fe7b7790700 1 -- 172.21.15.73:0/3076512855 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7b00fbe50 con 0x7fe7b00f5410 2023-12-01T22:13:35.800 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:35.797+0000 7fe7a5ffb700 1 -- 172.21.15.73:0/3076512855 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe7a0025070 con 0x7fe7b00f5410 2023-12-01T22:13:35.801 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:35.797+0000 7fe7b7790700 1 -- 172.21.15.73:0/3076512855 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7b0059100 con 0x7fe7b00f5410 2023-12-01T22:13:35.801 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:35.797+0000 7fe7a5ffb700 1 --2- 172.21.15.73:0/3076512855 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fe79c02ff60 0x7fe79c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:13:35.801 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:35.797+0000 7fe7a5ffb700 1 -- 172.21.15.73:0/3076512855 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe7a0040940 con 0x7fe7b00f5410 2023-12-01T22:13:35.801 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:35.797+0000 7fe7b4d2b700 1 --2- 172.21.15.73:0/3076512855 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fe79c02ff60 0x7fe79c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:13:35.801 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:35.798+0000 7fe7b4d2b700 1 --2- 172.21.15.73:0/3076512855 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fe79c02ff60 0x7fe79c032420 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7fe7ac0096a0 tx=0x7fe7ac006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:13:35.803 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:35.803+0000 7fe7a5ffb700 1 -- 172.21.15.73:0/3076512855 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe7a00079c0 con 0x7fe7b00f5410 2023-12-01T22:13:36.009 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:36.007+0000 7fe7b7790700 1 -- 172.21.15.73:0/3076512855 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe7b00fae60 con 0x7fe7b00f5410 2023-12-01T22:13:36.010 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:36.008+0000 7fe7a5ffb700 1 -- 172.21.15.73:0/3076512855 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe7a000f300 con 0x7fe7b00f5410 2023-12-01T22:13:36.010 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:13:36.010 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:13:36.012 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:36.010+0000 7fe7b7790700 1 -- 172.21.15.73:0/3076512855 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fe79c02ff60 msgr2=0x7fe79c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:13:36.012 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:36.010+0000 7fe7b7790700 1 --2- 172.21.15.73:0/3076512855 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fe79c02ff60 0x7fe79c032420 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7fe7ac0096a0 tx=0x7fe7ac006b40).stop 2023-12-01T22:13:36.012 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:36.010+0000 7fe7b7790700 1 -- 172.21.15.73:0/3076512855 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe7b00f5410 msgr2=0x7fe7b00fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:13:36.012 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:36.010+0000 7fe7b7790700 1 --2- 172.21.15.73:0/3076512855 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe7b00f5410 0x7fe7b00fd3a0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7fe7a0014000 tx=0x7fe7a000b040).stop 2023-12-01T22:13:36.012 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:36.010+0000 7fe7b7790700 1 -- 172.21.15.73:0/3076512855 shutdown_connections 2023-12-01T22:13:36.012 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:36.010+0000 7fe7b7790700 1 --2- 172.21.15.73:0/3076512855 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fe79c02ff60 0x7fe79c032420 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:13:36.012 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:36.010+0000 7fe7b7790700 1 --2- 172.21.15.73:0/3076512855 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe7b00f5410 0x7fe7b00fd3a0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:13:36.012 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:36.010+0000 7fe7b7790700 1 -- 172.21.15.73:0/3076512855 >> 172.21.15.73:0/3076512855 conn(0x7fe7b00f0b70 msgr2=0x7fe7b005fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:13:36.013 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:36.010+0000 7fe7b7790700 1 -- 172.21.15.73:0/3076512855 shutdown_connections 2023-12-01T22:13:36.013 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:36.010+0000 7fe7b7790700 1 -- 172.21.15.73:0/3076512855 wait complete. 2023-12-01T22:13:36.013 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:13:36.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:13:35 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:13:34.639771+0000 mgr.smithi029.loxrfu (mgr.14164) 72 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:13:37.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:13:36 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:13:36.009534+0000 mon.smithi029 (mon.0) 162 : audit [DBG] from='client.? 172.21.15.73:0/3076512855' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:13:37.412 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:13:37.413 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:13:38.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:13:37 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:13:36.640113+0000 mgr.smithi029.loxrfu (mgr.14164) 73 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:13:38.605 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:38.603+0000 7f9243417700 1 -- 172.21.15.73:0/4024710349 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f923c060180 msgr2=0x7f923c0f8f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:13:38.605 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:38.603+0000 7f9243417700 1 --2- 172.21.15.73:0/4024710349 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f923c060180 0x7f923c0f8f80 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f9238009fb0 tx=0x7f9238009b30).stop 2023-12-01T22:13:38.605 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:38.603+0000 7f9243417700 1 -- 172.21.15.73:0/4024710349 shutdown_connections 2023-12-01T22:13:38.605 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:38.603+0000 7f9243417700 1 --2- 172.21.15.73:0/4024710349 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f923c060180 0x7f923c0f8f80 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:13:38.605 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:38.603+0000 7f9243417700 1 -- 172.21.15.73:0/4024710349 >> 172.21.15.73:0/4024710349 conn(0x7f923c0f09a0 msgr2=0x7f923c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:13:38.605 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:38.603+0000 7f9243417700 1 -- 172.21.15.73:0/4024710349 shutdown_connections 2023-12-01T22:13:38.605 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:38.603+0000 7f9243417700 1 -- 172.21.15.73:0/4024710349 wait complete. 2023-12-01T22:13:38.605 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:38.604+0000 7f9243417700 1 Processor -- start 2023-12-01T22:13:38.605 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:38.604+0000 7f9243417700 1 -- start start 2023-12-01T22:13:38.605 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:38.604+0000 7f9243417700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f923c060180 0x7f923c106500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:13:38.606 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:38.604+0000 7f9243417700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f923c1069f0 con 0x7f923c060180 2023-12-01T22:13:38.606 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:38.604+0000 7f9242415700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f923c060180 0x7f923c106500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:13:38.606 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:38.604+0000 7f9242415700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f923c060180 0x7f923c106500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:44772/0 (socket says 172.21.15.73:44772) 2023-12-01T22:13:38.606 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:38.604+0000 7f9242415700 1 -- 172.21.15.73:0/2046524109 learned_addr learned my addr 172.21.15.73:0/2046524109 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:13:38.607 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:38.605+0000 7f9242415700 1 -- 172.21.15.73:0/2046524109 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9238005040 con 0x7f923c060180 2023-12-01T22:13:38.607 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:38.605+0000 7f9242415700 1 --2- 172.21.15.73:0/2046524109 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f923c060180 0x7f923c106500 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f9238013510 tx=0x7f923800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:13:38.607 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:38.605+0000 7f9232ffd700 1 -- 172.21.15.73:0/2046524109 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9238009320 con 0x7f923c060180 2023-12-01T22:13:38.607 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:38.605+0000 7f9243417700 1 -- 172.21.15.73:0/2046524109 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f923c106bf0 con 0x7f923c060180 2023-12-01T22:13:38.607 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:38.605+0000 7f9232ffd700 1 -- 172.21.15.73:0/2046524109 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9238013f00 con 0x7f923c060180 2023-12-01T22:13:38.607 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:38.605+0000 7f9232ffd700 1 -- 172.21.15.73:0/2046524109 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9238014d90 con 0x7f923c060180 2023-12-01T22:13:38.608 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:38.605+0000 7f9243417700 1 -- 172.21.15.73:0/2046524109 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f923c107080 con 0x7f923c060180 2023-12-01T22:13:38.608 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:38.606+0000 7f9243417700 1 -- 172.21.15.73:0/2046524109 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f923c045ad0 con 0x7f923c060180 2023-12-01T22:13:38.608 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:38.606+0000 7f9232ffd700 1 -- 172.21.15.73:0/2046524109 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f92380095f0 con 0x7f923c060180 2023-12-01T22:13:38.608 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:38.606+0000 7f9232ffd700 1 --2- 172.21.15.73:0/2046524109 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f922802ff60 0x7f9228032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:13:38.608 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:38.606+0000 7f9232ffd700 1 -- 172.21.15.73:0/2046524109 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9238040fe0 con 0x7f923c060180 2023-12-01T22:13:38.608 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:38.606+0000 7f9241c14700 1 --2- 172.21.15.73:0/2046524109 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f922802ff60 0x7f9228032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:13:38.608 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:38.606+0000 7f9241c14700 1 --2- 172.21.15.73:0/2046524109 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f922802ff60 0x7f9228032420 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f922c0096a0 tx=0x7f922c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:13:38.611 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:38.610+0000 7f9232ffd700 1 -- 172.21.15.73:0/2046524109 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f923801d7f0 con 0x7f923c060180 2023-12-01T22:13:38.815 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:38.814+0000 7f9243417700 1 -- 172.21.15.73:0/2046524109 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f923c107f30 con 0x7f923c060180 2023-12-01T22:13:38.816 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:38.815+0000 7f9232ffd700 1 -- 172.21.15.73:0/2046524109 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9238010230 con 0x7f923c060180 2023-12-01T22:13:38.816 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:13:38.816 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:13:38.817 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:38.817+0000 7f9243417700 1 -- 172.21.15.73:0/2046524109 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f922802ff60 msgr2=0x7f9228032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:13:38.817 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:38.817+0000 7f9243417700 1 --2- 172.21.15.73:0/2046524109 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f922802ff60 0x7f9228032420 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f922c0096a0 tx=0x7f922c006b40).stop 2023-12-01T22:13:38.818 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:38.817+0000 7f9243417700 1 -- 172.21.15.73:0/2046524109 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f923c060180 msgr2=0x7f923c106500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:13:38.818 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:38.817+0000 7f9243417700 1 --2- 172.21.15.73:0/2046524109 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f923c060180 0x7f923c106500 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f9238013510 tx=0x7f923800b040).stop 2023-12-01T22:13:38.818 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:38.817+0000 7f9243417700 1 -- 172.21.15.73:0/2046524109 shutdown_connections 2023-12-01T22:13:38.818 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:38.817+0000 7f9243417700 1 --2- 172.21.15.73:0/2046524109 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f922802ff60 0x7f9228032420 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:13:38.818 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:38.817+0000 7f9243417700 1 --2- 172.21.15.73:0/2046524109 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f923c060180 0x7f923c106500 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:13:38.818 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:38.817+0000 7f9243417700 1 -- 172.21.15.73:0/2046524109 >> 172.21.15.73:0/2046524109 conn(0x7f923c0f09a0 msgr2=0x7f923c0f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:13:38.818 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:38.817+0000 7f9243417700 1 -- 172.21.15.73:0/2046524109 shutdown_connections 2023-12-01T22:13:38.818 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:38.817+0000 7f9243417700 1 -- 172.21.15.73:0/2046524109 wait complete. 2023-12-01T22:13:38.818 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:13:40.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:13:39 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:13:38.640443+0000 mgr.smithi029.loxrfu (mgr.14164) 74 : cluster [DBG] pgmap v67: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:13:40.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:13:39 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:13:38.815404+0000 mon.smithi029 (mon.0) 163 : audit [DBG] from='client.? 172.21.15.73:0/2046524109' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:13:40.220 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:13:40.221 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:13:41.325 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:41.324+0000 7fcc5e991700 1 -- 172.21.15.73:0/2075030590 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc580f5410 msgr2=0x7fcc580f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:13:41.325 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:41.324+0000 7fcc5e991700 1 --2- 172.21.15.73:0/2075030590 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc580f5410 0x7fcc580f5830 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7fcc48009fb0 tx=0x7fcc48009b30).stop 2023-12-01T22:13:41.325 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:41.324+0000 7fcc5e991700 1 -- 172.21.15.73:0/2075030590 shutdown_connections 2023-12-01T22:13:41.325 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:41.324+0000 7fcc5e991700 1 --2- 172.21.15.73:0/2075030590 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc580f5410 0x7fcc580f5830 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:13:41.325 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:41.324+0000 7fcc5e991700 1 -- 172.21.15.73:0/2075030590 >> 172.21.15.73:0/2075030590 conn(0x7fcc580f0b70 msgr2=0x7fcc580f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:13:41.325 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:41.324+0000 7fcc5e991700 1 -- 172.21.15.73:0/2075030590 shutdown_connections 2023-12-01T22:13:41.325 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:41.324+0000 7fcc5e991700 1 -- 172.21.15.73:0/2075030590 wait complete. 2023-12-01T22:13:41.325 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:41.324+0000 7fcc5e991700 1 Processor -- start 2023-12-01T22:13:41.325 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:41.324+0000 7fcc5e991700 1 -- start start 2023-12-01T22:13:41.327 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:41.324+0000 7fcc5e991700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc580f5410 0x7fcc58109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:13:41.327 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:41.324+0000 7fcc5e991700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc58060220 con 0x7fcc580f5410 2023-12-01T22:13:41.327 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:41.325+0000 7fcc57fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc580f5410 0x7fcc58109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:13:41.327 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:41.325+0000 7fcc57fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc580f5410 0x7fcc58109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:44794/0 (socket says 172.21.15.73:44794) 2023-12-01T22:13:41.327 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:41.325+0000 7fcc57fff700 1 -- 172.21.15.73:0/819784401 learned_addr learned my addr 172.21.15.73:0/819784401 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:13:41.327 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:41.325+0000 7fcc57fff700 1 -- 172.21.15.73:0/819784401 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc48005040 con 0x7fcc580f5410 2023-12-01T22:13:41.327 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:41.325+0000 7fcc57fff700 1 --2- 172.21.15.73:0/819784401 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc580f5410 0x7fcc58109450 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7fcc48013510 tx=0x7fcc4800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:13:41.327 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:41.325+0000 7fcc54ff9700 1 -- 172.21.15.73:0/819784401 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcc48009320 con 0x7fcc580f5410 2023-12-01T22:13:41.328 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:41.325+0000 7fcc5e991700 1 -- 172.21.15.73:0/819784401 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc58060420 con 0x7fcc580f5410 2023-12-01T22:13:41.328 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:41.325+0000 7fcc54ff9700 1 -- 172.21.15.73:0/819784401 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcc48013f00 con 0x7fcc580f5410 2023-12-01T22:13:41.328 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:41.325+0000 7fcc54ff9700 1 -- 172.21.15.73:0/819784401 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcc48014d90 con 0x7fcc580f5410 2023-12-01T22:13:41.328 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:41.325+0000 7fcc5e991700 1 -- 172.21.15.73:0/819784401 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc58109b00 con 0x7fcc580f5410 2023-12-01T22:13:41.328 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:41.326+0000 7fcc54ff9700 1 -- 172.21.15.73:0/819784401 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcc480095f0 con 0x7fcc580f5410 2023-12-01T22:13:41.328 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:41.326+0000 7fcc5e991700 1 -- 172.21.15.73:0/819784401 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc58059100 con 0x7fcc580f5410 2023-12-01T22:13:41.328 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:41.326+0000 7fcc54ff9700 1 --2- 172.21.15.73:0/819784401 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fcc4002ff60 0x7fcc40032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:13:41.328 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:41.326+0000 7fcc54ff9700 1 -- 172.21.15.73:0/819784401 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcc48041090 con 0x7fcc580f5410 2023-12-01T22:13:41.328 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:41.327+0000 7fcc577fe700 1 --2- 172.21.15.73:0/819784401 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fcc4002ff60 0x7fcc40032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:13:41.328 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:41.327+0000 7fcc577fe700 1 --2- 172.21.15.73:0/819784401 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fcc4002ff60 0x7fcc40032420 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7fcc4c0096a0 tx=0x7fcc4c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:13:41.332 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:41.331+0000 7fcc54ff9700 1 -- 172.21.15.73:0/819784401 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcc48014420 con 0x7fcc580f5410 2023-12-01T22:13:41.535 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:41.535+0000 7fcc5e991700 1 -- 172.21.15.73:0/819784401 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcc58045ad0 con 0x7fcc580f5410 2023-12-01T22:13:41.536 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:41.535+0000 7fcc54ff9700 1 -- 172.21.15.73:0/819784401 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fcc4801d7f0 con 0x7fcc580f5410 2023-12-01T22:13:41.536 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:13:41.536 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:13:41.538 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:41.537+0000 7fcc5e991700 1 -- 172.21.15.73:0/819784401 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fcc4002ff60 msgr2=0x7fcc40032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:13:41.538 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:41.537+0000 7fcc5e991700 1 --2- 172.21.15.73:0/819784401 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fcc4002ff60 0x7fcc40032420 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7fcc4c0096a0 tx=0x7fcc4c006b40).stop 2023-12-01T22:13:41.538 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:41.537+0000 7fcc5e991700 1 -- 172.21.15.73:0/819784401 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc580f5410 msgr2=0x7fcc58109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:13:41.538 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:41.537+0000 7fcc5e991700 1 --2- 172.21.15.73:0/819784401 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc580f5410 0x7fcc58109450 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7fcc48013510 tx=0x7fcc4800b040).stop 2023-12-01T22:13:41.538 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:41.537+0000 7fcc5e991700 1 -- 172.21.15.73:0/819784401 shutdown_connections 2023-12-01T22:13:41.538 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:41.537+0000 7fcc5e991700 1 --2- 172.21.15.73:0/819784401 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fcc4002ff60 0x7fcc40032420 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:13:41.538 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:41.537+0000 7fcc5e991700 1 --2- 172.21.15.73:0/819784401 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc580f5410 0x7fcc58109450 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:13:41.538 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:41.537+0000 7fcc5e991700 1 -- 172.21.15.73:0/819784401 >> 172.21.15.73:0/819784401 conn(0x7fcc580f0b70 msgr2=0x7fcc580f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:13:41.538 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:41.537+0000 7fcc5e991700 1 -- 172.21.15.73:0/819784401 shutdown_connections 2023-12-01T22:13:41.539 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:41.537+0000 7fcc5e991700 1 -- 172.21.15.73:0/819784401 wait complete. 2023-12-01T22:13:41.539 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:13:41.776 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:13:41 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:13:40.640722+0000 mgr.smithi029.loxrfu (mgr.14164) 75 : cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:13:42.777 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:13:42 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:13:41.535655+0000 mon.smithi029 (mon.0) 164 : audit [DBG] from='client.? 172.21.15.73:0/819784401' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:13:42.929 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:13:42.930 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:13:43.776 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:13:43 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:13:42.640970+0000 mgr.smithi029.loxrfu (mgr.14164) 76 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:13:44.111 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:44.110+0000 7f9ef091c700 1 -- 172.21.15.73:0/441500629 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9eec0f6510 msgr2=0x7f9eec0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:13:44.112 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:44.110+0000 7f9ef091c700 1 --2- 172.21.15.73:0/441500629 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9eec0f6510 0x7f9eec0f6930 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f9ee0009fb0 tx=0x7f9ee0009b30).stop 2023-12-01T22:13:44.112 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:44.110+0000 7f9ef091c700 1 -- 172.21.15.73:0/441500629 shutdown_connections 2023-12-01T22:13:44.112 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:44.110+0000 7f9ef091c700 1 --2- 172.21.15.73:0/441500629 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9eec0f6510 0x7f9eec0f6930 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:13:44.112 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:44.110+0000 7f9ef091c700 1 -- 172.21.15.73:0/441500629 >> 172.21.15.73:0/441500629 conn(0x7f9eec0f1c70 msgr2=0x7f9eec0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:13:44.112 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:44.110+0000 7f9ef091c700 1 -- 172.21.15.73:0/441500629 shutdown_connections 2023-12-01T22:13:44.112 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:44.110+0000 7f9ef091c700 1 -- 172.21.15.73:0/441500629 wait complete. 2023-12-01T22:13:44.112 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:44.111+0000 7f9ef091c700 1 Processor -- start 2023-12-01T22:13:44.112 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:44.111+0000 7f9ef091c700 1 -- start start 2023-12-01T22:13:44.112 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:44.111+0000 7f9ef091c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9eec0f6510 0x7f9eec106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:13:44.112 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:44.111+0000 7f9ef091c700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9eec106c20 con 0x7f9eec0f6510 2023-12-01T22:13:44.114 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:44.111+0000 7f9eea59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9eec0f6510 0x7f9eec106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:13:44.114 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:44.111+0000 7f9eea59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9eec0f6510 0x7f9eec106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:44812/0 (socket says 172.21.15.73:44812) 2023-12-01T22:13:44.114 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:44.111+0000 7f9eea59c700 1 -- 172.21.15.73:0/1006743801 learned_addr learned my addr 172.21.15.73:0/1006743801 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:13:44.114 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:44.112+0000 7f9eea59c700 1 -- 172.21.15.73:0/1006743801 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ee0005040 con 0x7f9eec0f6510 2023-12-01T22:13:44.114 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:44.112+0000 7f9eea59c700 1 --2- 172.21.15.73:0/1006743801 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9eec0f6510 0x7f9eec106730 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f9ee0013510 tx=0x7f9ee000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:13:44.114 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:44.112+0000 7f9edaffd700 1 -- 172.21.15.73:0/1006743801 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9ee0009320 con 0x7f9eec0f6510 2023-12-01T22:13:44.114 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:44.112+0000 7f9ef091c700 1 -- 172.21.15.73:0/1006743801 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9eec106e20 con 0x7f9eec0f6510 2023-12-01T22:13:44.115 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:44.112+0000 7f9edaffd700 1 -- 172.21.15.73:0/1006743801 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9ee0013f00 con 0x7f9eec0f6510 2023-12-01T22:13:44.115 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:44.112+0000 7f9edaffd700 1 -- 172.21.15.73:0/1006743801 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9ee0014d90 con 0x7f9eec0f6510 2023-12-01T22:13:44.115 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:44.112+0000 7f9ef091c700 1 -- 172.21.15.73:0/1006743801 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9eec1072b0 con 0x7f9eec0f6510 2023-12-01T22:13:44.115 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:44.113+0000 7f9edaffd700 1 -- 172.21.15.73:0/1006743801 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9ee0014420 con 0x7f9eec0f6510 2023-12-01T22:13:44.115 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:44.113+0000 7f9ef091c700 1 -- 172.21.15.73:0/1006743801 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9eec045ad0 con 0x7f9eec0f6510 2023-12-01T22:13:44.115 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:44.113+0000 7f9edaffd700 1 --2- 172.21.15.73:0/1006743801 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f9ed402ff60 0x7f9ed4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:13:44.115 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:44.113+0000 7f9edaffd700 1 -- 172.21.15.73:0/1006743801 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9ee0041080 con 0x7f9eec0f6510 2023-12-01T22:13:44.115 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:44.113+0000 7f9ee9d9b700 1 --2- 172.21.15.73:0/1006743801 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f9ed402ff60 0x7f9ed4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:13:44.115 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:44.114+0000 7f9ee9d9b700 1 --2- 172.21.15.73:0/1006743801 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f9ed402ff60 0x7f9ed4032420 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f9edc0096a0 tx=0x7f9edc006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:13:44.118 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:44.118+0000 7f9edaffd700 1 -- 172.21.15.73:0/1006743801 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9ee00095f0 con 0x7f9eec0f6510 2023-12-01T22:13:44.324 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:44.323+0000 7f9ef091c700 1 -- 172.21.15.73:0/1006743801 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9eec0fbf60 con 0x7f9eec0f6510 2023-12-01T22:13:44.325 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:44.324+0000 7f9edaffd700 1 -- 172.21.15.73:0/1006743801 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9ee001db70 con 0x7f9eec0f6510 2023-12-01T22:13:44.325 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:13:44.325 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:13:44.327 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:44.325+0000 7f9ef091c700 1 -- 172.21.15.73:0/1006743801 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f9ed402ff60 msgr2=0x7f9ed4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:13:44.327 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:44.325+0000 7f9ef091c700 1 --2- 172.21.15.73:0/1006743801 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f9ed402ff60 0x7f9ed4032420 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f9edc0096a0 tx=0x7f9edc006b40).stop 2023-12-01T22:13:44.327 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:44.326+0000 7f9ef091c700 1 -- 172.21.15.73:0/1006743801 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9eec0f6510 msgr2=0x7f9eec106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:13:44.327 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:44.326+0000 7f9ef091c700 1 --2- 172.21.15.73:0/1006743801 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9eec0f6510 0x7f9eec106730 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f9ee0013510 tx=0x7f9ee000b040).stop 2023-12-01T22:13:44.327 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:44.326+0000 7f9ef091c700 1 -- 172.21.15.73:0/1006743801 shutdown_connections 2023-12-01T22:13:44.327 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:44.326+0000 7f9ef091c700 1 --2- 172.21.15.73:0/1006743801 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f9ed402ff60 0x7f9ed4032420 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:13:44.327 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:44.326+0000 7f9ef091c700 1 --2- 172.21.15.73:0/1006743801 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9eec0f6510 0x7f9eec106730 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:13:44.327 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:44.326+0000 7f9ef091c700 1 -- 172.21.15.73:0/1006743801 >> 172.21.15.73:0/1006743801 conn(0x7f9eec0f1c70 msgr2=0x7f9eec0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:13:44.328 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:44.326+0000 7f9ef091c700 1 -- 172.21.15.73:0/1006743801 shutdown_connections 2023-12-01T22:13:44.328 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:44.326+0000 7f9ef091c700 1 -- 172.21.15.73:0/1006743801 wait complete. 2023-12-01T22:13:44.328 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:13:44.776 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:13:44 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:13:44.324419+0000 mon.smithi029 (mon.0) 165 : audit [DBG] from='client.? 172.21.15.73:0/1006743801' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:13:45.726 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:13:45.727 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:13:45.776 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:13:45 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:13:44.641303+0000 mgr.smithi029.loxrfu (mgr.14164) 77 : cluster [DBG] pgmap v70: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:13:46.911 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:46.907+0000 7f05e4b58700 1 -- 172.21.15.73:0/694484047 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05e00f62a0 msgr2=0x7f05e00f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:13:46.913 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:46.907+0000 7f05e4b58700 1 --2- 172.21.15.73:0/694484047 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05e00f62a0 0x7f05e00f66c0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f05d0009fb0 tx=0x7f05d0009b30).stop 2023-12-01T22:13:46.913 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:46.907+0000 7f05e4b58700 1 -- 172.21.15.73:0/694484047 shutdown_connections 2023-12-01T22:13:46.913 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:46.907+0000 7f05e4b58700 1 --2- 172.21.15.73:0/694484047 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05e00f62a0 0x7f05e00f66c0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:13:46.913 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:46.907+0000 7f05e4b58700 1 -- 172.21.15.73:0/694484047 >> 172.21.15.73:0/694484047 conn(0x7f05e00f1aa0 msgr2=0x7f05e00f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:13:46.914 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:46.907+0000 7f05e4b58700 1 -- 172.21.15.73:0/694484047 shutdown_connections 2023-12-01T22:13:46.914 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:46.907+0000 7f05e4b58700 1 -- 172.21.15.73:0/694484047 wait complete. 2023-12-01T22:13:46.914 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:46.907+0000 7f05e4b58700 1 Processor -- start 2023-12-01T22:13:46.914 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:46.908+0000 7f05e4b58700 1 -- start start 2023-12-01T22:13:46.914 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:46.908+0000 7f05e4b58700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05e00f62a0 0x7f05e01064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:13:46.914 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:46.908+0000 7f05e4b58700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05e01069a0 con 0x7f05e00f62a0 2023-12-01T22:13:46.915 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:46.908+0000 7f05df7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05e00f62a0 0x7f05e01064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:13:46.915 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:46.908+0000 7f05df7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05e00f62a0 0x7f05e01064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:39942/0 (socket says 172.21.15.73:39942) 2023-12-01T22:13:46.915 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:46.908+0000 7f05df7fe700 1 -- 172.21.15.73:0/3824855339 learned_addr learned my addr 172.21.15.73:0/3824855339 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:13:46.915 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:46.908+0000 7f05df7fe700 1 -- 172.21.15.73:0/3824855339 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05d0005040 con 0x7f05e00f62a0 2023-12-01T22:13:46.915 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:46.908+0000 7f05df7fe700 1 --2- 172.21.15.73:0/3824855339 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05e00f62a0 0x7f05e01064b0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f05d0009830 tx=0x7f05d000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:13:46.915 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:46.909+0000 7f05c7fff700 1 -- 172.21.15.73:0/3824855339 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f05d0009320 con 0x7f05e00f62a0 2023-12-01T22:13:46.915 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:46.909+0000 7f05e4b58700 1 -- 172.21.15.73:0/3824855339 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05e0106ba0 con 0x7f05e00f62a0 2023-12-01T22:13:46.916 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:46.909+0000 7f05c7fff700 1 -- 172.21.15.73:0/3824855339 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f05d00136c0 con 0x7f05e00f62a0 2023-12-01T22:13:46.916 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:46.909+0000 7f05c7fff700 1 -- 172.21.15.73:0/3824855339 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f05d001ee90 con 0x7f05e00f62a0 2023-12-01T22:13:46.916 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:46.909+0000 7f05e4b58700 1 -- 172.21.15.73:0/3824855339 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05e0107030 con 0x7f05e00f62a0 2023-12-01T22:13:46.916 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:46.909+0000 7f05c7fff700 1 -- 172.21.15.73:0/3824855339 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f05d001e420 con 0x7f05e00f62a0 2023-12-01T22:13:46.916 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:46.909+0000 7f05e4b58700 1 -- 172.21.15.73:0/3824855339 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05e0045ad0 con 0x7f05e00f62a0 2023-12-01T22:13:46.916 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:46.910+0000 7f05c7fff700 1 --2- 172.21.15.73:0/3824855339 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f05c802ff10 0x7f05c80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:13:46.917 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:46.910+0000 7f05c7fff700 1 -- 172.21.15.73:0/3824855339 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f05d0040f70 con 0x7f05e00f62a0 2023-12-01T22:13:46.917 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:46.910+0000 7f05deffd700 1 --2- 172.21.15.73:0/3824855339 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f05c802ff10 0x7f05c80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:13:46.917 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:46.910+0000 7f05deffd700 1 --2- 172.21.15.73:0/3824855339 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f05c802ff10 0x7f05c80323d0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f05d40096a0 tx=0x7f05d4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:13:46.918 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:46.915+0000 7f05c7fff700 1 -- 172.21.15.73:0/3824855339 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f05d006d2b0 con 0x7f05e00f62a0 2023-12-01T22:13:47.123 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:47.122+0000 7f05e4b58700 1 -- 172.21.15.73:0/3824855339 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f05e0107ee0 con 0x7f05e00f62a0 2023-12-01T22:13:47.123 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:47.123+0000 7f05c7fff700 1 -- 172.21.15.73:0/3824855339 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f05d002c2d0 con 0x7f05e00f62a0 2023-12-01T22:13:47.123 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:13:47.124 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:13:47.125 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:47.124+0000 7f05e4b58700 1 -- 172.21.15.73:0/3824855339 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f05c802ff10 msgr2=0x7f05c80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:13:47.125 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:47.124+0000 7f05e4b58700 1 --2- 172.21.15.73:0/3824855339 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f05c802ff10 0x7f05c80323d0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f05d40096a0 tx=0x7f05d4006b40).stop 2023-12-01T22:13:47.125 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:47.124+0000 7f05e4b58700 1 -- 172.21.15.73:0/3824855339 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05e00f62a0 msgr2=0x7f05e01064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:13:47.125 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:47.124+0000 7f05e4b58700 1 --2- 172.21.15.73:0/3824855339 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05e00f62a0 0x7f05e01064b0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f05d0009830 tx=0x7f05d000b040).stop 2023-12-01T22:13:47.125 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:47.124+0000 7f05e4b58700 1 -- 172.21.15.73:0/3824855339 shutdown_connections 2023-12-01T22:13:47.125 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:47.124+0000 7f05e4b58700 1 --2- 172.21.15.73:0/3824855339 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f05c802ff10 0x7f05c80323d0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:13:47.125 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:47.124+0000 7f05e4b58700 1 --2- 172.21.15.73:0/3824855339 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05e00f62a0 0x7f05e01064b0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:13:47.125 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:47.124+0000 7f05e4b58700 1 -- 172.21.15.73:0/3824855339 >> 172.21.15.73:0/3824855339 conn(0x7f05e00f1aa0 msgr2=0x7f05e00f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:13:47.125 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:47.124+0000 7f05e4b58700 1 -- 172.21.15.73:0/3824855339 shutdown_connections 2023-12-01T22:13:47.125 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:47.124+0000 7f05e4b58700 1 -- 172.21.15.73:0/3824855339 wait complete. 2023-12-01T22:13:47.126 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:13:48.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:13:47 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:13:46.641618+0000 mgr.smithi029.loxrfu (mgr.14164) 78 : cluster [DBG] pgmap v71: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:13:48.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:13:47 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:13:47.123075+0000 mon.smithi029 (mon.0) 166 : audit [DBG] from='client.? 172.21.15.73:0/3824855339' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:13:48.522 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:13:48.522 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:13:49.665 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:49.662+0000 7f0a7dac5700 1 -- 172.21.15.73:0/2736098553 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a780f47c0 msgr2=0x7f0a780f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:13:49.665 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:49.662+0000 7f0a7dac5700 1 --2- 172.21.15.73:0/2736098553 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a780f47c0 0x7f0a780f4be0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f0a6c009fb0 tx=0x7f0a6c009b30).stop 2023-12-01T22:13:49.665 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:49.662+0000 7f0a7dac5700 1 -- 172.21.15.73:0/2736098553 shutdown_connections 2023-12-01T22:13:49.665 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:49.662+0000 7f0a7dac5700 1 --2- 172.21.15.73:0/2736098553 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a780f47c0 0x7f0a780f4be0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:13:49.665 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:49.662+0000 7f0a7dac5700 1 -- 172.21.15.73:0/2736098553 >> 172.21.15.73:0/2736098553 conn(0x7f0a780f02e0 msgr2=0x7f0a780f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:13:49.665 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:49.662+0000 7f0a7dac5700 1 -- 172.21.15.73:0/2736098553 shutdown_connections 2023-12-01T22:13:49.665 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:49.662+0000 7f0a7dac5700 1 -- 172.21.15.73:0/2736098553 wait complete. 2023-12-01T22:13:49.665 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:49.663+0000 7f0a7dac5700 1 Processor -- start 2023-12-01T22:13:49.665 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:49.663+0000 7f0a7dac5700 1 -- start start 2023-12-01T22:13:49.665 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:49.663+0000 7f0a7dac5700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a780f47c0 0x7f0a78106700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:13:49.665 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:49.663+0000 7f0a7dac5700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a78106bf0 con 0x7f0a780f47c0 2023-12-01T22:13:49.667 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:49.663+0000 7f0a777fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a780f47c0 0x7f0a78106700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:13:49.667 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:49.663+0000 7f0a777fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a780f47c0 0x7f0a78106700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:39958/0 (socket says 172.21.15.73:39958) 2023-12-01T22:13:49.667 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:49.663+0000 7f0a777fe700 1 -- 172.21.15.73:0/1400345354 learned_addr learned my addr 172.21.15.73:0/1400345354 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:13:49.667 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:49.664+0000 7f0a777fe700 1 -- 172.21.15.73:0/1400345354 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a6c005040 con 0x7f0a780f47c0 2023-12-01T22:13:49.667 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:49.664+0000 7f0a777fe700 1 --2- 172.21.15.73:0/1400345354 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a780f47c0 0x7f0a78106700 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f0a6c013510 tx=0x7f0a6c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:13:49.667 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:49.664+0000 7f0a5ffff700 1 -- 172.21.15.73:0/1400345354 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0a6c009320 con 0x7f0a780f47c0 2023-12-01T22:13:49.667 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:49.664+0000 7f0a7dac5700 1 -- 172.21.15.73:0/1400345354 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a78106df0 con 0x7f0a780f47c0 2023-12-01T22:13:49.668 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:49.664+0000 7f0a5ffff700 1 -- 172.21.15.73:0/1400345354 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0a6c013f00 con 0x7f0a780f47c0 2023-12-01T22:13:49.668 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:49.664+0000 7f0a5ffff700 1 -- 172.21.15.73:0/1400345354 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0a6c014d90 con 0x7f0a780f47c0 2023-12-01T22:13:49.668 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:49.664+0000 7f0a7dac5700 1 -- 172.21.15.73:0/1400345354 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a78107280 con 0x7f0a780f47c0 2023-12-01T22:13:49.668 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:49.665+0000 7f0a5ffff700 1 -- 172.21.15.73:0/1400345354 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0a6c014420 con 0x7f0a780f47c0 2023-12-01T22:13:49.668 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:49.665+0000 7f0a7dac5700 1 -- 172.21.15.73:0/1400345354 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a78045ad0 con 0x7f0a780f47c0 2023-12-01T22:13:49.668 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:49.665+0000 7f0a5ffff700 1 --2- 172.21.15.73:0/1400345354 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f0a6002ff60 0x7f0a60032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:13:49.668 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:49.665+0000 7f0a5ffff700 1 -- 172.21.15.73:0/1400345354 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0a6c041070 con 0x7f0a780f47c0 2023-12-01T22:13:49.668 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:49.665+0000 7f0a76ffd700 1 --2- 172.21.15.73:0/1400345354 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f0a6002ff60 0x7f0a60032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:13:49.668 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:49.665+0000 7f0a76ffd700 1 --2- 172.21.15.73:0/1400345354 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f0a6002ff60 0x7f0a60032420 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f0a680096a0 tx=0x7f0a68006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:13:49.671 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:49.670+0000 7f0a5ffff700 1 -- 172.21.15.73:0/1400345354 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0a6c0095f0 con 0x7f0a780f47c0 2023-12-01T22:13:49.876 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:49.874+0000 7f0a7dac5700 1 -- 172.21.15.73:0/1400345354 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0a78108130 con 0x7f0a780f47c0 2023-12-01T22:13:49.877 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:49.875+0000 7f0a5ffff700 1 -- 172.21.15.73:0/1400345354 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0a6c01db70 con 0x7f0a780f47c0 2023-12-01T22:13:49.877 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:13:49.877 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:13:49.878 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:49.876+0000 7f0a7dac5700 1 -- 172.21.15.73:0/1400345354 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f0a6002ff60 msgr2=0x7f0a60032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:13:49.878 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:49.876+0000 7f0a7dac5700 1 --2- 172.21.15.73:0/1400345354 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f0a6002ff60 0x7f0a60032420 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f0a680096a0 tx=0x7f0a68006b40).stop 2023-12-01T22:13:49.878 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:49.876+0000 7f0a7dac5700 1 -- 172.21.15.73:0/1400345354 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a780f47c0 msgr2=0x7f0a78106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:13:49.879 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:49.876+0000 7f0a7dac5700 1 --2- 172.21.15.73:0/1400345354 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a780f47c0 0x7f0a78106700 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f0a6c013510 tx=0x7f0a6c00b040).stop 2023-12-01T22:13:49.879 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:49.876+0000 7f0a7dac5700 1 -- 172.21.15.73:0/1400345354 shutdown_connections 2023-12-01T22:13:49.879 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:49.876+0000 7f0a7dac5700 1 --2- 172.21.15.73:0/1400345354 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f0a6002ff60 0x7f0a60032420 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:13:49.879 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:49.876+0000 7f0a7dac5700 1 --2- 172.21.15.73:0/1400345354 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a780f47c0 0x7f0a78106700 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:13:49.879 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:49.876+0000 7f0a7dac5700 1 -- 172.21.15.73:0/1400345354 >> 172.21.15.73:0/1400345354 conn(0x7f0a780f02e0 msgr2=0x7f0a780f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:13:49.879 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:49.876+0000 7f0a7dac5700 1 -- 172.21.15.73:0/1400345354 shutdown_connections 2023-12-01T22:13:49.879 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:49.877+0000 7f0a7dac5700 1 -- 172.21.15.73:0/1400345354 wait complete. 2023-12-01T22:13:49.879 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:13:50.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:13:49 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:13:48.641971+0000 mgr.smithi029.loxrfu (mgr.14164) 79 : cluster [DBG] pgmap v72: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:13:51.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:13:50 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:13:49.876283+0000 mon.smithi029 (mon.0) 167 : audit [DBG] from='client.? 172.21.15.73:0/1400345354' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:13:51.247 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:13:51.247 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:13:52.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:13:51 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:13:50.642270+0000 mgr.smithi029.loxrfu (mgr.14164) 80 : cluster [DBG] pgmap v73: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:13:52.403 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:52.399+0000 7ff149893700 1 -- 172.21.15.73:0/3796583817 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1440f5410 msgr2=0x7ff1440f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:13:52.403 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:52.399+0000 7ff149893700 1 --2- 172.21.15.73:0/3796583817 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1440f5410 0x7ff1440f5830 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7ff12c009fb0 tx=0x7ff12c009b30).stop 2023-12-01T22:13:52.403 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:52.399+0000 7ff149893700 1 -- 172.21.15.73:0/3796583817 shutdown_connections 2023-12-01T22:13:52.403 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:52.399+0000 7ff149893700 1 --2- 172.21.15.73:0/3796583817 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1440f5410 0x7ff1440f5830 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:13:52.403 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:52.399+0000 7ff149893700 1 -- 172.21.15.73:0/3796583817 >> 172.21.15.73:0/3796583817 conn(0x7ff1440f0b70 msgr2=0x7ff1440f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:13:52.403 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:52.399+0000 7ff149893700 1 -- 172.21.15.73:0/3796583817 shutdown_connections 2023-12-01T22:13:52.403 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:52.399+0000 7ff149893700 1 -- 172.21.15.73:0/3796583817 wait complete. 2023-12-01T22:13:52.403 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:52.400+0000 7ff149893700 1 Processor -- start 2023-12-01T22:13:52.404 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:52.400+0000 7ff149893700 1 -- start start 2023-12-01T22:13:52.404 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:52.400+0000 7ff149893700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1440f5410 0x7ff1440fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:13:52.404 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:52.400+0000 7ff149893700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1440fb820 con 0x7ff1440f5410 2023-12-01T22:13:52.404 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:52.400+0000 7ff142ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1440f5410 0x7ff1440fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:13:52.404 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:52.401+0000 7ff142ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1440f5410 0x7ff1440fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:39976/0 (socket says 172.21.15.73:39976) 2023-12-01T22:13:52.404 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:52.401+0000 7ff142ffd700 1 -- 172.21.15.73:0/418864356 learned_addr learned my addr 172.21.15.73:0/418864356 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:13:52.404 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:52.401+0000 7ff142ffd700 1 -- 172.21.15.73:0/418864356 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff12c005040 con 0x7ff1440f5410 2023-12-01T22:13:52.404 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:52.401+0000 7ff142ffd700 1 --2- 172.21.15.73:0/418864356 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1440f5410 0x7ff1440fd3a0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7ff12c014000 tx=0x7ff12c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:13:52.404 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:52.401+0000 7ff13b7fe700 1 -- 172.21.15.73:0/418864356 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff12c0095f0 con 0x7ff1440f5410 2023-12-01T22:13:52.404 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:52.401+0000 7ff149893700 1 -- 172.21.15.73:0/418864356 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1440fb9c0 con 0x7ff1440f5410 2023-12-01T22:13:52.404 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:52.401+0000 7ff13b7fe700 1 -- 172.21.15.73:0/418864356 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff12c007e70 con 0x7ff1440f5410 2023-12-01T22:13:52.404 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:52.401+0000 7ff13b7fe700 1 -- 172.21.15.73:0/418864356 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff12c014ea0 con 0x7ff1440f5410 2023-12-01T22:13:52.404 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:52.401+0000 7ff149893700 1 -- 172.21.15.73:0/418864356 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1440fbe50 con 0x7ff1440f5410 2023-12-01T22:13:52.405 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:52.402+0000 7ff13b7fe700 1 -- 172.21.15.73:0/418864356 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff12c025070 con 0x7ff1440f5410 2023-12-01T22:13:52.405 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:52.402+0000 7ff13b7fe700 1 --2- 172.21.15.73:0/418864356 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7ff13002ff60 0x7ff130032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:13:52.405 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:52.402+0000 7ff149893700 1 -- 172.21.15.73:0/418864356 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff144059100 con 0x7ff1440f5410 2023-12-01T22:13:52.406 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:52.402+0000 7ff13b7fe700 1 -- 172.21.15.73:0/418864356 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff12c040930 con 0x7ff1440f5410 2023-12-01T22:13:52.406 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:52.402+0000 7ff1427fc700 1 --2- 172.21.15.73:0/418864356 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7ff13002ff60 0x7ff130032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:13:52.406 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:52.403+0000 7ff1427fc700 1 --2- 172.21.15.73:0/418864356 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7ff13002ff60 0x7ff130032420 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7ff1340096a0 tx=0x7ff134006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:13:52.408 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:52.407+0000 7ff13b7fe700 1 -- 172.21.15.73:0/418864356 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff12c00f4e0 con 0x7ff1440f5410 2023-12-01T22:13:52.617 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:52.615+0000 7ff149893700 1 -- 172.21.15.73:0/418864356 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff1440fae60 con 0x7ff1440f5410 2023-12-01T22:13:52.618 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:52.616+0000 7ff13b7fe700 1 -- 172.21.15.73:0/418864356 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff12c02d090 con 0x7ff1440f5410 2023-12-01T22:13:52.618 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:13:52.618 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:13:52.619 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:52.618+0000 7ff149893700 1 -- 172.21.15.73:0/418864356 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7ff13002ff60 msgr2=0x7ff130032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:13:52.619 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:52.618+0000 7ff149893700 1 --2- 172.21.15.73:0/418864356 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7ff13002ff60 0x7ff130032420 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7ff1340096a0 tx=0x7ff134006b40).stop 2023-12-01T22:13:52.619 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:52.618+0000 7ff149893700 1 -- 172.21.15.73:0/418864356 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1440f5410 msgr2=0x7ff1440fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:13:52.620 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:52.618+0000 7ff149893700 1 --2- 172.21.15.73:0/418864356 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1440f5410 0x7ff1440fd3a0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7ff12c014000 tx=0x7ff12c00b040).stop 2023-12-01T22:13:52.620 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:52.618+0000 7ff149893700 1 -- 172.21.15.73:0/418864356 shutdown_connections 2023-12-01T22:13:52.620 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:52.618+0000 7ff149893700 1 --2- 172.21.15.73:0/418864356 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7ff13002ff60 0x7ff130032420 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:13:52.620 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:52.618+0000 7ff149893700 1 --2- 172.21.15.73:0/418864356 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1440f5410 0x7ff1440fd3a0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:13:52.620 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:52.618+0000 7ff149893700 1 -- 172.21.15.73:0/418864356 >> 172.21.15.73:0/418864356 conn(0x7ff1440f0b70 msgr2=0x7ff14405fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:13:52.620 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:52.618+0000 7ff149893700 1 -- 172.21.15.73:0/418864356 shutdown_connections 2023-12-01T22:13:52.620 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:52.618+0000 7ff149893700 1 -- 172.21.15.73:0/418864356 wait complete. 2023-12-01T22:13:52.620 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:13:53.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:13:52 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:13:52.617402+0000 mon.smithi029 (mon.0) 168 : audit [DBG] from='client.? 172.21.15.73:0/418864356' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:13:54.020 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:13:54.021 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:13:54.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:13:53 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:13:52.642504+0000 mgr.smithi029.loxrfu (mgr.14164) 81 : cluster [DBG] pgmap v74: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:13:55.169 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:55.167+0000 7f32ee0d6700 1 -- 172.21.15.73:0/2973328297 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32e80f51a0 msgr2=0x7f32e80f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:13:55.169 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:55.167+0000 7f32ee0d6700 1 --2- 172.21.15.73:0/2973328297 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32e80f51a0 0x7f32e80f55c0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f32e4009fb0 tx=0x7f32e4009b30).stop 2023-12-01T22:13:55.169 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:55.167+0000 7f32ee0d6700 1 -- 172.21.15.73:0/2973328297 shutdown_connections 2023-12-01T22:13:55.169 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:55.167+0000 7f32ee0d6700 1 --2- 172.21.15.73:0/2973328297 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32e80f51a0 0x7f32e80f55c0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:13:55.169 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:55.167+0000 7f32ee0d6700 1 -- 172.21.15.73:0/2973328297 >> 172.21.15.73:0/2973328297 conn(0x7f32e80f09a0 msgr2=0x7f32e80f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:13:55.169 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:55.167+0000 7f32ee0d6700 1 -- 172.21.15.73:0/2973328297 shutdown_connections 2023-12-01T22:13:55.169 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:55.167+0000 7f32ee0d6700 1 -- 172.21.15.73:0/2973328297 wait complete. 2023-12-01T22:13:55.171 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:55.167+0000 7f32ee0d6700 1 Processor -- start 2023-12-01T22:13:55.171 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:55.168+0000 7f32ee0d6700 1 -- start start 2023-12-01T22:13:55.171 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:55.168+0000 7f32ee0d6700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32e80f51a0 0x7f32e81053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:13:55.171 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:55.168+0000 7f32ee0d6700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32e81058a0 con 0x7f32e80f51a0 2023-12-01T22:13:55.171 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:55.168+0000 7f32ed0d4700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32e80f51a0 0x7f32e81053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:13:55.171 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:55.168+0000 7f32ed0d4700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32e80f51a0 0x7f32e81053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:37340/0 (socket says 172.21.15.73:37340) 2023-12-01T22:13:55.171 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:55.168+0000 7f32ed0d4700 1 -- 172.21.15.73:0/3343754868 learned_addr learned my addr 172.21.15.73:0/3343754868 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:13:55.171 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:55.168+0000 7f32ed0d4700 1 -- 172.21.15.73:0/3343754868 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32e4005040 con 0x7f32e80f51a0 2023-12-01T22:13:55.171 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:55.169+0000 7f32ed0d4700 1 --2- 172.21.15.73:0/3343754868 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32e80f51a0 0x7f32e81053b0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f32e4013510 tx=0x7f32e400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:13:55.171 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:55.169+0000 7f32d9ffb700 1 -- 172.21.15.73:0/3343754868 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f32e4009320 con 0x7f32e80f51a0 2023-12-01T22:13:55.171 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:55.169+0000 7f32ee0d6700 1 -- 172.21.15.73:0/3343754868 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32e8105aa0 con 0x7f32e80f51a0 2023-12-01T22:13:55.171 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:55.169+0000 7f32d9ffb700 1 -- 172.21.15.73:0/3343754868 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f32e4013f00 con 0x7f32e80f51a0 2023-12-01T22:13:55.172 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:55.169+0000 7f32d9ffb700 1 -- 172.21.15.73:0/3343754868 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f32e4014dc0 con 0x7f32e80f51a0 2023-12-01T22:13:55.172 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:55.169+0000 7f32ee0d6700 1 -- 172.21.15.73:0/3343754868 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32e8105f30 con 0x7f32e80f51a0 2023-12-01T22:13:55.172 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:55.170+0000 7f32d9ffb700 1 -- 172.21.15.73:0/3343754868 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f32e40095f0 con 0x7f32e80f51a0 2023-12-01T22:13:55.172 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:55.170+0000 7f32ee0d6700 1 -- 172.21.15.73:0/3343754868 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32e8045ad0 con 0x7f32e80f51a0 2023-12-01T22:13:55.173 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:55.170+0000 7f32d9ffb700 1 --2- 172.21.15.73:0/3343754868 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f32d402ff60 0x7f32d4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:13:55.173 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:55.170+0000 7f32d9ffb700 1 -- 172.21.15.73:0/3343754868 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f32e4042ea0 con 0x7f32e80f51a0 2023-12-01T22:13:55.173 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:55.170+0000 7f32ec8d3700 1 --2- 172.21.15.73:0/3343754868 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f32d402ff60 0x7f32d4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:13:55.173 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:55.170+0000 7f32ec8d3700 1 --2- 172.21.15.73:0/3343754868 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f32d402ff60 0x7f32d4032420 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f32dc0096a0 tx=0x7f32dc006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:13:55.176 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:55.175+0000 7f32d9ffb700 1 -- 172.21.15.73:0/3343754868 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f32e4014420 con 0x7f32e80f51a0 2023-12-01T22:13:55.385 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:55.383+0000 7f32ee0d6700 1 -- 172.21.15.73:0/3343754868 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f32e8106de0 con 0x7f32e80f51a0 2023-12-01T22:13:55.385 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:55.384+0000 7f32d9ffb700 1 -- 172.21.15.73:0/3343754868 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f32e401f7f0 con 0x7f32e80f51a0 2023-12-01T22:13:55.386 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:13:55.386 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:13:55.387 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:55.385+0000 7f32ee0d6700 1 -- 172.21.15.73:0/3343754868 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f32d402ff60 msgr2=0x7f32d4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:13:55.388 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:55.385+0000 7f32ee0d6700 1 --2- 172.21.15.73:0/3343754868 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f32d402ff60 0x7f32d4032420 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f32dc0096a0 tx=0x7f32dc006b40).stop 2023-12-01T22:13:55.388 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:55.385+0000 7f32ee0d6700 1 -- 172.21.15.73:0/3343754868 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32e80f51a0 msgr2=0x7f32e81053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:13:55.388 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:55.385+0000 7f32ee0d6700 1 --2- 172.21.15.73:0/3343754868 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32e80f51a0 0x7f32e81053b0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f32e4013510 tx=0x7f32e400b040).stop 2023-12-01T22:13:55.388 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:55.386+0000 7f32ee0d6700 1 -- 172.21.15.73:0/3343754868 shutdown_connections 2023-12-01T22:13:55.388 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:55.386+0000 7f32ee0d6700 1 --2- 172.21.15.73:0/3343754868 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f32d402ff60 0x7f32d4032420 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:13:55.388 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:55.386+0000 7f32ee0d6700 1 --2- 172.21.15.73:0/3343754868 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32e80f51a0 0x7f32e81053b0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:13:55.388 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:55.386+0000 7f32ee0d6700 1 -- 172.21.15.73:0/3343754868 >> 172.21.15.73:0/3343754868 conn(0x7f32e80f09a0 msgr2=0x7f32e80f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:13:55.388 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:55.386+0000 7f32ee0d6700 1 -- 172.21.15.73:0/3343754868 shutdown_connections 2023-12-01T22:13:55.388 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:55.386+0000 7f32ee0d6700 1 -- 172.21.15.73:0/3343754868 wait complete. 2023-12-01T22:13:55.389 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:13:56.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:13:55 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:13:54.642881+0000 mgr.smithi029.loxrfu (mgr.14164) 82 : cluster [DBG] pgmap v75: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:13:56.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:13:55 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:13:55.385232+0000 mon.smithi029 (mon.0) 169 : audit [DBG] from='client.? 172.21.15.73:0/3343754868' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:13:56.783 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:13:56.784 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:13:57.929 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:57.925+0000 7f3541928700 1 -- 172.21.15.73:0/637882211 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f353c0f5410 msgr2=0x7f353c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:13:57.929 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:57.925+0000 7f3541928700 1 --2- 172.21.15.73:0/637882211 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f353c0f5410 0x7f353c0f5830 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f3524009fb0 tx=0x7f3524009b30).stop 2023-12-01T22:13:57.929 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:57.925+0000 7f3541928700 1 -- 172.21.15.73:0/637882211 shutdown_connections 2023-12-01T22:13:57.929 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:57.925+0000 7f3541928700 1 --2- 172.21.15.73:0/637882211 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f353c0f5410 0x7f353c0f5830 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:13:57.929 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:57.925+0000 7f3541928700 1 -- 172.21.15.73:0/637882211 >> 172.21.15.73:0/637882211 conn(0x7f353c0f0b70 msgr2=0x7f353c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:13:57.929 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:57.925+0000 7f3541928700 1 -- 172.21.15.73:0/637882211 shutdown_connections 2023-12-01T22:13:57.929 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:57.925+0000 7f3541928700 1 -- 172.21.15.73:0/637882211 wait complete. 2023-12-01T22:13:57.929 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:57.926+0000 7f3541928700 1 Processor -- start 2023-12-01T22:13:57.929 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:57.926+0000 7f3541928700 1 -- start start 2023-12-01T22:13:57.930 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:57.926+0000 7f3541928700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f353c105720 0x7f353c105b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:13:57.930 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:57.926+0000 7f3541928700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f353c106030 con 0x7f353c105720 2023-12-01T22:13:57.930 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:57.926+0000 7f353affd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f353c105720 0x7f353c105b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:13:57.930 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:57.926+0000 7f353affd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f353c105720 0x7f353c105b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:37358/0 (socket says 172.21.15.73:37358) 2023-12-01T22:13:57.930 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:57.926+0000 7f353affd700 1 -- 172.21.15.73:0/2739969914 learned_addr learned my addr 172.21.15.73:0/2739969914 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:13:57.930 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:57.927+0000 7f353affd700 1 -- 172.21.15.73:0/2739969914 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3524005040 con 0x7f353c105720 2023-12-01T22:13:57.930 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:57.927+0000 7f353affd700 1 --2- 172.21.15.73:0/2739969914 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f353c105720 0x7f353c105b40 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f3524009f80 tx=0x7f3524009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:13:57.930 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:57.927+0000 7f35337fe700 1 -- 172.21.15.73:0/2739969914 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f35240095f0 con 0x7f353c105720 2023-12-01T22:13:57.930 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:57.927+0000 7f35337fe700 1 -- 172.21.15.73:0/2739969914 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3524007e70 con 0x7f353c105720 2023-12-01T22:13:57.930 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:57.927+0000 7f3541928700 1 -- 172.21.15.73:0/2739969914 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f353c106230 con 0x7f353c105720 2023-12-01T22:13:57.930 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:57.927+0000 7f35337fe700 1 -- 172.21.15.73:0/2739969914 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f352401fd90 con 0x7f353c105720 2023-12-01T22:13:57.930 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:57.927+0000 7f3541928700 1 -- 172.21.15.73:0/2739969914 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f353c106d80 con 0x7f353c105720 2023-12-01T22:13:57.931 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:57.928+0000 7f35337fe700 1 -- 172.21.15.73:0/2739969914 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f35240079c0 con 0x7f353c105720 2023-12-01T22:13:57.931 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:57.928+0000 7f3541928700 1 -- 172.21.15.73:0/2739969914 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f353c059100 con 0x7f353c105720 2023-12-01T22:13:57.931 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:57.928+0000 7f35337fe700 1 --2- 172.21.15.73:0/2739969914 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f352802ff60 0x7f3528032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:13:57.931 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:57.928+0000 7f35337fe700 1 -- 172.21.15.73:0/2739969914 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3524043310 con 0x7f353c105720 2023-12-01T22:13:57.931 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:57.928+0000 7f353a7fc700 1 --2- 172.21.15.73:0/2739969914 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f352802ff60 0x7f3528032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:13:57.932 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:57.928+0000 7f353a7fc700 1 --2- 172.21.15.73:0/2739969914 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f352802ff60 0x7f3528032420 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f352c0096a0 tx=0x7f352c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:13:57.934 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:57.933+0000 7f35337fe700 1 -- 172.21.15.73:0/2739969914 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f352401f590 con 0x7f353c105720 2023-12-01T22:13:58.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:13:57 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:13:56.643239+0000 mgr.smithi029.loxrfu (mgr.14164) 83 : cluster [DBG] pgmap v76: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:13:58.143 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:58.142+0000 7f3541928700 1 -- 172.21.15.73:0/2739969914 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f353c045ad0 con 0x7f353c105720 2023-12-01T22:13:58.144 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:58.142+0000 7f35337fe700 1 -- 172.21.15.73:0/2739969914 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3524027390 con 0x7f353c105720 2023-12-01T22:13:58.144 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:13:58.144 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:13:58.146 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:58.144+0000 7f3541928700 1 -- 172.21.15.73:0/2739969914 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f352802ff60 msgr2=0x7f3528032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:13:58.146 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:58.144+0000 7f3541928700 1 --2- 172.21.15.73:0/2739969914 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f352802ff60 0x7f3528032420 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f352c0096a0 tx=0x7f352c006b40).stop 2023-12-01T22:13:58.146 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:58.144+0000 7f3541928700 1 -- 172.21.15.73:0/2739969914 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f353c105720 msgr2=0x7f353c105b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:13:58.146 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:58.144+0000 7f3541928700 1 --2- 172.21.15.73:0/2739969914 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f353c105720 0x7f353c105b40 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f3524009f80 tx=0x7f3524009b30).stop 2023-12-01T22:13:58.146 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:58.144+0000 7f3541928700 1 -- 172.21.15.73:0/2739969914 shutdown_connections 2023-12-01T22:13:58.146 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:58.144+0000 7f3541928700 1 --2- 172.21.15.73:0/2739969914 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f352802ff60 0x7f3528032420 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:13:58.146 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:58.144+0000 7f3541928700 1 --2- 172.21.15.73:0/2739969914 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f353c105720 0x7f353c105b40 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:13:58.146 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:58.144+0000 7f3541928700 1 -- 172.21.15.73:0/2739969914 >> 172.21.15.73:0/2739969914 conn(0x7f353c0f0b70 msgr2=0x7f353c0f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:13:58.146 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:58.144+0000 7f3541928700 1 -- 172.21.15.73:0/2739969914 shutdown_connections 2023-12-01T22:13:58.146 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:13:58.144+0000 7f3541928700 1 -- 172.21.15.73:0/2739969914 wait complete. 2023-12-01T22:13:58.147 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:13:59.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:13:58 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:13:58.143676+0000 mon.smithi029 (mon.0) 170 : audit [DBG] from='client.? 172.21.15.73:0/2739969914' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:13:59.540 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:13:59.541 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:14:00.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:13:59 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:13:58.643638+0000 mgr.smithi029.loxrfu (mgr.14164) 84 : cluster [DBG] pgmap v77: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:14:00.700 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:00.697+0000 7f091e53f700 1 -- 172.21.15.73:0/3129787243 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f09180f6510 msgr2=0x7f09180f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:14:00.700 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:00.697+0000 7f091e53f700 1 --2- 172.21.15.73:0/3129787243 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f09180f6510 0x7f09180f6930 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f0908009fb0 tx=0x7f0908009b30).stop 2023-12-01T22:14:00.700 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:00.698+0000 7f091e53f700 1 -- 172.21.15.73:0/3129787243 shutdown_connections 2023-12-01T22:14:00.700 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:00.698+0000 7f091e53f700 1 --2- 172.21.15.73:0/3129787243 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f09180f6510 0x7f09180f6930 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:14:00.700 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:00.698+0000 7f091e53f700 1 -- 172.21.15.73:0/3129787243 >> 172.21.15.73:0/3129787243 conn(0x7f09180f1c70 msgr2=0x7f09180f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:14:00.700 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:00.698+0000 7f091e53f700 1 -- 172.21.15.73:0/3129787243 shutdown_connections 2023-12-01T22:14:00.700 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:00.698+0000 7f091e53f700 1 -- 172.21.15.73:0/3129787243 wait complete. 2023-12-01T22:14:00.700 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:00.698+0000 7f091e53f700 1 Processor -- start 2023-12-01T22:14:00.700 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:00.698+0000 7f091e53f700 1 -- start start 2023-12-01T22:14:00.702 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:00.698+0000 7f091e53f700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f09180f6510 0x7f0918106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:14:00.702 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:00.698+0000 7f091e53f700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0918106c20 con 0x7f09180f6510 2023-12-01T22:14:00.703 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:00.699+0000 7f0917fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f09180f6510 0x7f0918106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:14:00.703 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:00.699+0000 7f0917fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f09180f6510 0x7f0918106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:37378/0 (socket says 172.21.15.73:37378) 2023-12-01T22:14:00.703 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:00.699+0000 7f0917fff700 1 -- 172.21.15.73:0/3516953536 learned_addr learned my addr 172.21.15.73:0/3516953536 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:14:00.703 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:00.699+0000 7f0917fff700 1 -- 172.21.15.73:0/3516953536 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0908005040 con 0x7f09180f6510 2023-12-01T22:14:00.703 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:00.699+0000 7f0917fff700 1 --2- 172.21.15.73:0/3516953536 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f09180f6510 0x7f0918106730 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f0908013510 tx=0x7f090800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:14:00.703 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:00.699+0000 7f0914ff9700 1 -- 172.21.15.73:0/3516953536 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0908009320 con 0x7f09180f6510 2023-12-01T22:14:00.703 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:00.699+0000 7f091e53f700 1 -- 172.21.15.73:0/3516953536 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0918106e20 con 0x7f09180f6510 2023-12-01T22:14:00.703 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:00.699+0000 7f0914ff9700 1 -- 172.21.15.73:0/3516953536 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0908013f00 con 0x7f09180f6510 2023-12-01T22:14:00.703 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:00.699+0000 7f0914ff9700 1 -- 172.21.15.73:0/3516953536 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0908014d90 con 0x7f09180f6510 2023-12-01T22:14:00.703 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:00.699+0000 7f091e53f700 1 -- 172.21.15.73:0/3516953536 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09181072b0 con 0x7f09180f6510 2023-12-01T22:14:00.703 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:00.700+0000 7f0914ff9700 1 -- 172.21.15.73:0/3516953536 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f09080263a0 con 0x7f09180f6510 2023-12-01T22:14:00.703 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:00.700+0000 7f091e53f700 1 -- 172.21.15.73:0/3516953536 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0918045ad0 con 0x7f09180f6510 2023-12-01T22:14:00.704 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:00.700+0000 7f0914ff9700 1 --2- 172.21.15.73:0/3516953536 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f090002ff60 0x7f0900032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:14:00.704 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:00.700+0000 7f0914ff9700 1 -- 172.21.15.73:0/3516953536 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0908041300 con 0x7f09180f6510 2023-12-01T22:14:00.704 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:00.700+0000 7f09177fe700 1 --2- 172.21.15.73:0/3516953536 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f090002ff60 0x7f0900032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:14:00.704 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:00.701+0000 7f09177fe700 1 --2- 172.21.15.73:0/3516953536 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f090002ff60 0x7f0900032420 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f090c0096a0 tx=0x7f090c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:14:00.707 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:00.705+0000 7f0914ff9700 1 -- 172.21.15.73:0/3516953536 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f090801cb10 con 0x7f09180f6510 2023-12-01T22:14:00.916 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:00.914+0000 7f091e53f700 1 -- 172.21.15.73:0/3516953536 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f09180fbf60 con 0x7f09180f6510 2023-12-01T22:14:00.917 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:00.915+0000 7f0914ff9700 1 -- 172.21.15.73:0/3516953536 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f090801cb10 con 0x7f09180f6510 2023-12-01T22:14:00.917 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:14:00.917 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:14:00.918 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:00.917+0000 7f091e53f700 1 -- 172.21.15.73:0/3516953536 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f090002ff60 msgr2=0x7f0900032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:14:00.919 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:00.917+0000 7f091e53f700 1 --2- 172.21.15.73:0/3516953536 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f090002ff60 0x7f0900032420 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f090c0096a0 tx=0x7f090c006b40).stop 2023-12-01T22:14:00.919 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:00.917+0000 7f091e53f700 1 -- 172.21.15.73:0/3516953536 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f09180f6510 msgr2=0x7f0918106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:14:00.919 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:00.917+0000 7f091e53f700 1 --2- 172.21.15.73:0/3516953536 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f09180f6510 0x7f0918106730 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f0908013510 tx=0x7f090800b040).stop 2023-12-01T22:14:00.919 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:00.917+0000 7f091e53f700 1 -- 172.21.15.73:0/3516953536 shutdown_connections 2023-12-01T22:14:00.919 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:00.917+0000 7f091e53f700 1 --2- 172.21.15.73:0/3516953536 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f090002ff60 0x7f0900032420 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:14:00.919 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:00.917+0000 7f091e53f700 1 --2- 172.21.15.73:0/3516953536 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f09180f6510 0x7f0918106730 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:14:00.919 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:00.917+0000 7f091e53f700 1 -- 172.21.15.73:0/3516953536 >> 172.21.15.73:0/3516953536 conn(0x7f09180f1c70 msgr2=0x7f09180f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:14:00.919 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:00.917+0000 7f091e53f700 1 -- 172.21.15.73:0/3516953536 shutdown_connections 2023-12-01T22:14:00.919 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:00.917+0000 7f091e53f700 1 -- 172.21.15.73:0/3516953536 wait complete. 2023-12-01T22:14:00.920 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:14:01.776 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:14:01 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:14:00.643999+0000 mgr.smithi029.loxrfu (mgr.14164) 85 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:14:01.777 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:14:01 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:14:00.916385+0000 mon.smithi029 (mon.0) 171 : audit [DBG] from='client.? 172.21.15.73:0/3516953536' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:14:02.343 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:14:02.343 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:14:03.526 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:03.523+0000 7f84833bc700 1 -- 172.21.15.73:0/2344269687 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f847c0f5410 msgr2=0x7f847c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:14:03.527 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:03.523+0000 7f84833bc700 1 --2- 172.21.15.73:0/2344269687 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f847c0f5410 0x7f847c0f5830 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f846c009fb0 tx=0x7f846c009b30).stop 2023-12-01T22:14:03.527 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:03.523+0000 7f84833bc700 1 -- 172.21.15.73:0/2344269687 shutdown_connections 2023-12-01T22:14:03.527 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:03.523+0000 7f84833bc700 1 --2- 172.21.15.73:0/2344269687 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f847c0f5410 0x7f847c0f5830 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:14:03.528 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:03.523+0000 7f84833bc700 1 -- 172.21.15.73:0/2344269687 >> 172.21.15.73:0/2344269687 conn(0x7f847c0f0b70 msgr2=0x7f847c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:14:03.528 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:03.523+0000 7f84833bc700 1 -- 172.21.15.73:0/2344269687 shutdown_connections 2023-12-01T22:14:03.528 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:03.523+0000 7f84833bc700 1 -- 172.21.15.73:0/2344269687 wait complete. 2023-12-01T22:14:03.528 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:03.523+0000 7f84833bc700 1 Processor -- start 2023-12-01T22:14:03.528 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:03.523+0000 7f84833bc700 1 -- start start 2023-12-01T22:14:03.528 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:03.524+0000 7f84833bc700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f847c0f5410 0x7f847c109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:14:03.528 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:03.524+0000 7f84833bc700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f847c060240 con 0x7f847c0f5410 2023-12-01T22:14:03.528 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:03.524+0000 7f8481158700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f847c0f5410 0x7f847c109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:14:03.528 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:03.524+0000 7f8481158700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f847c0f5410 0x7f847c109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:37396/0 (socket says 172.21.15.73:37396) 2023-12-01T22:14:03.528 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:03.524+0000 7f8481158700 1 -- 172.21.15.73:0/1816827333 learned_addr learned my addr 172.21.15.73:0/1816827333 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:14:03.528 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:03.524+0000 7f8481158700 1 -- 172.21.15.73:0/1816827333 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f846c005040 con 0x7f847c0f5410 2023-12-01T22:14:03.528 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:03.524+0000 7f8481158700 1 --2- 172.21.15.73:0/1816827333 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f847c0f5410 0x7f847c109430 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f846c013510 tx=0x7f846c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:14:03.528 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:03.524+0000 7f8471ffb700 1 -- 172.21.15.73:0/1816827333 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f846c009320 con 0x7f847c0f5410 2023-12-01T22:14:03.528 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:03.524+0000 7f84833bc700 1 -- 172.21.15.73:0/1816827333 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f847c060440 con 0x7f847c0f5410 2023-12-01T22:14:03.529 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:03.524+0000 7f8471ffb700 1 -- 172.21.15.73:0/1816827333 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f846c013f00 con 0x7f847c0f5410 2023-12-01T22:14:03.529 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:03.524+0000 7f8471ffb700 1 -- 172.21.15.73:0/1816827333 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f846c014d90 con 0x7f847c0f5410 2023-12-01T22:14:03.529 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:03.525+0000 7f84833bc700 1 -- 172.21.15.73:0/1816827333 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f847c109ae0 con 0x7f847c0f5410 2023-12-01T22:14:03.530 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:03.525+0000 7f8471ffb700 1 -- 172.21.15.73:0/1816827333 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f846c0095f0 con 0x7f847c0f5410 2023-12-01T22:14:03.530 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:03.525+0000 7f84833bc700 1 -- 172.21.15.73:0/1816827333 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f847c045ad0 con 0x7f847c0f5410 2023-12-01T22:14:03.531 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:03.526+0000 7f8471ffb700 1 --2- 172.21.15.73:0/1816827333 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f846802ff60 0x7f8468032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:14:03.531 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:03.526+0000 7f8471ffb700 1 -- 172.21.15.73:0/1816827333 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f846c0410c0 con 0x7f847c0f5410 2023-12-01T22:14:03.531 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:03.526+0000 7f8480957700 1 --2- 172.21.15.73:0/1816827333 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f846802ff60 0x7f8468032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:14:03.531 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:03.526+0000 7f8480957700 1 --2- 172.21.15.73:0/1816827333 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f846802ff60 0x7f8468032420 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f84780096a0 tx=0x7f8478006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:14:03.532 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:03.530+0000 7f8471ffb700 1 -- 172.21.15.73:0/1816827333 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f846c01d7f0 con 0x7f847c0f5410 2023-12-01T22:14:03.738 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:03.736+0000 7f84833bc700 1 -- 172.21.15.73:0/1816827333 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f847c0fae60 con 0x7f847c0f5410 2023-12-01T22:14:03.738 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:03.737+0000 7f8471ffb700 1 -- 172.21.15.73:0/1816827333 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f846c010230 con 0x7f847c0f5410 2023-12-01T22:14:03.739 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:14:03.739 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:14:03.740 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:03.738+0000 7f84833bc700 1 -- 172.21.15.73:0/1816827333 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f846802ff60 msgr2=0x7f8468032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:14:03.741 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:03.738+0000 7f84833bc700 1 --2- 172.21.15.73:0/1816827333 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f846802ff60 0x7f8468032420 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f84780096a0 tx=0x7f8478006b40).stop 2023-12-01T22:14:03.741 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:03.738+0000 7f84833bc700 1 -- 172.21.15.73:0/1816827333 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f847c0f5410 msgr2=0x7f847c109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:14:03.741 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:03.738+0000 7f84833bc700 1 --2- 172.21.15.73:0/1816827333 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f847c0f5410 0x7f847c109430 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f846c013510 tx=0x7f846c00b040).stop 2023-12-01T22:14:03.741 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:03.738+0000 7f84833bc700 1 -- 172.21.15.73:0/1816827333 shutdown_connections 2023-12-01T22:14:03.741 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:03.738+0000 7f84833bc700 1 --2- 172.21.15.73:0/1816827333 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f846802ff60 0x7f8468032420 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:14:03.741 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:03.738+0000 7f84833bc700 1 --2- 172.21.15.73:0/1816827333 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f847c0f5410 0x7f847c109430 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:14:03.741 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:03.738+0000 7f84833bc700 1 -- 172.21.15.73:0/1816827333 >> 172.21.15.73:0/1816827333 conn(0x7f847c0f0b70 msgr2=0x7f847c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:14:03.741 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:03.738+0000 7f84833bc700 1 -- 172.21.15.73:0/1816827333 shutdown_connections 2023-12-01T22:14:03.741 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:03.738+0000 7f84833bc700 1 -- 172.21.15.73:0/1816827333 wait complete. 2023-12-01T22:14:03.741 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:14:04.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:14:03 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:14:02.644318+0000 mgr.smithi029.loxrfu (mgr.14164) 86 : cluster [DBG] pgmap v79: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:14:05.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:14:04 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:14:03.738048+0000 mon.smithi029 (mon.0) 172 : audit [DBG] from='client.? 172.21.15.73:0/1816827333' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:14:05.157 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:14:05.157 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:14:06.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:14:05 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:14:04.644596+0000 mgr.smithi029.loxrfu (mgr.14164) 87 : cluster [DBG] pgmap v80: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:14:06.369 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:06.363+0000 7f6a7e2b3700 1 -- 172.21.15.73:0/2852218148 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a780f7ae0 msgr2=0x7f6a780f7f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:14:06.369 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:06.363+0000 7f6a7e2b3700 1 --2- 172.21.15.73:0/2852218148 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a780f7ae0 0x7f6a780f7f00 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f6a68009fb0 tx=0x7f6a68009b30).stop 2023-12-01T22:14:06.369 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:06.363+0000 7f6a7e2b3700 1 -- 172.21.15.73:0/2852218148 shutdown_connections 2023-12-01T22:14:06.369 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:06.363+0000 7f6a7e2b3700 1 --2- 172.21.15.73:0/2852218148 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a780f7ae0 0x7f6a780f7f00 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:14:06.370 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:06.363+0000 7f6a7e2b3700 1 -- 172.21.15.73:0/2852218148 >> 172.21.15.73:0/2852218148 conn(0x7f6a780f0260 msgr2=0x7f6a780f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:14:06.370 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:06.363+0000 7f6a7e2b3700 1 -- 172.21.15.73:0/2852218148 shutdown_connections 2023-12-01T22:14:06.370 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:06.363+0000 7f6a7e2b3700 1 -- 172.21.15.73:0/2852218148 wait complete. 2023-12-01T22:14:06.370 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:06.363+0000 7f6a7e2b3700 1 Processor -- start 2023-12-01T22:14:06.370 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:06.363+0000 7f6a7e2b3700 1 -- start start 2023-12-01T22:14:06.370 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:06.364+0000 7f6a7e2b3700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a780f7ae0 0x7f6a78106710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:14:06.371 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:06.364+0000 7f6a7e2b3700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a78106c00 con 0x7f6a780f7ae0 2023-12-01T22:14:06.371 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:06.364+0000 7f6a77fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a780f7ae0 0x7f6a78106710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:14:06.371 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:06.364+0000 7f6a77fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a780f7ae0 0x7f6a78106710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:45660/0 (socket says 172.21.15.73:45660) 2023-12-01T22:14:06.371 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:06.364+0000 7f6a77fff700 1 -- 172.21.15.73:0/764463676 learned_addr learned my addr 172.21.15.73:0/764463676 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:14:06.371 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:06.364+0000 7f6a77fff700 1 -- 172.21.15.73:0/764463676 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a68005040 con 0x7f6a780f7ae0 2023-12-01T22:14:06.371 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:06.364+0000 7f6a77fff700 1 --2- 172.21.15.73:0/764463676 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a780f7ae0 0x7f6a78106710 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f6a68013510 tx=0x7f6a6800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:14:06.372 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:06.364+0000 7f6a74ff9700 1 -- 172.21.15.73:0/764463676 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6a68009320 con 0x7f6a780f7ae0 2023-12-01T22:14:06.372 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:06.364+0000 7f6a7e2b3700 1 -- 172.21.15.73:0/764463676 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a78106e00 con 0x7f6a780f7ae0 2023-12-01T22:14:06.372 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:06.364+0000 7f6a74ff9700 1 -- 172.21.15.73:0/764463676 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6a68013f00 con 0x7f6a780f7ae0 2023-12-01T22:14:06.372 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:06.364+0000 7f6a74ff9700 1 -- 172.21.15.73:0/764463676 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6a68014d90 con 0x7f6a780f7ae0 2023-12-01T22:14:06.372 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:06.364+0000 7f6a7e2b3700 1 -- 172.21.15.73:0/764463676 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a78107290 con 0x7f6a780f7ae0 2023-12-01T22:14:06.372 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:06.365+0000 7f6a74ff9700 1 -- 172.21.15.73:0/764463676 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6a680263a0 con 0x7f6a780f7ae0 2023-12-01T22:14:06.372 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:06.365+0000 7f6a74ff9700 1 --2- 172.21.15.73:0/764463676 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f6a6002ff60 0x7f6a60032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:14:06.373 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:06.365+0000 7f6a74ff9700 1 -- 172.21.15.73:0/764463676 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6a680413c0 con 0x7f6a780f7ae0 2023-12-01T22:14:06.373 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:06.365+0000 7f6a7e2b3700 1 -- 172.21.15.73:0/764463676 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a78045ad0 con 0x7f6a780f7ae0 2023-12-01T22:14:06.373 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:06.366+0000 7f6a777fe700 1 --2- 172.21.15.73:0/764463676 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f6a6002ff60 0x7f6a60032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:14:06.373 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:06.366+0000 7f6a777fe700 1 --2- 172.21.15.73:0/764463676 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f6a6002ff60 0x7f6a60032420 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f6a6c0096a0 tx=0x7f6a6c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:14:06.374 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:06.371+0000 7f6a74ff9700 1 -- 172.21.15.73:0/764463676 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6a68012920 con 0x7f6a780f7ae0 2023-12-01T22:14:06.577 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:06.575+0000 7f6a7e2b3700 1 -- 172.21.15.73:0/764463676 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6a780fbfc0 con 0x7f6a780f7ae0 2023-12-01T22:14:06.577 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:06.576+0000 7f6a74ff9700 1 -- 172.21.15.73:0/764463676 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6a6802b030 con 0x7f6a780f7ae0 2023-12-01T22:14:06.578 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:14:06.578 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:14:06.579 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:06.578+0000 7f6a7e2b3700 1 -- 172.21.15.73:0/764463676 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f6a6002ff60 msgr2=0x7f6a60032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:14:06.579 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:06.578+0000 7f6a7e2b3700 1 --2- 172.21.15.73:0/764463676 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f6a6002ff60 0x7f6a60032420 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f6a6c0096a0 tx=0x7f6a6c006b40).stop 2023-12-01T22:14:06.579 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:06.578+0000 7f6a7e2b3700 1 -- 172.21.15.73:0/764463676 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a780f7ae0 msgr2=0x7f6a78106710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:14:06.579 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:06.578+0000 7f6a7e2b3700 1 --2- 172.21.15.73:0/764463676 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a780f7ae0 0x7f6a78106710 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f6a68013510 tx=0x7f6a6800b040).stop 2023-12-01T22:14:06.580 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:06.578+0000 7f6a7e2b3700 1 -- 172.21.15.73:0/764463676 shutdown_connections 2023-12-01T22:14:06.580 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:06.578+0000 7f6a7e2b3700 1 --2- 172.21.15.73:0/764463676 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f6a6002ff60 0x7f6a60032420 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:14:06.580 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:06.578+0000 7f6a7e2b3700 1 --2- 172.21.15.73:0/764463676 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a780f7ae0 0x7f6a78106710 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:14:06.580 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:06.578+0000 7f6a7e2b3700 1 -- 172.21.15.73:0/764463676 >> 172.21.15.73:0/764463676 conn(0x7f6a780f0260 msgr2=0x7f6a780f0d50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:14:06.580 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:06.578+0000 7f6a7e2b3700 1 -- 172.21.15.73:0/764463676 shutdown_connections 2023-12-01T22:14:06.580 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:06.578+0000 7f6a7e2b3700 1 -- 172.21.15.73:0/764463676 wait complete. 2023-12-01T22:14:06.580 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:14:07.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:14:06 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:14:06.577051+0000 mon.smithi029 (mon.0) 173 : audit [DBG] from='client.? 172.21.15.73:0/764463676' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:14:07.977 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:14:07.977 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:14:08.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:14:07 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:14:06.644932+0000 mgr.smithi029.loxrfu (mgr.14164) 88 : cluster [DBG] pgmap v81: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:14:09.114 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:09.110+0000 7f1f179c7700 1 -- 172.21.15.73:0/2695965457 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f100f6510 msgr2=0x7f1f100f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:14:09.114 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:09.110+0000 7f1f179c7700 1 --2- 172.21.15.73:0/2695965457 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f100f6510 0x7f1f100f6930 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7f1f00009fb0 tx=0x7f1f00009b30).stop 2023-12-01T22:14:09.114 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:09.110+0000 7f1f179c7700 1 -- 172.21.15.73:0/2695965457 shutdown_connections 2023-12-01T22:14:09.114 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:09.110+0000 7f1f179c7700 1 --2- 172.21.15.73:0/2695965457 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f100f6510 0x7f1f100f6930 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:14:09.114 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:09.110+0000 7f1f179c7700 1 -- 172.21.15.73:0/2695965457 >> 172.21.15.73:0/2695965457 conn(0x7f1f100f1c70 msgr2=0x7f1f100f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:14:09.114 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:09.111+0000 7f1f179c7700 1 -- 172.21.15.73:0/2695965457 shutdown_connections 2023-12-01T22:14:09.114 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:09.111+0000 7f1f179c7700 1 -- 172.21.15.73:0/2695965457 wait complete. 2023-12-01T22:14:09.114 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:09.111+0000 7f1f179c7700 1 Processor -- start 2023-12-01T22:14:09.114 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:09.111+0000 7f1f179c7700 1 -- start start 2023-12-01T22:14:09.115 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:09.111+0000 7f1f179c7700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f100f6510 0x7f1f1006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:14:09.115 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:09.111+0000 7f1f179c7700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f10069e20 con 0x7f1f100f6510 2023-12-01T22:14:09.115 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:09.112+0000 7f1f15763700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f100f6510 0x7f1f1006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:14:09.115 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:09.112+0000 7f1f15763700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f100f6510 0x7f1f1006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:45680/0 (socket says 172.21.15.73:45680) 2023-12-01T22:14:09.115 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:09.112+0000 7f1f15763700 1 -- 172.21.15.73:0/907072163 learned_addr learned my addr 172.21.15.73:0/907072163 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:14:09.115 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:09.112+0000 7f1f15763700 1 -- 172.21.15.73:0/907072163 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f00005040 con 0x7f1f100f6510 2023-12-01T22:14:09.115 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:09.112+0000 7f1f15763700 1 --2- 172.21.15.73:0/907072163 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f100f6510 0x7f1f1006b9a0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7f1f00013510 tx=0x7f1f0000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:14:09.115 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:09.112+0000 7f1f05ffb700 1 -- 172.21.15.73:0/907072163 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1f00009320 con 0x7f1f100f6510 2023-12-01T22:14:09.115 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:09.112+0000 7f1f179c7700 1 -- 172.21.15.73:0/907072163 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f10069fc0 con 0x7f1f100f6510 2023-12-01T22:14:09.115 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:09.112+0000 7f1f05ffb700 1 -- 172.21.15.73:0/907072163 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1f00013f00 con 0x7f1f100f6510 2023-12-01T22:14:09.115 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:09.112+0000 7f1f05ffb700 1 -- 172.21.15.73:0/907072163 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1f00014d90 con 0x7f1f100f6510 2023-12-01T22:14:09.115 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:09.112+0000 7f1f179c7700 1 -- 172.21.15.73:0/907072163 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f1006a450 con 0x7f1f100f6510 2023-12-01T22:14:09.116 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:09.113+0000 7f1f05ffb700 1 -- 172.21.15.73:0/907072163 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1f000263a0 con 0x7f1f100f6510 2023-12-01T22:14:09.116 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:09.113+0000 7f1f05ffb700 1 --2- 172.21.15.73:0/907072163 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f1efc02ff60 0x7f1efc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:14:09.116 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:09.113+0000 7f1f05ffb700 1 -- 172.21.15.73:0/907072163 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1f000410b0 con 0x7f1f100f6510 2023-12-01T22:14:09.116 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:09.113+0000 7f1f179c7700 1 -- 172.21.15.73:0/907072163 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f10059100 con 0x7f1f100f6510 2023-12-01T22:14:09.117 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:09.114+0000 7f1f14f62700 1 --2- 172.21.15.73:0/907072163 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f1efc02ff60 0x7f1efc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:14:09.117 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:09.114+0000 7f1f14f62700 1 --2- 172.21.15.73:0/907072163 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f1efc02ff60 0x7f1efc032420 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f1f0c0096a0 tx=0x7f1f0c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:14:09.120 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:09.118+0000 7f1f05ffb700 1 -- 172.21.15.73:0/907072163 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1f00012920 con 0x7f1f100f6510 2023-12-01T22:14:09.325 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:09.323+0000 7f1f179c7700 1 -- 172.21.15.73:0/907072163 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1f100fbf60 con 0x7f1f100f6510 2023-12-01T22:14:09.325 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:09.325+0000 7f1f05ffb700 1 -- 172.21.15.73:0/907072163 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1f0000f340 con 0x7f1f100f6510 2023-12-01T22:14:09.326 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:14:09.326 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:14:09.327 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:09.325+0000 7f1f179c7700 1 -- 172.21.15.73:0/907072163 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f1efc02ff60 msgr2=0x7f1efc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:14:09.327 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:09.325+0000 7f1f179c7700 1 --2- 172.21.15.73:0/907072163 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f1efc02ff60 0x7f1efc032420 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f1f0c0096a0 tx=0x7f1f0c006b40).stop 2023-12-01T22:14:09.327 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:09.325+0000 7f1f179c7700 1 -- 172.21.15.73:0/907072163 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f100f6510 msgr2=0x7f1f1006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:14:09.327 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:09.325+0000 7f1f179c7700 1 --2- 172.21.15.73:0/907072163 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f100f6510 0x7f1f1006b9a0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7f1f00013510 tx=0x7f1f0000b040).stop 2023-12-01T22:14:09.327 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:09.325+0000 7f1f179c7700 1 -- 172.21.15.73:0/907072163 shutdown_connections 2023-12-01T22:14:09.328 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:09.325+0000 7f1f179c7700 1 --2- 172.21.15.73:0/907072163 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f1efc02ff60 0x7f1efc032420 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:14:09.328 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:09.326+0000 7f1f179c7700 1 --2- 172.21.15.73:0/907072163 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f100f6510 0x7f1f1006b9a0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:14:09.328 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:09.326+0000 7f1f179c7700 1 -- 172.21.15.73:0/907072163 >> 172.21.15.73:0/907072163 conn(0x7f1f100f1c70 msgr2=0x7f1f100f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:14:09.328 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:09.326+0000 7f1f179c7700 1 -- 172.21.15.73:0/907072163 shutdown_connections 2023-12-01T22:14:09.328 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:09.326+0000 7f1f179c7700 1 -- 172.21.15.73:0/907072163 wait complete. 2023-12-01T22:14:09.328 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:14:10.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:14:09 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:14:08.645326+0000 mgr.smithi029.loxrfu (mgr.14164) 89 : cluster [DBG] pgmap v82: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:14:10.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:14:09 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:14:09.325225+0000 mon.smithi029 (mon.0) 174 : audit [DBG] from='client.? 172.21.15.73:0/907072163' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:14:10.736 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:14:10.736 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:14:11.777 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:14:11 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:14:10.645652+0000 mgr.smithi029.loxrfu (mgr.14164) 90 : cluster [DBG] pgmap v83: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:14:11.894 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:11.892+0000 7faff9ee5700 1 -- 172.21.15.73:0/1504154405 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faff40f5410 msgr2=0x7faff40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:14:11.894 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:11.892+0000 7faff9ee5700 1 --2- 172.21.15.73:0/1504154405 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faff40f5410 0x7faff40f5830 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7fafe4009fb0 tx=0x7fafe4009b30).stop 2023-12-01T22:14:11.894 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:11.892+0000 7faff9ee5700 1 -- 172.21.15.73:0/1504154405 shutdown_connections 2023-12-01T22:14:11.895 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:11.892+0000 7faff9ee5700 1 --2- 172.21.15.73:0/1504154405 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faff40f5410 0x7faff40f5830 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:14:11.895 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:11.892+0000 7faff9ee5700 1 -- 172.21.15.73:0/1504154405 >> 172.21.15.73:0/1504154405 conn(0x7faff40f0b70 msgr2=0x7faff40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:14:11.895 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:11.892+0000 7faff9ee5700 1 -- 172.21.15.73:0/1504154405 shutdown_connections 2023-12-01T22:14:11.895 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:11.892+0000 7faff9ee5700 1 -- 172.21.15.73:0/1504154405 wait complete. 2023-12-01T22:14:11.895 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:11.893+0000 7faff9ee5700 1 Processor -- start 2023-12-01T22:14:11.895 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:11.893+0000 7faff9ee5700 1 -- start start 2023-12-01T22:14:11.896 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:11.893+0000 7faff9ee5700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faff40f5410 0x7faff4105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:14:11.896 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:11.893+0000 7faff9ee5700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faff4105b20 con 0x7faff40f5410 2023-12-01T22:14:11.897 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:11.893+0000 7faff37fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faff40f5410 0x7faff4105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:14:11.897 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:11.893+0000 7faff37fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faff40f5410 0x7faff4105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:45710/0 (socket says 172.21.15.73:45710) 2023-12-01T22:14:11.897 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:11.893+0000 7faff37fe700 1 -- 172.21.15.73:0/617340494 learned_addr learned my addr 172.21.15.73:0/617340494 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:14:11.897 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:11.894+0000 7faff37fe700 1 -- 172.21.15.73:0/617340494 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafe4005040 con 0x7faff40f5410 2023-12-01T22:14:11.897 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:11.894+0000 7faff37fe700 1 --2- 172.21.15.73:0/617340494 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faff40f5410 0x7faff4105630 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7fafe4013510 tx=0x7fafe400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:14:11.897 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:11.894+0000 7fafebfff700 1 -- 172.21.15.73:0/617340494 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fafe4009320 con 0x7faff40f5410 2023-12-01T22:14:11.897 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:11.894+0000 7faff9ee5700 1 -- 172.21.15.73:0/617340494 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faff4105d20 con 0x7faff40f5410 2023-12-01T22:14:11.897 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:11.894+0000 7fafebfff700 1 -- 172.21.15.73:0/617340494 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fafe4013f00 con 0x7faff40f5410 2023-12-01T22:14:11.897 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:11.894+0000 7fafebfff700 1 -- 172.21.15.73:0/617340494 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fafe4014d90 con 0x7faff40f5410 2023-12-01T22:14:11.897 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:11.894+0000 7faff9ee5700 1 -- 172.21.15.73:0/617340494 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faff41061b0 con 0x7faff40f5410 2023-12-01T22:14:11.897 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:11.895+0000 7fafebfff700 1 -- 172.21.15.73:0/617340494 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fafe4014420 con 0x7faff40f5410 2023-12-01T22:14:11.897 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:11.895+0000 7fafebfff700 1 --2- 172.21.15.73:0/617340494 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fafe002ff60 0x7fafe0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:14:11.898 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:11.895+0000 7fafebfff700 1 -- 172.21.15.73:0/617340494 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fafe4041080 con 0x7faff40f5410 2023-12-01T22:14:11.898 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:11.895+0000 7faff9ee5700 1 -- 172.21.15.73:0/617340494 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faff4045ad0 con 0x7faff40f5410 2023-12-01T22:14:11.898 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:11.895+0000 7faff2ffd700 1 --2- 172.21.15.73:0/617340494 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fafe002ff60 0x7fafe0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:14:11.898 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:11.895+0000 7faff2ffd700 1 --2- 172.21.15.73:0/617340494 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fafe002ff60 0x7fafe0032420 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7fafdc0096a0 tx=0x7fafdc006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:14:11.901 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:11.900+0000 7fafebfff700 1 -- 172.21.15.73:0/617340494 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fafe40095f0 con 0x7faff40f5410 2023-12-01T22:14:12.107 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:12.105+0000 7faff9ee5700 1 -- 172.21.15.73:0/617340494 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7faff40fae60 con 0x7faff40f5410 2023-12-01T22:14:12.108 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:12.106+0000 7fafebfff700 1 -- 172.21.15.73:0/617340494 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fafe401db70 con 0x7faff40f5410 2023-12-01T22:14:12.109 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:14:12.109 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:14:12.109 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:12.108+0000 7faff9ee5700 1 -- 172.21.15.73:0/617340494 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fafe002ff60 msgr2=0x7fafe0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:14:12.109 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:12.108+0000 7faff9ee5700 1 --2- 172.21.15.73:0/617340494 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fafe002ff60 0x7fafe0032420 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7fafdc0096a0 tx=0x7fafdc006b40).stop 2023-12-01T22:14:12.110 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:12.108+0000 7faff9ee5700 1 -- 172.21.15.73:0/617340494 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faff40f5410 msgr2=0x7faff4105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:14:12.110 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:12.108+0000 7faff9ee5700 1 --2- 172.21.15.73:0/617340494 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faff40f5410 0x7faff4105630 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7fafe4013510 tx=0x7fafe400b040).stop 2023-12-01T22:14:12.110 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:12.108+0000 7faff9ee5700 1 -- 172.21.15.73:0/617340494 shutdown_connections 2023-12-01T22:14:12.110 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:12.108+0000 7faff9ee5700 1 --2- 172.21.15.73:0/617340494 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fafe002ff60 0x7fafe0032420 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:14:12.110 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:12.108+0000 7faff9ee5700 1 --2- 172.21.15.73:0/617340494 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faff40f5410 0x7faff4105630 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:14:12.110 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:12.108+0000 7faff9ee5700 1 -- 172.21.15.73:0/617340494 >> 172.21.15.73:0/617340494 conn(0x7faff40f0b70 msgr2=0x7faff40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:14:12.110 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:12.108+0000 7faff9ee5700 1 -- 172.21.15.73:0/617340494 shutdown_connections 2023-12-01T22:14:12.110 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:12.108+0000 7faff9ee5700 1 -- 172.21.15.73:0/617340494 wait complete. 2023-12-01T22:14:12.110 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:14:12.776 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:14:12 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:14:12.107419+0000 mon.smithi029 (mon.0) 175 : audit [DBG] from='client.? 172.21.15.73:0/617340494' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:14:13.491 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:14:13.491 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:14:13.777 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:14:13 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:14:12.646067+0000 mgr.smithi029.loxrfu (mgr.14164) 91 : cluster [DBG] pgmap v84: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:14:13.777 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:14:13 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:14:12.778563+0000 mon.smithi029 (mon.0) 176 : audit [INF] from='mgr.14164 172.21.15.29:0/3195927111' entity='mgr.smithi029.loxrfu' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi029.loxrfu/mirror_snapshot_schedule"}]: dispatch 2023-12-01T22:14:13.777 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:14:13 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:14:12.782337+0000 mon.smithi029 (mon.0) 177 : audit [INF] from='mgr.14164 172.21.15.29:0/3195927111' entity='mgr.smithi029.loxrfu' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi029.loxrfu/trash_purge_schedule"}]: dispatch 2023-12-01T22:14:14.631 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:14.627+0000 7f20741b1700 1 -- 172.21.15.73:0/3553439793 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f206c0f5410 msgr2=0x7f206c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:14:14.631 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:14.627+0000 7f20741b1700 1 --2- 172.21.15.73:0/3553439793 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f206c0f5410 0x7f206c0f5830 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7f205c009fb0 tx=0x7f205c009b30).stop 2023-12-01T22:14:14.631 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:14.627+0000 7f20741b1700 1 -- 172.21.15.73:0/3553439793 shutdown_connections 2023-12-01T22:14:14.631 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:14.627+0000 7f20741b1700 1 --2- 172.21.15.73:0/3553439793 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f206c0f5410 0x7f206c0f5830 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:14:14.631 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:14.627+0000 7f20741b1700 1 -- 172.21.15.73:0/3553439793 >> 172.21.15.73:0/3553439793 conn(0x7f206c0f0b70 msgr2=0x7f206c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:14:14.631 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:14.628+0000 7f20741b1700 1 -- 172.21.15.73:0/3553439793 shutdown_connections 2023-12-01T22:14:14.631 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:14.628+0000 7f20741b1700 1 -- 172.21.15.73:0/3553439793 wait complete. 2023-12-01T22:14:14.631 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:14.628+0000 7f20741b1700 1 Processor -- start 2023-12-01T22:14:14.631 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:14.628+0000 7f20741b1700 1 -- start start 2023-12-01T22:14:14.631 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:14.628+0000 7f20741b1700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f206c0f5410 0x7f206c0fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:14:14.631 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:14.628+0000 7f20741b1700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f206c0fb820 con 0x7f206c0f5410 2023-12-01T22:14:14.632 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:14.628+0000 7f2071f4d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f206c0f5410 0x7f206c0fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:14:14.632 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:14.628+0000 7f2071f4d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f206c0f5410 0x7f206c0fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:41242/0 (socket says 172.21.15.73:41242) 2023-12-01T22:14:14.632 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:14.628+0000 7f2071f4d700 1 -- 172.21.15.73:0/2912712276 learned_addr learned my addr 172.21.15.73:0/2912712276 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:14:14.632 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:14.629+0000 7f2071f4d700 1 -- 172.21.15.73:0/2912712276 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f205c005040 con 0x7f206c0f5410 2023-12-01T22:14:14.632 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:14.629+0000 7f2071f4d700 1 --2- 172.21.15.73:0/2912712276 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f206c0f5410 0x7f206c0fd3a0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7f205c013510 tx=0x7f205c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:14:14.632 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:14.629+0000 7f20627fc700 1 -- 172.21.15.73:0/2912712276 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f205c009320 con 0x7f206c0f5410 2023-12-01T22:14:14.632 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:14.629+0000 7f20741b1700 1 -- 172.21.15.73:0/2912712276 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f206c0fb9c0 con 0x7f206c0f5410 2023-12-01T22:14:14.632 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:14.629+0000 7f20627fc700 1 -- 172.21.15.73:0/2912712276 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f205c013f00 con 0x7f206c0f5410 2023-12-01T22:14:14.632 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:14.629+0000 7f20627fc700 1 -- 172.21.15.73:0/2912712276 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f205c014d90 con 0x7f206c0f5410 2023-12-01T22:14:14.632 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:14.629+0000 7f20741b1700 1 -- 172.21.15.73:0/2912712276 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f206c0fbe50 con 0x7f206c0f5410 2023-12-01T22:14:14.633 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:14.630+0000 7f20627fc700 1 -- 172.21.15.73:0/2912712276 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f205c0095f0 con 0x7f206c0f5410 2023-12-01T22:14:14.633 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:14.630+0000 7f20741b1700 1 -- 172.21.15.73:0/2912712276 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f206c059100 con 0x7f206c0f5410 2023-12-01T22:14:14.633 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:14.630+0000 7f20627fc700 1 --2- 172.21.15.73:0/2912712276 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f205802ff60 0x7f2058032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:14:14.633 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:14.630+0000 7f20627fc700 1 -- 172.21.15.73:0/2912712276 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f205c041080 con 0x7f206c0f5410 2023-12-01T22:14:14.633 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:14.630+0000 7f207174c700 1 --2- 172.21.15.73:0/2912712276 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f205802ff60 0x7f2058032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:14:14.633 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:14.631+0000 7f207174c700 1 --2- 172.21.15.73:0/2912712276 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f205802ff60 0x7f2058032420 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f20680096a0 tx=0x7f2068006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:14:14.636 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:14.635+0000 7f20627fc700 1 -- 172.21.15.73:0/2912712276 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f205c014420 con 0x7f206c0f5410 2023-12-01T22:14:14.845 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:14.843+0000 7f20741b1700 1 -- 172.21.15.73:0/2912712276 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f206c0fae60 con 0x7f206c0f5410 2023-12-01T22:14:14.845 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:14.845+0000 7f20627fc700 1 -- 172.21.15.73:0/2912712276 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f205c010230 con 0x7f206c0f5410 2023-12-01T22:14:14.846 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:14:14.846 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:14:14.847 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:14.845+0000 7f20741b1700 1 -- 172.21.15.73:0/2912712276 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f205802ff60 msgr2=0x7f2058032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:14:14.847 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:14.845+0000 7f20741b1700 1 --2- 172.21.15.73:0/2912712276 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f205802ff60 0x7f2058032420 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f20680096a0 tx=0x7f2068006b40).stop 2023-12-01T22:14:14.847 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:14.845+0000 7f20741b1700 1 -- 172.21.15.73:0/2912712276 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f206c0f5410 msgr2=0x7f206c0fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:14:14.847 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:14.845+0000 7f20741b1700 1 --2- 172.21.15.73:0/2912712276 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f206c0f5410 0x7f206c0fd3a0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7f205c013510 tx=0x7f205c00b040).stop 2023-12-01T22:14:14.848 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:14.846+0000 7f20741b1700 1 -- 172.21.15.73:0/2912712276 shutdown_connections 2023-12-01T22:14:14.848 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:14.846+0000 7f20741b1700 1 --2- 172.21.15.73:0/2912712276 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f205802ff60 0x7f2058032420 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:14:14.848 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:14.846+0000 7f20741b1700 1 --2- 172.21.15.73:0/2912712276 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f206c0f5410 0x7f206c0fd3a0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:14:14.848 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:14.846+0000 7f20741b1700 1 -- 172.21.15.73:0/2912712276 >> 172.21.15.73:0/2912712276 conn(0x7f206c0f0b70 msgr2=0x7f206c05fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:14:14.848 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:14.846+0000 7f20741b1700 1 -- 172.21.15.73:0/2912712276 shutdown_connections 2023-12-01T22:14:14.848 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:14.846+0000 7f20741b1700 1 -- 172.21.15.73:0/2912712276 wait complete. 2023-12-01T22:14:14.848 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:14:16.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:14:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:14:14.646382+0000 mgr.smithi029.loxrfu (mgr.14164) 92 : cluster [DBG] pgmap v85: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:14:16.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:14:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:14:14.845077+0000 mon.smithi029 (mon.0) 178 : audit [DBG] from='client.? 172.21.15.73:0/2912712276' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:14:16.235 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:14:16.235 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:14:17.365 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:17.363+0000 7fa983730700 1 -- 172.21.15.73:0/2944029133 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa97c0f6510 msgr2=0x7fa97c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:14:17.365 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:17.363+0000 7fa983730700 1 --2- 172.21.15.73:0/2944029133 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa97c0f6510 0x7fa97c0f6930 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7fa96c009fb0 tx=0x7fa96c009b30).stop 2023-12-01T22:14:17.365 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:17.363+0000 7fa983730700 1 -- 172.21.15.73:0/2944029133 shutdown_connections 2023-12-01T22:14:17.365 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:17.363+0000 7fa983730700 1 --2- 172.21.15.73:0/2944029133 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa97c0f6510 0x7fa97c0f6930 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:14:17.366 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:17.363+0000 7fa983730700 1 -- 172.21.15.73:0/2944029133 >> 172.21.15.73:0/2944029133 conn(0x7fa97c0f1c70 msgr2=0x7fa97c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:14:17.366 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:17.363+0000 7fa983730700 1 -- 172.21.15.73:0/2944029133 shutdown_connections 2023-12-01T22:14:17.366 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:17.363+0000 7fa983730700 1 -- 172.21.15.73:0/2944029133 wait complete. 2023-12-01T22:14:17.368 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:17.364+0000 7fa983730700 1 Processor -- start 2023-12-01T22:14:17.368 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:17.364+0000 7fa983730700 1 -- start start 2023-12-01T22:14:17.368 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:17.364+0000 7fa983730700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa97c0f6510 0x7fa97c06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:14:17.368 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:17.364+0000 7fa983730700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa97c069e20 con 0x7fa97c0f6510 2023-12-01T22:14:17.368 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:17.364+0000 7fa9814cc700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa97c0f6510 0x7fa97c06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:14:17.368 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:17.364+0000 7fa9814cc700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa97c0f6510 0x7fa97c06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:41270/0 (socket says 172.21.15.73:41270) 2023-12-01T22:14:17.368 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:17.364+0000 7fa9814cc700 1 -- 172.21.15.73:0/3370707602 learned_addr learned my addr 172.21.15.73:0/3370707602 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:14:17.368 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:17.365+0000 7fa9814cc700 1 -- 172.21.15.73:0/3370707602 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa96c005040 con 0x7fa97c0f6510 2023-12-01T22:14:17.368 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:17.365+0000 7fa9814cc700 1 --2- 172.21.15.73:0/3370707602 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa97c0f6510 0x7fa97c06b9a0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7fa96c013510 tx=0x7fa96c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:14:17.368 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:17.365+0000 7fa971ffb700 1 -- 172.21.15.73:0/3370707602 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa96c009320 con 0x7fa97c0f6510 2023-12-01T22:14:17.368 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:17.365+0000 7fa983730700 1 -- 172.21.15.73:0/3370707602 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa97c069fc0 con 0x7fa97c0f6510 2023-12-01T22:14:17.369 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:17.365+0000 7fa971ffb700 1 -- 172.21.15.73:0/3370707602 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa96c013f00 con 0x7fa97c0f6510 2023-12-01T22:14:17.369 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:17.365+0000 7fa971ffb700 1 -- 172.21.15.73:0/3370707602 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa96c014d90 con 0x7fa97c0f6510 2023-12-01T22:14:17.369 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:17.365+0000 7fa983730700 1 -- 172.21.15.73:0/3370707602 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa97c06a450 con 0x7fa97c0f6510 2023-12-01T22:14:17.369 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:17.366+0000 7fa971ffb700 1 -- 172.21.15.73:0/3370707602 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa96c014420 con 0x7fa97c0f6510 2023-12-01T22:14:17.369 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:17.366+0000 7fa983730700 1 -- 172.21.15.73:0/3370707602 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa97c059100 con 0x7fa97c0f6510 2023-12-01T22:14:17.369 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:17.366+0000 7fa971ffb700 1 --2- 172.21.15.73:0/3370707602 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fa96802ff60 0x7fa968032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:14:17.369 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:17.366+0000 7fa971ffb700 1 -- 172.21.15.73:0/3370707602 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa96c0410c0 con 0x7fa97c0f6510 2023-12-01T22:14:17.369 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:17.366+0000 7fa980ccb700 1 --2- 172.21.15.73:0/3370707602 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fa96802ff60 0x7fa968032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:14:17.369 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:17.367+0000 7fa980ccb700 1 --2- 172.21.15.73:0/3370707602 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fa96802ff60 0x7fa968032420 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7fa9780096a0 tx=0x7fa978006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:14:17.373 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:17.371+0000 7fa971ffb700 1 -- 172.21.15.73:0/3370707602 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa96c0095f0 con 0x7fa97c0f6510 2023-12-01T22:14:17.581 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:17.579+0000 7fa983730700 1 -- 172.21.15.73:0/3370707602 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa97c0fbf60 con 0x7fa97c0f6510 2023-12-01T22:14:17.582 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:17.580+0000 7fa971ffb700 1 -- 172.21.15.73:0/3370707602 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa96c01db70 con 0x7fa97c0f6510 2023-12-01T22:14:17.582 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:14:17.582 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:14:17.584 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:17.582+0000 7fa983730700 1 -- 172.21.15.73:0/3370707602 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fa96802ff60 msgr2=0x7fa968032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:14:17.584 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:17.582+0000 7fa983730700 1 --2- 172.21.15.73:0/3370707602 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fa96802ff60 0x7fa968032420 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7fa9780096a0 tx=0x7fa978006b40).stop 2023-12-01T22:14:17.584 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:17.582+0000 7fa983730700 1 -- 172.21.15.73:0/3370707602 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa97c0f6510 msgr2=0x7fa97c06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:14:17.584 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:17.582+0000 7fa983730700 1 --2- 172.21.15.73:0/3370707602 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa97c0f6510 0x7fa97c06b9a0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7fa96c013510 tx=0x7fa96c00b040).stop 2023-12-01T22:14:17.584 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:17.582+0000 7fa983730700 1 -- 172.21.15.73:0/3370707602 shutdown_connections 2023-12-01T22:14:17.584 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:17.582+0000 7fa983730700 1 --2- 172.21.15.73:0/3370707602 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fa96802ff60 0x7fa968032420 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:14:17.584 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:17.582+0000 7fa983730700 1 --2- 172.21.15.73:0/3370707602 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa97c0f6510 0x7fa97c06b9a0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:14:17.584 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:17.582+0000 7fa983730700 1 -- 172.21.15.73:0/3370707602 >> 172.21.15.73:0/3370707602 conn(0x7fa97c0f1c70 msgr2=0x7fa97c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:14:17.584 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:17.582+0000 7fa983730700 1 -- 172.21.15.73:0/3370707602 shutdown_connections 2023-12-01T22:14:17.584 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:17.582+0000 7fa983730700 1 -- 172.21.15.73:0/3370707602 wait complete. 2023-12-01T22:14:17.585 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:14:18.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:14:17 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:14:16.646721+0000 mgr.smithi029.loxrfu (mgr.14164) 93 : cluster [DBG] pgmap v86: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:14:18.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:14:17 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:14:17.581603+0000 mon.smithi029 (mon.0) 179 : audit [DBG] from='client.? 172.21.15.73:0/3370707602' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:14:18.974 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:14:18.974 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:14:20.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:14:19 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:14:18.647108+0000 mgr.smithi029.loxrfu (mgr.14164) 94 : cluster [DBG] pgmap v87: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:14:20.136 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:20.133+0000 7f2e74827700 1 -- 172.21.15.73:0/841552163 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2e700f51a0 msgr2=0x7f2e700f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:14:20.136 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:20.133+0000 7f2e74827700 1 --2- 172.21.15.73:0/841552163 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2e700f51a0 0x7f2e700f55c0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7f2e64009fb0 tx=0x7f2e64009b30).stop 2023-12-01T22:14:20.136 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:20.134+0000 7f2e74827700 1 -- 172.21.15.73:0/841552163 shutdown_connections 2023-12-01T22:14:20.136 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:20.134+0000 7f2e74827700 1 --2- 172.21.15.73:0/841552163 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2e700f51a0 0x7f2e700f55c0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:14:20.137 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:20.134+0000 7f2e74827700 1 -- 172.21.15.73:0/841552163 >> 172.21.15.73:0/841552163 conn(0x7f2e700f09a0 msgr2=0x7f2e700f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:14:20.137 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:20.134+0000 7f2e74827700 1 -- 172.21.15.73:0/841552163 shutdown_connections 2023-12-01T22:14:20.137 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:20.134+0000 7f2e74827700 1 -- 172.21.15.73:0/841552163 wait complete. 2023-12-01T22:14:20.137 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:20.134+0000 7f2e74827700 1 Processor -- start 2023-12-01T22:14:20.137 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:20.134+0000 7f2e74827700 1 -- start start 2023-12-01T22:14:20.137 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:20.134+0000 7f2e74827700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2e700f51a0 0x7f2e701064c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:14:20.137 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:20.134+0000 7f2e74827700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e701069b0 con 0x7f2e700f51a0 2023-12-01T22:14:20.137 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:20.135+0000 7f2e6ed9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2e700f51a0 0x7f2e701064c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:14:20.137 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:20.135+0000 7f2e6ed9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2e700f51a0 0x7f2e701064c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:41288/0 (socket says 172.21.15.73:41288) 2023-12-01T22:14:20.137 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:20.135+0000 7f2e6ed9d700 1 -- 172.21.15.73:0/4024773159 learned_addr learned my addr 172.21.15.73:0/4024773159 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:14:20.139 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:20.135+0000 7f2e6ed9d700 1 -- 172.21.15.73:0/4024773159 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e64005040 con 0x7f2e700f51a0 2023-12-01T22:14:20.139 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:20.135+0000 7f2e6ed9d700 1 --2- 172.21.15.73:0/4024773159 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2e700f51a0 0x7f2e701064c0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7f2e64013510 tx=0x7f2e6400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:14:20.139 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:20.135+0000 7f2e577fe700 1 -- 172.21.15.73:0/4024773159 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2e64009320 con 0x7f2e700f51a0 2023-12-01T22:14:20.139 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:20.135+0000 7f2e74827700 1 -- 172.21.15.73:0/4024773159 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e70106bb0 con 0x7f2e700f51a0 2023-12-01T22:14:20.139 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:20.135+0000 7f2e74827700 1 -- 172.21.15.73:0/4024773159 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e70107040 con 0x7f2e700f51a0 2023-12-01T22:14:20.139 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:20.136+0000 7f2e577fe700 1 -- 172.21.15.73:0/4024773159 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2e64013f00 con 0x7f2e700f51a0 2023-12-01T22:14:20.140 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:20.136+0000 7f2e577fe700 1 -- 172.21.15.73:0/4024773159 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2e64014cc0 con 0x7f2e700f51a0 2023-12-01T22:14:20.140 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:20.136+0000 7f2e74827700 1 -- 172.21.15.73:0/4024773159 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e50004fa0 con 0x7f2e700f51a0 2023-12-01T22:14:20.140 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:20.136+0000 7f2e577fe700 1 -- 172.21.15.73:0/4024773159 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2e64014420 con 0x7f2e700f51a0 2023-12-01T22:14:20.140 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:20.136+0000 7f2e577fe700 1 --2- 172.21.15.73:0/4024773159 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f2e5802ff60 0x7f2e58032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:14:20.140 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:20.136+0000 7f2e577fe700 1 -- 172.21.15.73:0/4024773159 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2e64040f00 con 0x7f2e700f51a0 2023-12-01T22:14:20.140 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:20.137+0000 7f2e6e59c700 1 --2- 172.21.15.73:0/4024773159 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f2e5802ff60 0x7f2e58032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:14:20.141 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:20.137+0000 7f2e6e59c700 1 --2- 172.21.15.73:0/4024773159 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f2e5802ff60 0x7f2e58032420 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f2e600096a0 tx=0x7f2e60006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:14:20.143 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:20.142+0000 7f2e577fe700 1 -- 172.21.15.73:0/4024773159 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2e640095f0 con 0x7f2e700f51a0 2023-12-01T22:14:20.350 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:20.349+0000 7f2e74827700 1 -- 172.21.15.73:0/4024773159 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2e50005670 con 0x7f2e700f51a0 2023-12-01T22:14:20.355 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:20.350+0000 7f2e577fe700 1 -- 172.21.15.73:0/4024773159 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2e64010e60 con 0x7f2e700f51a0 2023-12-01T22:14:20.355 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:14:20.355 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:14:20.356 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:20.352+0000 7f2e74827700 1 -- 172.21.15.73:0/4024773159 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f2e5802ff60 msgr2=0x7f2e58032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:14:20.356 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:20.352+0000 7f2e74827700 1 --2- 172.21.15.73:0/4024773159 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f2e5802ff60 0x7f2e58032420 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f2e600096a0 tx=0x7f2e60006b40).stop 2023-12-01T22:14:20.356 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:20.352+0000 7f2e74827700 1 -- 172.21.15.73:0/4024773159 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2e700f51a0 msgr2=0x7f2e701064c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:14:20.356 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:20.352+0000 7f2e74827700 1 --2- 172.21.15.73:0/4024773159 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2e700f51a0 0x7f2e701064c0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7f2e64013510 tx=0x7f2e6400b040).stop 2023-12-01T22:14:20.356 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:20.352+0000 7f2e74827700 1 -- 172.21.15.73:0/4024773159 shutdown_connections 2023-12-01T22:14:20.356 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:20.352+0000 7f2e74827700 1 --2- 172.21.15.73:0/4024773159 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f2e5802ff60 0x7f2e58032420 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:14:20.356 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:20.352+0000 7f2e74827700 1 --2- 172.21.15.73:0/4024773159 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2e700f51a0 0x7f2e701064c0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:14:20.356 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:20.352+0000 7f2e74827700 1 -- 172.21.15.73:0/4024773159 >> 172.21.15.73:0/4024773159 conn(0x7f2e700f09a0 msgr2=0x7f2e700f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:14:20.356 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:20.352+0000 7f2e74827700 1 -- 172.21.15.73:0/4024773159 shutdown_connections 2023-12-01T22:14:20.357 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:20.352+0000 7f2e74827700 1 -- 172.21.15.73:0/4024773159 wait complete. 2023-12-01T22:14:20.357 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:14:21.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:14:20 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:14:20.350314+0000 mon.smithi029 (mon.0) 180 : audit [DBG] from='client.? 172.21.15.73:0/4024773159' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:14:21.736 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:14:21.736 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:14:22.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:14:21 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:14:20.647358+0000 mgr.smithi029.loxrfu (mgr.14164) 95 : cluster [DBG] pgmap v88: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:14:22.889 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:22.888+0000 7f6226035700 1 -- 172.21.15.73:0/1061090429 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62200f6510 msgr2=0x7f62200f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:14:22.889 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:22.888+0000 7f6226035700 1 --2- 172.21.15.73:0/1061090429 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62200f6510 0x7f62200f6930 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7f6214009fb0 tx=0x7f6214009b30).stop 2023-12-01T22:14:22.890 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:22.888+0000 7f6226035700 1 -- 172.21.15.73:0/1061090429 shutdown_connections 2023-12-01T22:14:22.890 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:22.888+0000 7f6226035700 1 --2- 172.21.15.73:0/1061090429 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62200f6510 0x7f62200f6930 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:14:22.890 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:22.888+0000 7f6226035700 1 -- 172.21.15.73:0/1061090429 >> 172.21.15.73:0/1061090429 conn(0x7f62200f1c70 msgr2=0x7f62200f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:14:22.890 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:22.888+0000 7f6226035700 1 -- 172.21.15.73:0/1061090429 shutdown_connections 2023-12-01T22:14:22.890 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:22.888+0000 7f6226035700 1 -- 172.21.15.73:0/1061090429 wait complete. 2023-12-01T22:14:22.890 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:22.889+0000 7f6226035700 1 Processor -- start 2023-12-01T22:14:22.891 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:22.889+0000 7f6226035700 1 -- start start 2023-12-01T22:14:22.891 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:22.889+0000 7f6226035700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62200f6510 0x7f622006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:14:22.892 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:22.889+0000 7f6226035700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6220069e20 con 0x7f62200f6510 2023-12-01T22:14:22.892 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:22.890+0000 7f621f7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62200f6510 0x7f622006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:14:22.892 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:22.890+0000 7f621f7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62200f6510 0x7f622006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:41300/0 (socket says 172.21.15.73:41300) 2023-12-01T22:14:22.892 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:22.890+0000 7f621f7fe700 1 -- 172.21.15.73:0/1989767751 learned_addr learned my addr 172.21.15.73:0/1989767751 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:14:22.892 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:22.890+0000 7f621f7fe700 1 -- 172.21.15.73:0/1989767751 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6214005040 con 0x7f62200f6510 2023-12-01T22:14:22.892 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:22.890+0000 7f621f7fe700 1 --2- 172.21.15.73:0/1989767751 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62200f6510 0x7f622006b9a0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7f6214014000 tx=0x7f621400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:14:22.892 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:22.890+0000 7f6207fff700 1 -- 172.21.15.73:0/1989767751 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f62140095f0 con 0x7f62200f6510 2023-12-01T22:14:22.892 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:22.890+0000 7f6226035700 1 -- 172.21.15.73:0/1989767751 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6220069fc0 con 0x7f62200f6510 2023-12-01T22:14:22.892 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:22.890+0000 7f6226035700 1 -- 172.21.15.73:0/1989767751 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f622006a450 con 0x7f62200f6510 2023-12-01T22:14:22.892 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:22.891+0000 7f6207fff700 1 -- 172.21.15.73:0/1989767751 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6214007e70 con 0x7f62200f6510 2023-12-01T22:14:22.892 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:22.891+0000 7f6207fff700 1 -- 172.21.15.73:0/1989767751 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6214014ce0 con 0x7f62200f6510 2023-12-01T22:14:22.893 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:22.891+0000 7f6226035700 1 -- 172.21.15.73:0/1989767751 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6200004fa0 con 0x7f62200f6510 2023-12-01T22:14:22.893 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:22.891+0000 7f6207fff700 1 -- 172.21.15.73:0/1989767751 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6214025070 con 0x7f62200f6510 2023-12-01T22:14:22.893 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:22.891+0000 7f6207fff700 1 --2- 172.21.15.73:0/1989767751 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f620802ff60 0x7f6208032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:14:22.893 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:22.891+0000 7f6207fff700 1 -- 172.21.15.73:0/1989767751 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6214040b10 con 0x7f62200f6510 2023-12-01T22:14:22.893 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:22.892+0000 7f621effd700 1 --2- 172.21.15.73:0/1989767751 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f620802ff60 0x7f6208032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:14:22.893 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:22.892+0000 7f621effd700 1 --2- 172.21.15.73:0/1989767751 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f620802ff60 0x7f6208032420 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f62100096a0 tx=0x7f6210006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:14:22.897 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:22.896+0000 7f6207fff700 1 -- 172.21.15.73:0/1989767751 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f62140254e0 con 0x7f62200f6510 2023-12-01T22:14:23.103 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:23.102+0000 7f6226035700 1 -- 172.21.15.73:0/1989767751 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6200005670 con 0x7f62200f6510 2023-12-01T22:14:23.104 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:23.103+0000 7f6207fff700 1 -- 172.21.15.73:0/1989767751 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6214025300 con 0x7f62200f6510 2023-12-01T22:14:23.104 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:14:23.104 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:14:23.105 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:23.105+0000 7f6226035700 1 -- 172.21.15.73:0/1989767751 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f620802ff60 msgr2=0x7f6208032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:14:23.105 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:23.105+0000 7f6226035700 1 --2- 172.21.15.73:0/1989767751 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f620802ff60 0x7f6208032420 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f62100096a0 tx=0x7f6210006b40).stop 2023-12-01T22:14:23.106 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:23.105+0000 7f6226035700 1 -- 172.21.15.73:0/1989767751 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62200f6510 msgr2=0x7f622006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:14:23.106 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:23.105+0000 7f6226035700 1 --2- 172.21.15.73:0/1989767751 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62200f6510 0x7f622006b9a0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7f6214014000 tx=0x7f621400b040).stop 2023-12-01T22:14:23.106 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:23.105+0000 7f6226035700 1 -- 172.21.15.73:0/1989767751 shutdown_connections 2023-12-01T22:14:23.106 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:23.105+0000 7f6226035700 1 --2- 172.21.15.73:0/1989767751 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f620802ff60 0x7f6208032420 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:14:23.106 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:23.105+0000 7f6226035700 1 --2- 172.21.15.73:0/1989767751 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62200f6510 0x7f622006b9a0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:14:23.106 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:23.105+0000 7f6226035700 1 -- 172.21.15.73:0/1989767751 >> 172.21.15.73:0/1989767751 conn(0x7f62200f1c70 msgr2=0x7f62200f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:14:23.106 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:23.105+0000 7f6226035700 1 -- 172.21.15.73:0/1989767751 shutdown_connections 2023-12-01T22:14:23.106 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:23.105+0000 7f6226035700 1 -- 172.21.15.73:0/1989767751 wait complete. 2023-12-01T22:14:23.106 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:14:24.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:14:23 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:14:22.647684+0000 mgr.smithi029.loxrfu (mgr.14164) 96 : cluster [DBG] pgmap v89: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:14:24.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:14:23 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:14:23.103453+0000 mon.smithi029 (mon.0) 181 : audit [DBG] from='client.? 172.21.15.73:0/1989767751' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:14:24.483 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:14:24.483 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:14:25.629 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:25.628+0000 7fa20b476700 1 -- 172.21.15.73:0/2466377352 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2040f6510 msgr2=0x7fa2040f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:14:25.630 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:25.628+0000 7fa20b476700 1 --2- 172.21.15.73:0/2466377352 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2040f6510 0x7fa2040f6930 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7fa1f4009fb0 tx=0x7fa1f4009b30).stop 2023-12-01T22:14:25.630 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:25.628+0000 7fa20b476700 1 -- 172.21.15.73:0/2466377352 shutdown_connections 2023-12-01T22:14:25.630 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:25.628+0000 7fa20b476700 1 --2- 172.21.15.73:0/2466377352 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2040f6510 0x7fa2040f6930 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:14:25.630 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:25.628+0000 7fa20b476700 1 -- 172.21.15.73:0/2466377352 >> 172.21.15.73:0/2466377352 conn(0x7fa2040f1c70 msgr2=0x7fa2040f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:14:25.630 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:25.628+0000 7fa20b476700 1 -- 172.21.15.73:0/2466377352 shutdown_connections 2023-12-01T22:14:25.630 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:25.628+0000 7fa20b476700 1 -- 172.21.15.73:0/2466377352 wait complete. 2023-12-01T22:14:25.630 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:25.629+0000 7fa20b476700 1 Processor -- start 2023-12-01T22:14:25.631 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:25.629+0000 7fa20b476700 1 -- start start 2023-12-01T22:14:25.631 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:25.629+0000 7fa20b476700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2040f6510 0x7fa204106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:14:25.631 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:25.629+0000 7fa20b476700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa204106c20 con 0x7fa2040f6510 2023-12-01T22:14:25.633 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:25.629+0000 7fa209212700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2040f6510 0x7fa204106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:14:25.633 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:25.629+0000 7fa209212700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2040f6510 0x7fa204106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:60612/0 (socket says 172.21.15.73:60612) 2023-12-01T22:14:25.633 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:25.629+0000 7fa209212700 1 -- 172.21.15.73:0/287802263 learned_addr learned my addr 172.21.15.73:0/287802263 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:14:25.633 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:25.630+0000 7fa209212700 1 -- 172.21.15.73:0/287802263 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1f4005040 con 0x7fa2040f6510 2023-12-01T22:14:25.633 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:25.630+0000 7fa209212700 1 --2- 172.21.15.73:0/287802263 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2040f6510 0x7fa204106730 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7fa1f4013510 tx=0x7fa1f400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:14:25.634 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:25.630+0000 7fa1f9ffb700 1 -- 172.21.15.73:0/287802263 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa1f4009320 con 0x7fa2040f6510 2023-12-01T22:14:25.634 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:25.630+0000 7fa20b476700 1 -- 172.21.15.73:0/287802263 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa204106e20 con 0x7fa2040f6510 2023-12-01T22:14:25.634 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:25.630+0000 7fa1f9ffb700 1 -- 172.21.15.73:0/287802263 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa1f4013f00 con 0x7fa2040f6510 2023-12-01T22:14:25.634 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:25.630+0000 7fa1f9ffb700 1 -- 172.21.15.73:0/287802263 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa1f4014d90 con 0x7fa2040f6510 2023-12-01T22:14:25.634 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:25.630+0000 7fa20b476700 1 -- 172.21.15.73:0/287802263 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2041072b0 con 0x7fa2040f6510 2023-12-01T22:14:25.634 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:25.631+0000 7fa1f9ffb700 1 -- 172.21.15.73:0/287802263 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa1f40095f0 con 0x7fa2040f6510 2023-12-01T22:14:25.634 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:25.631+0000 7fa20b476700 1 -- 172.21.15.73:0/287802263 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa204045ad0 con 0x7fa2040f6510 2023-12-01T22:14:25.634 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:25.631+0000 7fa1f9ffb700 1 --2- 172.21.15.73:0/287802263 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fa1f002ffb0 0x7fa1f0032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:14:25.634 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:25.631+0000 7fa1f9ffb700 1 -- 172.21.15.73:0/287802263 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa1f4041060 con 0x7fa2040f6510 2023-12-01T22:14:25.634 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:25.631+0000 7fa208a11700 1 --2- 172.21.15.73:0/287802263 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fa1f002ffb0 0x7fa1f0032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:14:25.634 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:25.632+0000 7fa208a11700 1 --2- 172.21.15.73:0/287802263 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fa1f002ffb0 0x7fa1f0032470 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7fa2000096a0 tx=0x7fa200006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:14:25.637 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:25.636+0000 7fa1f9ffb700 1 -- 172.21.15.73:0/287802263 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa1f406d2b0 con 0x7fa2040f6510 2023-12-01T22:14:25.841 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:25.841+0000 7fa20b476700 1 -- 172.21.15.73:0/287802263 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa2040fbf60 con 0x7fa2040f6510 2023-12-01T22:14:25.842 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:25.842+0000 7fa1f9ffb700 1 -- 172.21.15.73:0/287802263 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa1f4014420 con 0x7fa2040f6510 2023-12-01T22:14:25.842 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:14:25.842 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:14:25.844 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:25.843+0000 7fa20b476700 1 -- 172.21.15.73:0/287802263 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fa1f002ffb0 msgr2=0x7fa1f0032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:14:25.844 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:25.843+0000 7fa20b476700 1 --2- 172.21.15.73:0/287802263 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fa1f002ffb0 0x7fa1f0032470 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7fa2000096a0 tx=0x7fa200006b40).stop 2023-12-01T22:14:25.844 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:25.843+0000 7fa20b476700 1 -- 172.21.15.73:0/287802263 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2040f6510 msgr2=0x7fa204106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:14:25.844 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:25.843+0000 7fa20b476700 1 --2- 172.21.15.73:0/287802263 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2040f6510 0x7fa204106730 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7fa1f4013510 tx=0x7fa1f400b040).stop 2023-12-01T22:14:25.844 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:25.843+0000 7fa20b476700 1 -- 172.21.15.73:0/287802263 shutdown_connections 2023-12-01T22:14:25.844 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:25.843+0000 7fa20b476700 1 --2- 172.21.15.73:0/287802263 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fa1f002ffb0 0x7fa1f0032470 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:14:25.844 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:25.843+0000 7fa20b476700 1 --2- 172.21.15.73:0/287802263 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2040f6510 0x7fa204106730 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:14:25.844 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:25.843+0000 7fa20b476700 1 -- 172.21.15.73:0/287802263 >> 172.21.15.73:0/287802263 conn(0x7fa2040f1c70 msgr2=0x7fa2040f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:14:25.844 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:25.843+0000 7fa20b476700 1 -- 172.21.15.73:0/287802263 shutdown_connections 2023-12-01T22:14:25.845 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:25.843+0000 7fa20b476700 1 -- 172.21.15.73:0/287802263 wait complete. 2023-12-01T22:14:25.845 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:14:26.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:14:25 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:14:24.648104+0000 mgr.smithi029.loxrfu (mgr.14164) 97 : cluster [DBG] pgmap v90: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:14:27.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:14:26 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:14:25.841834+0000 mon.smithi029 (mon.0) 182 : audit [DBG] from='client.? 172.21.15.73:0/287802263' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:14:27.226 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:14:27.227 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:14:28.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:14:27 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:14:26.648467+0000 mgr.smithi029.loxrfu (mgr.14164) 98 : cluster [DBG] pgmap v91: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:14:28.373 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:28.370+0000 7f1377988700 1 -- 172.21.15.73:0/1936644419 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13700f5410 msgr2=0x7f13700f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:14:28.373 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:28.370+0000 7f1377988700 1 --2- 172.21.15.73:0/1936644419 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13700f5410 0x7f13700f5830 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7f1360009fb0 tx=0x7f1360009b30).stop 2023-12-01T22:14:28.373 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:28.370+0000 7f1377988700 1 -- 172.21.15.73:0/1936644419 shutdown_connections 2023-12-01T22:14:28.373 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:28.370+0000 7f1377988700 1 --2- 172.21.15.73:0/1936644419 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13700f5410 0x7f13700f5830 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:14:28.373 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:28.370+0000 7f1377988700 1 -- 172.21.15.73:0/1936644419 >> 172.21.15.73:0/1936644419 conn(0x7f13700f0b70 msgr2=0x7f13700f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:14:28.373 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:28.371+0000 7f1377988700 1 -- 172.21.15.73:0/1936644419 shutdown_connections 2023-12-01T22:14:28.373 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:28.371+0000 7f1377988700 1 -- 172.21.15.73:0/1936644419 wait complete. 2023-12-01T22:14:28.373 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:28.371+0000 7f1377988700 1 Processor -- start 2023-12-01T22:14:28.374 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:28.371+0000 7f1377988700 1 -- start start 2023-12-01T22:14:28.374 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:28.371+0000 7f1377988700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13700f5410 0x7f137010b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:14:28.374 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:28.371+0000 7f1377988700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1370107210 con 0x7f13700f5410 2023-12-01T22:14:28.374 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:28.372+0000 7f1375724700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13700f5410 0x7f137010b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:14:28.374 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:28.372+0000 7f1375724700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13700f5410 0x7f137010b820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:60634/0 (socket says 172.21.15.73:60634) 2023-12-01T22:14:28.374 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:28.372+0000 7f1375724700 1 -- 172.21.15.73:0/4094114603 learned_addr learned my addr 172.21.15.73:0/4094114603 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:14:28.374 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:28.372+0000 7f1375724700 1 -- 172.21.15.73:0/4094114603 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1360005040 con 0x7f13700f5410 2023-12-01T22:14:28.374 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:28.372+0000 7f1375724700 1 --2- 172.21.15.73:0/4094114603 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13700f5410 0x7f137010b820 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7f1360013510 tx=0x7f136000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:14:28.374 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:28.372+0000 7f1365ffb700 1 -- 172.21.15.73:0/4094114603 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1360009320 con 0x7f13700f5410 2023-12-01T22:14:28.374 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:28.372+0000 7f1377988700 1 -- 172.21.15.73:0/4094114603 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1370060220 con 0x7f13700f5410 2023-12-01T22:14:28.374 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:28.372+0000 7f1365ffb700 1 -- 172.21.15.73:0/4094114603 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1360013f00 con 0x7f13700f5410 2023-12-01T22:14:28.374 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:28.372+0000 7f1365ffb700 1 -- 172.21.15.73:0/4094114603 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1360014d90 con 0x7f13700f5410 2023-12-01T22:14:28.374 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:28.372+0000 7f1377988700 1 -- 172.21.15.73:0/4094114603 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f137010bd10 con 0x7f13700f5410 2023-12-01T22:14:28.375 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:28.373+0000 7f1365ffb700 1 -- 172.21.15.73:0/4094114603 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f13600095f0 con 0x7f13700f5410 2023-12-01T22:14:28.375 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:28.373+0000 7f1377988700 1 -- 172.21.15.73:0/4094114603 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1370045ad0 con 0x7f13700f5410 2023-12-01T22:14:28.375 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:28.373+0000 7f1365ffb700 1 --2- 172.21.15.73:0/4094114603 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f135c02ff60 0x7f135c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:14:28.375 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:28.373+0000 7f1365ffb700 1 -- 172.21.15.73:0/4094114603 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1360040fc0 con 0x7f13700f5410 2023-12-01T22:14:28.376 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:28.374+0000 7f1374f23700 1 --2- 172.21.15.73:0/4094114603 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f135c02ff60 0x7f135c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:14:28.376 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:28.374+0000 7f1374f23700 1 --2- 172.21.15.73:0/4094114603 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f135c02ff60 0x7f135c032420 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f136c0096a0 tx=0x7f136c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:14:28.379 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:28.378+0000 7f1365ffb700 1 -- 172.21.15.73:0/4094114603 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1360014420 con 0x7f13700f5410 2023-12-01T22:14:28.588 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:28.587+0000 7f1377988700 1 -- 172.21.15.73:0/4094114603 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f13700fae60 con 0x7f13700f5410 2023-12-01T22:14:28.589 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:28.588+0000 7f1365ffb700 1 -- 172.21.15.73:0/4094114603 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1360010230 con 0x7f13700f5410 2023-12-01T22:14:28.589 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:14:28.589 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:14:28.591 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:28.589+0000 7f1377988700 1 -- 172.21.15.73:0/4094114603 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f135c02ff60 msgr2=0x7f135c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:14:28.591 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:28.589+0000 7f1377988700 1 --2- 172.21.15.73:0/4094114603 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f135c02ff60 0x7f135c032420 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f136c0096a0 tx=0x7f136c006b40).stop 2023-12-01T22:14:28.591 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:28.589+0000 7f1377988700 1 -- 172.21.15.73:0/4094114603 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13700f5410 msgr2=0x7f137010b820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:14:28.591 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:28.589+0000 7f1377988700 1 --2- 172.21.15.73:0/4094114603 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13700f5410 0x7f137010b820 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7f1360013510 tx=0x7f136000b040).stop 2023-12-01T22:14:28.591 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:28.589+0000 7f1377988700 1 -- 172.21.15.73:0/4094114603 shutdown_connections 2023-12-01T22:14:28.591 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:28.589+0000 7f1377988700 1 --2- 172.21.15.73:0/4094114603 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f135c02ff60 0x7f135c032420 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:14:28.591 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:28.589+0000 7f1377988700 1 --2- 172.21.15.73:0/4094114603 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13700f5410 0x7f137010b820 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:14:28.591 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:28.589+0000 7f1377988700 1 -- 172.21.15.73:0/4094114603 >> 172.21.15.73:0/4094114603 conn(0x7f13700f0b70 msgr2=0x7f13700f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:14:28.592 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:28.589+0000 7f1377988700 1 -- 172.21.15.73:0/4094114603 shutdown_connections 2023-12-01T22:14:28.592 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:28.589+0000 7f1377988700 1 -- 172.21.15.73:0/4094114603 wait complete. 2023-12-01T22:14:28.592 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:14:29.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:14:28 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:14:28.587985+0000 mon.smithi029 (mon.0) 183 : audit [DBG] from='client.? 172.21.15.73:0/4094114603' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:14:29.981 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:14:29.981 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:14:30.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:14:29 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:14:28.648817+0000 mgr.smithi029.loxrfu (mgr.14164) 99 : cluster [DBG] pgmap v92: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:14:31.150 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:31.148+0000 7f2276bab700 1 -- 172.21.15.73:0/319777381 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f22700f6510 msgr2=0x7f22700f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:14:31.150 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:31.148+0000 7f2276bab700 1 --2- 172.21.15.73:0/319777381 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f22700f6510 0x7f22700f6930 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7f2264009fb0 tx=0x7f2264009b30).stop 2023-12-01T22:14:31.151 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:31.148+0000 7f2276bab700 1 -- 172.21.15.73:0/319777381 shutdown_connections 2023-12-01T22:14:31.151 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:31.148+0000 7f2276bab700 1 --2- 172.21.15.73:0/319777381 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f22700f6510 0x7f22700f6930 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:14:31.151 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:31.148+0000 7f2276bab700 1 -- 172.21.15.73:0/319777381 >> 172.21.15.73:0/319777381 conn(0x7f22700f1c70 msgr2=0x7f22700f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:14:31.151 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:31.148+0000 7f2276bab700 1 -- 172.21.15.73:0/319777381 shutdown_connections 2023-12-01T22:14:31.151 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:31.148+0000 7f2276bab700 1 -- 172.21.15.73:0/319777381 wait complete. 2023-12-01T22:14:31.151 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:31.148+0000 7f2276bab700 1 Processor -- start 2023-12-01T22:14:31.151 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:31.148+0000 7f2276bab700 1 -- start start 2023-12-01T22:14:31.151 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:31.149+0000 7f2276bab700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f22700f6510 0x7f2270106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:14:31.151 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:31.149+0000 7f2276bab700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2270106c20 con 0x7f22700f6510 2023-12-01T22:14:31.153 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:31.149+0000 7f2274947700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f22700f6510 0x7f2270106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:14:31.153 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:31.149+0000 7f2274947700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f22700f6510 0x7f2270106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:60652/0 (socket says 172.21.15.73:60652) 2023-12-01T22:14:31.153 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:31.149+0000 7f2274947700 1 -- 172.21.15.73:0/2530422310 learned_addr learned my addr 172.21.15.73:0/2530422310 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:14:31.153 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:31.150+0000 7f2274947700 1 -- 172.21.15.73:0/2530422310 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2264005040 con 0x7f22700f6510 2023-12-01T22:14:31.153 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:31.150+0000 7f2274947700 1 --2- 172.21.15.73:0/2530422310 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f22700f6510 0x7f2270106730 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7f2264013510 tx=0x7f226400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:14:31.153 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:31.150+0000 7f226d7fa700 1 -- 172.21.15.73:0/2530422310 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2264009320 con 0x7f22700f6510 2023-12-01T22:14:31.153 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:31.150+0000 7f2276bab700 1 -- 172.21.15.73:0/2530422310 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2270106e20 con 0x7f22700f6510 2023-12-01T22:14:31.153 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:31.150+0000 7f226d7fa700 1 -- 172.21.15.73:0/2530422310 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2264013f00 con 0x7f22700f6510 2023-12-01T22:14:31.153 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:31.150+0000 7f226d7fa700 1 -- 172.21.15.73:0/2530422310 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2264014d90 con 0x7f22700f6510 2023-12-01T22:14:31.153 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:31.150+0000 7f2276bab700 1 -- 172.21.15.73:0/2530422310 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22701072b0 con 0x7f22700f6510 2023-12-01T22:14:31.154 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:31.151+0000 7f226d7fa700 1 -- 172.21.15.73:0/2530422310 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f22640095f0 con 0x7f22700f6510 2023-12-01T22:14:31.154 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:31.151+0000 7f2276bab700 1 -- 172.21.15.73:0/2530422310 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2270045ad0 con 0x7f22700f6510 2023-12-01T22:14:31.154 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:31.151+0000 7f226d7fa700 1 --2- 172.21.15.73:0/2530422310 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f226002ffb0 0x7f2260032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:14:31.154 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:31.151+0000 7f226d7fa700 1 -- 172.21.15.73:0/2530422310 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2264040fe0 con 0x7f22700f6510 2023-12-01T22:14:31.154 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:31.151+0000 7f226ffff700 1 --2- 172.21.15.73:0/2530422310 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f226002ffb0 0x7f2260032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:14:31.154 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:31.151+0000 7f226ffff700 1 --2- 172.21.15.73:0/2530422310 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f226002ffb0 0x7f2260032470 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f225c0096a0 tx=0x7f225c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:14:31.157 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:31.156+0000 7f226d7fa700 1 -- 172.21.15.73:0/2530422310 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f226406d2b0 con 0x7f22700f6510 2023-12-01T22:14:31.364 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:31.363+0000 7f2276bab700 1 -- 172.21.15.73:0/2530422310 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f22700fbf60 con 0x7f22700f6510 2023-12-01T22:14:31.365 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:31.363+0000 7f226d7fa700 1 -- 172.21.15.73:0/2530422310 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2264014420 con 0x7f22700f6510 2023-12-01T22:14:31.365 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:14:31.365 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:14:31.366 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:31.365+0000 7f2276bab700 1 -- 172.21.15.73:0/2530422310 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f226002ffb0 msgr2=0x7f2260032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:14:31.367 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:31.365+0000 7f2276bab700 1 --2- 172.21.15.73:0/2530422310 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f226002ffb0 0x7f2260032470 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f225c0096a0 tx=0x7f225c006b40).stop 2023-12-01T22:14:31.367 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:31.365+0000 7f2276bab700 1 -- 172.21.15.73:0/2530422310 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f22700f6510 msgr2=0x7f2270106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:14:31.367 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:31.365+0000 7f2276bab700 1 --2- 172.21.15.73:0/2530422310 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f22700f6510 0x7f2270106730 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7f2264013510 tx=0x7f226400b040).stop 2023-12-01T22:14:31.367 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:31.365+0000 7f2276bab700 1 -- 172.21.15.73:0/2530422310 shutdown_connections 2023-12-01T22:14:31.367 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:31.365+0000 7f2276bab700 1 --2- 172.21.15.73:0/2530422310 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f226002ffb0 0x7f2260032470 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:14:31.367 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:31.365+0000 7f2276bab700 1 --2- 172.21.15.73:0/2530422310 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f22700f6510 0x7f2270106730 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:14:31.367 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:31.365+0000 7f2276bab700 1 -- 172.21.15.73:0/2530422310 >> 172.21.15.73:0/2530422310 conn(0x7f22700f1c70 msgr2=0x7f22700f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:14:31.367 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:31.365+0000 7f2276bab700 1 -- 172.21.15.73:0/2530422310 shutdown_connections 2023-12-01T22:14:31.367 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:31.365+0000 7f2276bab700 1 -- 172.21.15.73:0/2530422310 wait complete. 2023-12-01T22:14:31.367 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:14:31.776 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:14:31 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:14:30.649132+0000 mgr.smithi029.loxrfu (mgr.14164) 100 : cluster [DBG] pgmap v93: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:14:31.777 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:14:31 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:14:31.364705+0000 mon.smithi029 (mon.0) 184 : audit [DBG] from='client.? 172.21.15.73:0/2530422310' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:14:32.741 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:14:32.741 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:14:33.912 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:33.909+0000 7ff25629c700 1 -- 172.21.15.73:0/2923803925 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2500f4740 msgr2=0x7ff2500f4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:14:33.912 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:33.909+0000 7ff25629c700 1 --2- 172.21.15.73:0/2923803925 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2500f4740 0x7ff2500f4b60 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7ff244009fb0 tx=0x7ff244009b30).stop 2023-12-01T22:14:33.912 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:33.909+0000 7ff25629c700 1 -- 172.21.15.73:0/2923803925 shutdown_connections 2023-12-01T22:14:33.912 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:33.909+0000 7ff25629c700 1 --2- 172.21.15.73:0/2923803925 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2500f4740 0x7ff2500f4b60 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:14:33.912 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:33.909+0000 7ff25629c700 1 -- 172.21.15.73:0/2923803925 >> 172.21.15.73:0/2923803925 conn(0x7ff2500f0260 msgr2=0x7ff2500f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:14:33.912 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:33.909+0000 7ff25629c700 1 -- 172.21.15.73:0/2923803925 shutdown_connections 2023-12-01T22:14:33.912 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:33.909+0000 7ff25629c700 1 -- 172.21.15.73:0/2923803925 wait complete. 2023-12-01T22:14:33.912 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:33.910+0000 7ff25629c700 1 Processor -- start 2023-12-01T22:14:33.912 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:33.910+0000 7ff25629c700 1 -- start start 2023-12-01T22:14:33.912 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:33.910+0000 7ff25629c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2500f4740 0x7ff2501066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:14:33.912 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:33.910+0000 7ff25629c700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff250106bc0 con 0x7ff2500f4740 2023-12-01T22:14:33.913 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:33.910+0000 7ff24ffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2500f4740 0x7ff2501066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:14:33.913 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:33.910+0000 7ff24ffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2500f4740 0x7ff2501066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:60672/0 (socket says 172.21.15.73:60672) 2023-12-01T22:14:33.913 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:33.910+0000 7ff24ffff700 1 -- 172.21.15.73:0/2140327380 learned_addr learned my addr 172.21.15.73:0/2140327380 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:14:33.914 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:33.911+0000 7ff24ffff700 1 -- 172.21.15.73:0/2140327380 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff244005040 con 0x7ff2500f4740 2023-12-01T22:14:33.914 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:33.911+0000 7ff24ffff700 1 --2- 172.21.15.73:0/2140327380 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2500f4740 0x7ff2501066d0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7ff244013510 tx=0x7ff24400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:14:33.914 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:33.911+0000 7ff24cff9700 1 -- 172.21.15.73:0/2140327380 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff244009320 con 0x7ff2500f4740 2023-12-01T22:14:33.914 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:33.911+0000 7ff25629c700 1 -- 172.21.15.73:0/2140327380 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff250106dc0 con 0x7ff2500f4740 2023-12-01T22:14:33.914 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:33.911+0000 7ff24cff9700 1 -- 172.21.15.73:0/2140327380 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff244013f00 con 0x7ff2500f4740 2023-12-01T22:14:33.915 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:33.911+0000 7ff24cff9700 1 -- 172.21.15.73:0/2140327380 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff244014d90 con 0x7ff2500f4740 2023-12-01T22:14:33.915 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:33.911+0000 7ff25629c700 1 -- 172.21.15.73:0/2140327380 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff250107250 con 0x7ff2500f4740 2023-12-01T22:14:33.915 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:33.912+0000 7ff24cff9700 1 -- 172.21.15.73:0/2140327380 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff2440263a0 con 0x7ff2500f4740 2023-12-01T22:14:33.915 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:33.912+0000 7ff24cff9700 1 --2- 172.21.15.73:0/2140327380 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7ff23802ff60 0x7ff238032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:14:33.915 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:33.912+0000 7ff24cff9700 1 -- 172.21.15.73:0/2140327380 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff244041090 con 0x7ff2500f4740 2023-12-01T22:14:33.915 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:33.912+0000 7ff25629c700 1 -- 172.21.15.73:0/2140327380 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff250045ad0 con 0x7ff2500f4740 2023-12-01T22:14:33.915 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:33.912+0000 7ff24f7fe700 1 --2- 172.21.15.73:0/2140327380 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7ff23802ff60 0x7ff238032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:14:33.915 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:33.913+0000 7ff24f7fe700 1 --2- 172.21.15.73:0/2140327380 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7ff23802ff60 0x7ff238032420 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7ff2400096a0 tx=0x7ff240006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:14:33.919 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:33.917+0000 7ff24cff9700 1 -- 172.21.15.73:0/2140327380 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff244012920 con 0x7ff2500f4740 2023-12-01T22:14:34.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:14:33 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:14:32.649352+0000 mgr.smithi029.loxrfu (mgr.14164) 101 : cluster [DBG] pgmap v94: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:14:34.125 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:34.124+0000 7ff25629c700 1 -- 172.21.15.73:0/2140327380 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff2500fbf40 con 0x7ff2500f4740 2023-12-01T22:14:34.126 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:34.124+0000 7ff24cff9700 1 -- 172.21.15.73:0/2140327380 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff24402b020 con 0x7ff2500f4740 2023-12-01T22:14:34.126 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:14:34.126 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:14:34.128 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:34.126+0000 7ff25629c700 1 -- 172.21.15.73:0/2140327380 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7ff23802ff60 msgr2=0x7ff238032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:14:34.128 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:34.126+0000 7ff25629c700 1 --2- 172.21.15.73:0/2140327380 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7ff23802ff60 0x7ff238032420 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7ff2400096a0 tx=0x7ff240006b40).stop 2023-12-01T22:14:34.128 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:34.126+0000 7ff25629c700 1 -- 172.21.15.73:0/2140327380 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2500f4740 msgr2=0x7ff2501066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:14:34.128 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:34.126+0000 7ff25629c700 1 --2- 172.21.15.73:0/2140327380 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2500f4740 0x7ff2501066d0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7ff244013510 tx=0x7ff24400b040).stop 2023-12-01T22:14:34.128 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:34.126+0000 7ff25629c700 1 -- 172.21.15.73:0/2140327380 shutdown_connections 2023-12-01T22:14:34.128 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:34.126+0000 7ff25629c700 1 --2- 172.21.15.73:0/2140327380 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7ff23802ff60 0x7ff238032420 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:14:34.128 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:34.126+0000 7ff25629c700 1 --2- 172.21.15.73:0/2140327380 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2500f4740 0x7ff2501066d0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:14:34.128 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:34.126+0000 7ff25629c700 1 -- 172.21.15.73:0/2140327380 >> 172.21.15.73:0/2140327380 conn(0x7ff2500f0260 msgr2=0x7ff2500f0d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:14:34.128 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:34.126+0000 7ff25629c700 1 -- 172.21.15.73:0/2140327380 shutdown_connections 2023-12-01T22:14:34.128 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:34.126+0000 7ff25629c700 1 -- 172.21.15.73:0/2140327380 wait complete. 2023-12-01T22:14:34.129 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:14:35.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:14:34 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:14:34.125788+0000 mon.smithi029 (mon.0) 185 : audit [DBG] from='client.? 172.21.15.73:0/2140327380' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:14:35.552 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:14:35.553 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:14:36.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:14:35 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:14:34.649708+0000 mgr.smithi029.loxrfu (mgr.14164) 102 : cluster [DBG] pgmap v95: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:14:36.716 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:36.714+0000 7fb29ae9f700 1 -- 172.21.15.73:0/2780571086 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2940f6510 msgr2=0x7fb2940f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:14:36.716 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:36.714+0000 7fb29ae9f700 1 --2- 172.21.15.73:0/2780571086 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2940f6510 0x7fb2940f6930 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0x7fb288009fb0 tx=0x7fb288009b30).stop 2023-12-01T22:14:36.717 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:36.714+0000 7fb29ae9f700 1 -- 172.21.15.73:0/2780571086 shutdown_connections 2023-12-01T22:14:36.717 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:36.714+0000 7fb29ae9f700 1 --2- 172.21.15.73:0/2780571086 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2940f6510 0x7fb2940f6930 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:14:36.717 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:36.714+0000 7fb29ae9f700 1 -- 172.21.15.73:0/2780571086 >> 172.21.15.73:0/2780571086 conn(0x7fb2940f1c70 msgr2=0x7fb2940f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:14:36.717 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:36.714+0000 7fb29ae9f700 1 -- 172.21.15.73:0/2780571086 shutdown_connections 2023-12-01T22:14:36.717 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:36.714+0000 7fb29ae9f700 1 -- 172.21.15.73:0/2780571086 wait complete. 2023-12-01T22:14:36.717 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:36.714+0000 7fb29ae9f700 1 Processor -- start 2023-12-01T22:14:36.717 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:36.714+0000 7fb29ae9f700 1 -- start start 2023-12-01T22:14:36.717 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:36.715+0000 7fb29ae9f700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2940f6510 0x7fb29406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:14:36.717 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:36.715+0000 7fb29ae9f700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb294069e20 con 0x7fb2940f6510 2023-12-01T22:14:36.719 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:36.715+0000 7fb298c3b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2940f6510 0x7fb29406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:14:36.719 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:36.715+0000 7fb298c3b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2940f6510 0x7fb29406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:54564/0 (socket says 172.21.15.73:54564) 2023-12-01T22:14:36.719 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:36.715+0000 7fb298c3b700 1 -- 172.21.15.73:0/1695634868 learned_addr learned my addr 172.21.15.73:0/1695634868 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:14:36.719 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:36.716+0000 7fb298c3b700 1 -- 172.21.15.73:0/1695634868 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb288005040 con 0x7fb2940f6510 2023-12-01T22:14:36.719 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:36.716+0000 7fb298c3b700 1 --2- 172.21.15.73:0/1695634868 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2940f6510 0x7fb29406b9a0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7fb288013510 tx=0x7fb28800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:14:36.719 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:36.716+0000 7fb2917fa700 1 -- 172.21.15.73:0/1695634868 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb288009320 con 0x7fb2940f6510 2023-12-01T22:14:36.719 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:36.716+0000 7fb29ae9f700 1 -- 172.21.15.73:0/1695634868 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb294069fc0 con 0x7fb2940f6510 2023-12-01T22:14:36.719 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:36.716+0000 7fb2917fa700 1 -- 172.21.15.73:0/1695634868 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb288013f00 con 0x7fb2940f6510 2023-12-01T22:14:36.719 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:36.716+0000 7fb2917fa700 1 -- 172.21.15.73:0/1695634868 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb288014d90 con 0x7fb2940f6510 2023-12-01T22:14:36.719 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:36.716+0000 7fb29ae9f700 1 -- 172.21.15.73:0/1695634868 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb29406a450 con 0x7fb2940f6510 2023-12-01T22:14:36.719 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:36.717+0000 7fb2917fa700 1 -- 172.21.15.73:0/1695634868 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb288014420 con 0x7fb2940f6510 2023-12-01T22:14:36.719 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:36.717+0000 7fb29ae9f700 1 -- 172.21.15.73:0/1695634868 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb294059100 con 0x7fb2940f6510 2023-12-01T22:14:36.720 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:36.717+0000 7fb2917fa700 1 --2- 172.21.15.73:0/1695634868 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fb28402ff60 0x7fb284032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:14:36.720 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:36.717+0000 7fb2917fa700 1 -- 172.21.15.73:0/1695634868 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb288040fb0 con 0x7fb2940f6510 2023-12-01T22:14:36.720 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:36.717+0000 7fb293fff700 1 --2- 172.21.15.73:0/1695634868 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fb28402ff60 0x7fb284032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:14:36.720 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:36.718+0000 7fb293fff700 1 --2- 172.21.15.73:0/1695634868 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fb28402ff60 0x7fb284032420 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7fb2800096a0 tx=0x7fb280006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:14:36.723 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:36.722+0000 7fb2917fa700 1 -- 172.21.15.73:0/1695634868 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb288012330 con 0x7fb2940f6510 2023-12-01T22:14:36.930 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:36.929+0000 7fb29ae9f700 1 -- 172.21.15.73:0/1695634868 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb2940fbf60 con 0x7fb2940f6510 2023-12-01T22:14:36.931 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:36.929+0000 7fb2917fa700 1 -- 172.21.15.73:0/1695634868 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb28801db70 con 0x7fb2940f6510 2023-12-01T22:14:36.931 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:14:36.932 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:14:36.933 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:36.931+0000 7fb29ae9f700 1 -- 172.21.15.73:0/1695634868 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fb28402ff60 msgr2=0x7fb284032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:14:36.933 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:36.931+0000 7fb29ae9f700 1 --2- 172.21.15.73:0/1695634868 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fb28402ff60 0x7fb284032420 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7fb2800096a0 tx=0x7fb280006b40).stop 2023-12-01T22:14:36.933 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:36.931+0000 7fb29ae9f700 1 -- 172.21.15.73:0/1695634868 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2940f6510 msgr2=0x7fb29406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:14:36.933 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:36.931+0000 7fb29ae9f700 1 --2- 172.21.15.73:0/1695634868 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2940f6510 0x7fb29406b9a0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7fb288013510 tx=0x7fb28800b040).stop 2023-12-01T22:14:36.933 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:36.931+0000 7fb29ae9f700 1 -- 172.21.15.73:0/1695634868 shutdown_connections 2023-12-01T22:14:36.933 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:36.931+0000 7fb29ae9f700 1 --2- 172.21.15.73:0/1695634868 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fb28402ff60 0x7fb284032420 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:14:36.933 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:36.931+0000 7fb29ae9f700 1 --2- 172.21.15.73:0/1695634868 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2940f6510 0x7fb29406b9a0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:14:36.933 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:36.931+0000 7fb29ae9f700 1 -- 172.21.15.73:0/1695634868 >> 172.21.15.73:0/1695634868 conn(0x7fb2940f1c70 msgr2=0x7fb2940f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:14:36.934 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:36.931+0000 7fb29ae9f700 1 -- 172.21.15.73:0/1695634868 shutdown_connections 2023-12-01T22:14:36.934 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:36.931+0000 7fb29ae9f700 1 -- 172.21.15.73:0/1695634868 wait complete. 2023-12-01T22:14:36.934 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:14:38.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:14:37 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:14:36.650051+0000 mgr.smithi029.loxrfu (mgr.14164) 103 : cluster [DBG] pgmap v96: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:14:38.058 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:14:37 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:14:36.930854+0000 mon.smithi029 (mon.0) 186 : audit [DBG] from='client.? 172.21.15.73:0/1695634868' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:14:38.332 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:14:38.332 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:14:39.440 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:39.438+0000 7f607b285700 1 -- 172.21.15.73:0/1689205935 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60740f5410 msgr2=0x7f60740f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:14:39.440 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:39.438+0000 7f607b285700 1 --2- 172.21.15.73:0/1689205935 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60740f5410 0x7f60740f5830 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7f6064009fb0 tx=0x7f6064009b30).stop 2023-12-01T22:14:39.441 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:39.438+0000 7f607b285700 1 -- 172.21.15.73:0/1689205935 shutdown_connections 2023-12-01T22:14:39.441 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:39.438+0000 7f607b285700 1 --2- 172.21.15.73:0/1689205935 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60740f5410 0x7f60740f5830 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:14:39.441 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:39.438+0000 7f607b285700 1 -- 172.21.15.73:0/1689205935 >> 172.21.15.73:0/1689205935 conn(0x7f60740f0b70 msgr2=0x7f60740f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:14:39.441 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:39.438+0000 7f607b285700 1 -- 172.21.15.73:0/1689205935 shutdown_connections 2023-12-01T22:14:39.441 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:39.438+0000 7f607b285700 1 -- 172.21.15.73:0/1689205935 wait complete. 2023-12-01T22:14:39.441 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:39.439+0000 7f607b285700 1 Processor -- start 2023-12-01T22:14:39.441 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:39.439+0000 7f607b285700 1 -- start start 2023-12-01T22:14:39.441 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:39.439+0000 7f607b285700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60740f5410 0x7f6074109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:14:39.441 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:39.439+0000 7f607b285700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6074060240 con 0x7f60740f5410 2023-12-01T22:14:39.442 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:39.439+0000 7f6079021700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60740f5410 0x7f6074109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:14:39.442 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:39.439+0000 7f6079021700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60740f5410 0x7f6074109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:54584/0 (socket says 172.21.15.73:54584) 2023-12-01T22:14:39.442 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:39.439+0000 7f6079021700 1 -- 172.21.15.73:0/177205874 learned_addr learned my addr 172.21.15.73:0/177205874 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:14:39.443 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:39.439+0000 7f6079021700 1 -- 172.21.15.73:0/177205874 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6064005040 con 0x7f60740f5410 2023-12-01T22:14:39.443 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:39.440+0000 7f6079021700 1 --2- 172.21.15.73:0/177205874 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60740f5410 0x7f6074109430 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7f6064013510 tx=0x7f606400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:14:39.443 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:39.440+0000 7f6069ffb700 1 -- 172.21.15.73:0/177205874 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6064009320 con 0x7f60740f5410 2023-12-01T22:14:39.443 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:39.440+0000 7f607b285700 1 -- 172.21.15.73:0/177205874 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6074060440 con 0x7f60740f5410 2023-12-01T22:14:39.443 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:39.440+0000 7f6069ffb700 1 -- 172.21.15.73:0/177205874 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6064013f00 con 0x7f60740f5410 2023-12-01T22:14:39.443 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:39.440+0000 7f6069ffb700 1 -- 172.21.15.73:0/177205874 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6064014d90 con 0x7f60740f5410 2023-12-01T22:14:39.443 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:39.440+0000 7f607b285700 1 -- 172.21.15.73:0/177205874 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6074109ae0 con 0x7f60740f5410 2023-12-01T22:14:39.443 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:39.441+0000 7f6069ffb700 1 -- 172.21.15.73:0/177205874 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f60640095f0 con 0x7f60740f5410 2023-12-01T22:14:39.443 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:39.441+0000 7f607b285700 1 -- 172.21.15.73:0/177205874 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6074045ad0 con 0x7f60740f5410 2023-12-01T22:14:39.443 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:39.441+0000 7f6069ffb700 1 --2- 172.21.15.73:0/177205874 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f606002ff60 0x7f6060032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:14:39.443 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:39.441+0000 7f6069ffb700 1 -- 172.21.15.73:0/177205874 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6064041000 con 0x7f60740f5410 2023-12-01T22:14:39.444 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:39.441+0000 7f6078820700 1 --2- 172.21.15.73:0/177205874 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f606002ff60 0x7f6060032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:14:39.444 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:39.442+0000 7f6078820700 1 --2- 172.21.15.73:0/177205874 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f606002ff60 0x7f6060032420 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f60700096a0 tx=0x7f6070006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:14:39.447 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:39.446+0000 7f6069ffb700 1 -- 172.21.15.73:0/177205874 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f606401d7f0 con 0x7f60740f5410 2023-12-01T22:14:39.655 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:39.653+0000 7f607b285700 1 -- 172.21.15.73:0/177205874 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f60740fae60 con 0x7f60740f5410 2023-12-01T22:14:39.655 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:39.654+0000 7f6069ffb700 1 -- 172.21.15.73:0/177205874 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6064010230 con 0x7f60740f5410 2023-12-01T22:14:39.656 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:14:39.656 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:14:39.657 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:39.655+0000 7f607b285700 1 -- 172.21.15.73:0/177205874 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f606002ff60 msgr2=0x7f6060032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:14:39.657 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:39.655+0000 7f607b285700 1 --2- 172.21.15.73:0/177205874 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f606002ff60 0x7f6060032420 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f60700096a0 tx=0x7f6070006b40).stop 2023-12-01T22:14:39.657 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:39.655+0000 7f607b285700 1 -- 172.21.15.73:0/177205874 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60740f5410 msgr2=0x7f6074109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:14:39.657 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:39.655+0000 7f607b285700 1 --2- 172.21.15.73:0/177205874 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60740f5410 0x7f6074109430 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7f6064013510 tx=0x7f606400b040).stop 2023-12-01T22:14:39.657 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:39.655+0000 7f607b285700 1 -- 172.21.15.73:0/177205874 shutdown_connections 2023-12-01T22:14:39.657 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:39.655+0000 7f607b285700 1 --2- 172.21.15.73:0/177205874 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f606002ff60 0x7f6060032420 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:14:39.657 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:39.655+0000 7f607b285700 1 --2- 172.21.15.73:0/177205874 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60740f5410 0x7f6074109430 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:14:39.658 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:39.655+0000 7f607b285700 1 -- 172.21.15.73:0/177205874 >> 172.21.15.73:0/177205874 conn(0x7f60740f0b70 msgr2=0x7f60740f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:14:39.658 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:39.655+0000 7f607b285700 1 -- 172.21.15.73:0/177205874 shutdown_connections 2023-12-01T22:14:39.658 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:39.655+0000 7f607b285700 1 -- 172.21.15.73:0/177205874 wait complete. 2023-12-01T22:14:39.658 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:14:40.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:14:39 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:14:38.650334+0000 mgr.smithi029.loxrfu (mgr.14164) 104 : cluster [DBG] pgmap v97: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:14:40.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:14:39 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:14:39.655261+0000 mon.smithi029 (mon.0) 187 : audit [DBG] from='client.? 172.21.15.73:0/177205874' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:14:41.054 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:14:41.055 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:14:41.776 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:14:41 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:14:40.650722+0000 mgr.smithi029.loxrfu (mgr.14164) 105 : cluster [DBG] pgmap v98: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:14:42.194 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:42.192+0000 7f500d0d0700 1 -- 172.21.15.73:0/1921733230 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50080f5410 msgr2=0x7f50080f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:14:42.195 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:42.192+0000 7f500d0d0700 1 --2- 172.21.15.73:0/1921733230 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50080f5410 0x7f50080f5830 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7f4ffc009fb0 tx=0x7f4ffc009b30).stop 2023-12-01T22:14:42.195 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:42.192+0000 7f500d0d0700 1 -- 172.21.15.73:0/1921733230 shutdown_connections 2023-12-01T22:14:42.195 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:42.192+0000 7f500d0d0700 1 --2- 172.21.15.73:0/1921733230 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50080f5410 0x7f50080f5830 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:14:42.195 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:42.192+0000 7f500d0d0700 1 -- 172.21.15.73:0/1921733230 >> 172.21.15.73:0/1921733230 conn(0x7f50080f0b70 msgr2=0x7f50080f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:14:42.195 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:42.192+0000 7f500d0d0700 1 -- 172.21.15.73:0/1921733230 shutdown_connections 2023-12-01T22:14:42.195 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:42.192+0000 7f500d0d0700 1 -- 172.21.15.73:0/1921733230 wait complete. 2023-12-01T22:14:42.195 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:42.193+0000 7f500d0d0700 1 Processor -- start 2023-12-01T22:14:42.195 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:42.193+0000 7f500d0d0700 1 -- start start 2023-12-01T22:14:42.195 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:42.193+0000 7f500d0d0700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50080f5410 0x7f5008106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:14:42.195 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:42.193+0000 7f500d0d0700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5008106c10 con 0x7f50080f5410 2023-12-01T22:14:42.197 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:42.193+0000 7f5006d9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50080f5410 0x7f5008106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:14:42.197 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:42.193+0000 7f5006d9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50080f5410 0x7f5008106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:54614/0 (socket says 172.21.15.73:54614) 2023-12-01T22:14:42.197 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:42.193+0000 7f5006d9d700 1 -- 172.21.15.73:0/2529489997 learned_addr learned my addr 172.21.15.73:0/2529489997 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:14:42.197 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:42.193+0000 7f5006d9d700 1 -- 172.21.15.73:0/2529489997 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ffc005040 con 0x7f50080f5410 2023-12-01T22:14:42.197 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:42.194+0000 7f5006d9d700 1 --2- 172.21.15.73:0/2529489997 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50080f5410 0x7f5008106720 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7f4ffc014000 tx=0x7f4ffc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:14:42.197 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:42.194+0000 7f4fef7fe700 1 -- 172.21.15.73:0/2529489997 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4ffc0095f0 con 0x7f50080f5410 2023-12-01T22:14:42.198 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:42.194+0000 7f4fef7fe700 1 -- 172.21.15.73:0/2529489997 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4ffc007e70 con 0x7f50080f5410 2023-12-01T22:14:42.198 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:42.194+0000 7f500d0d0700 1 -- 172.21.15.73:0/2529489997 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5008106e10 con 0x7f50080f5410 2023-12-01T22:14:42.198 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:42.194+0000 7f4fef7fe700 1 -- 172.21.15.73:0/2529489997 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4ffc014ea0 con 0x7f50080f5410 2023-12-01T22:14:42.198 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:42.194+0000 7f500d0d0700 1 -- 172.21.15.73:0/2529489997 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50081072a0 con 0x7f50080f5410 2023-12-01T22:14:42.198 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:42.195+0000 7f4fef7fe700 1 -- 172.21.15.73:0/2529489997 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4ffc025070 con 0x7f50080f5410 2023-12-01T22:14:42.198 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:42.195+0000 7f500d0d0700 1 -- 172.21.15.73:0/2529489997 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5008045ad0 con 0x7f50080f5410 2023-12-01T22:14:42.198 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:42.195+0000 7f4fef7fe700 1 --2- 172.21.15.73:0/2529489997 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f4ff002ff60 0x7f4ff0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:14:42.198 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:42.195+0000 7f4fef7fe700 1 -- 172.21.15.73:0/2529489997 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4ffc040940 con 0x7f50080f5410 2023-12-01T22:14:42.198 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:42.195+0000 7f500659c700 1 --2- 172.21.15.73:0/2529489997 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f4ff002ff60 0x7f4ff0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:14:42.198 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:42.195+0000 7f500659c700 1 --2- 172.21.15.73:0/2529489997 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f4ff002ff60 0x7f4ff0032420 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f4ff80096a0 tx=0x7f4ff8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:14:42.201 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:42.200+0000 7f4fef7fe700 1 -- 172.21.15.73:0/2529489997 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4ffc0079c0 con 0x7f50080f5410 2023-12-01T22:14:42.408 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:42.406+0000 7f500d0d0700 1 -- 172.21.15.73:0/2529489997 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f50081080f0 con 0x7f50080f5410 2023-12-01T22:14:42.409 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:42.407+0000 7f4fef7fe700 1 -- 172.21.15.73:0/2529489997 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4ffc00f300 con 0x7f50080f5410 2023-12-01T22:14:42.409 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:14:42.409 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:14:42.410 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:42.408+0000 7f500d0d0700 1 -- 172.21.15.73:0/2529489997 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f4ff002ff60 msgr2=0x7f4ff0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:14:42.410 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:42.408+0000 7f500d0d0700 1 --2- 172.21.15.73:0/2529489997 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f4ff002ff60 0x7f4ff0032420 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f4ff80096a0 tx=0x7f4ff8006b40).stop 2023-12-01T22:14:42.411 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:42.408+0000 7f500d0d0700 1 -- 172.21.15.73:0/2529489997 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50080f5410 msgr2=0x7f5008106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:14:42.411 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:42.408+0000 7f500d0d0700 1 --2- 172.21.15.73:0/2529489997 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50080f5410 0x7f5008106720 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7f4ffc014000 tx=0x7f4ffc00b040).stop 2023-12-01T22:14:42.411 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:42.409+0000 7f500d0d0700 1 -- 172.21.15.73:0/2529489997 shutdown_connections 2023-12-01T22:14:42.411 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:42.409+0000 7f500d0d0700 1 --2- 172.21.15.73:0/2529489997 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f4ff002ff60 0x7f4ff0032420 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:14:42.411 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:42.409+0000 7f500d0d0700 1 --2- 172.21.15.73:0/2529489997 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50080f5410 0x7f5008106720 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:14:42.411 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:42.409+0000 7f500d0d0700 1 -- 172.21.15.73:0/2529489997 >> 172.21.15.73:0/2529489997 conn(0x7f50080f0b70 msgr2=0x7f500805fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:14:42.411 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:42.409+0000 7f500d0d0700 1 -- 172.21.15.73:0/2529489997 shutdown_connections 2023-12-01T22:14:42.411 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:42.409+0000 7f500d0d0700 1 -- 172.21.15.73:0/2529489997 wait complete. 2023-12-01T22:14:42.411 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:14:42.776 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:14:42 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:14:42.408369+0000 mon.smithi029 (mon.0) 188 : audit [DBG] from='client.? 172.21.15.73:0/2529489997' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:14:43.776 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:14:43 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:14:42.651128+0000 mgr.smithi029.loxrfu (mgr.14164) 106 : cluster [DBG] pgmap v99: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:14:43.819 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:14:43.819 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:14:44.935 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:44.933+0000 7f1c29627700 1 -- 172.21.15.73:0/2605529072 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c240f59d0 msgr2=0x7f1c240f5df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:14:44.935 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:44.933+0000 7f1c29627700 1 --2- 172.21.15.73:0/2605529072 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c240f59d0 0x7f1c240f5df0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7f1c14009fb0 tx=0x7f1c14009b30).stop 2023-12-01T22:14:44.936 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:44.933+0000 7f1c29627700 1 -- 172.21.15.73:0/2605529072 shutdown_connections 2023-12-01T22:14:44.936 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:44.933+0000 7f1c29627700 1 --2- 172.21.15.73:0/2605529072 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c240f59d0 0x7f1c240f5df0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:14:44.936 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:44.933+0000 7f1c29627700 1 -- 172.21.15.73:0/2605529072 >> 172.21.15.73:0/2605529072 conn(0x7f1c240f02e0 msgr2=0x7f1c240f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:14:44.936 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:44.933+0000 7f1c29627700 1 -- 172.21.15.73:0/2605529072 shutdown_connections 2023-12-01T22:14:44.936 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:44.934+0000 7f1c29627700 1 -- 172.21.15.73:0/2605529072 wait complete. 2023-12-01T22:14:44.937 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:44.934+0000 7f1c29627700 1 Processor -- start 2023-12-01T22:14:44.937 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:44.934+0000 7f1c29627700 1 -- start start 2023-12-01T22:14:44.937 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:44.934+0000 7f1c29627700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c240f59d0 0x7f1c241066c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:14:44.937 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:44.934+0000 7f1c29627700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c24106bb0 con 0x7f1c240f59d0 2023-12-01T22:14:44.938 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:44.934+0000 7f1c22ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c240f59d0 0x7f1c241066c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:14:44.938 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:44.935+0000 7f1c22ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c240f59d0 0x7f1c241066c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:58840/0 (socket says 172.21.15.73:58840) 2023-12-01T22:14:44.938 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:44.935+0000 7f1c22ffd700 1 -- 172.21.15.73:0/3586877439 learned_addr learned my addr 172.21.15.73:0/3586877439 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:14:44.938 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:44.935+0000 7f1c22ffd700 1 -- 172.21.15.73:0/3586877439 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c14005040 con 0x7f1c240f59d0 2023-12-01T22:14:44.938 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:44.935+0000 7f1c22ffd700 1 --2- 172.21.15.73:0/3586877439 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c240f59d0 0x7f1c241066c0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7f1c14013510 tx=0x7f1c1400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:14:44.938 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:44.935+0000 7f1c0b7fe700 1 -- 172.21.15.73:0/3586877439 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1c14009320 con 0x7f1c240f59d0 2023-12-01T22:14:44.938 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:44.935+0000 7f1c29627700 1 -- 172.21.15.73:0/3586877439 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c24106db0 con 0x7f1c240f59d0 2023-12-01T22:14:44.938 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:44.935+0000 7f1c0b7fe700 1 -- 172.21.15.73:0/3586877439 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1c14013f00 con 0x7f1c240f59d0 2023-12-01T22:14:44.938 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:44.935+0000 7f1c0b7fe700 1 -- 172.21.15.73:0/3586877439 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1c14014d90 con 0x7f1c240f59d0 2023-12-01T22:14:44.938 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:44.935+0000 7f1c29627700 1 -- 172.21.15.73:0/3586877439 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c24107240 con 0x7f1c240f59d0 2023-12-01T22:14:44.939 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:44.936+0000 7f1c0b7fe700 1 -- 172.21.15.73:0/3586877439 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1c14014420 con 0x7f1c240f59d0 2023-12-01T22:14:44.939 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:44.936+0000 7f1c29627700 1 -- 172.21.15.73:0/3586877439 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c24045ad0 con 0x7f1c240f59d0 2023-12-01T22:14:44.939 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:44.936+0000 7f1c0b7fe700 1 --2- 172.21.15.73:0/3586877439 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f1c0c02ff60 0x7f1c0c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:14:44.939 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:44.936+0000 7f1c0b7fe700 1 -- 172.21.15.73:0/3586877439 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1c14041000 con 0x7f1c240f59d0 2023-12-01T22:14:44.939 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:44.937+0000 7f1c227fc700 1 --2- 172.21.15.73:0/3586877439 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f1c0c02ff60 0x7f1c0c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:14:44.940 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:44.937+0000 7f1c227fc700 1 --2- 172.21.15.73:0/3586877439 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f1c0c02ff60 0x7f1c0c032420 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f1c180096a0 tx=0x7f1c18006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:14:44.943 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:44.941+0000 7f1c0b7fe700 1 -- 172.21.15.73:0/3586877439 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1c140095f0 con 0x7f1c240f59d0 2023-12-01T22:14:45.149 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:45.148+0000 7f1c29627700 1 -- 172.21.15.73:0/3586877439 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1c24108090 con 0x7f1c240f59d0 2023-12-01T22:14:45.150 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:45.148+0000 7f1c0b7fe700 1 -- 172.21.15.73:0/3586877439 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1c1401db70 con 0x7f1c240f59d0 2023-12-01T22:14:45.150 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:14:45.150 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:14:45.152 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:45.150+0000 7f1c29627700 1 -- 172.21.15.73:0/3586877439 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f1c0c02ff60 msgr2=0x7f1c0c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:14:45.152 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:45.150+0000 7f1c29627700 1 --2- 172.21.15.73:0/3586877439 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f1c0c02ff60 0x7f1c0c032420 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f1c180096a0 tx=0x7f1c18006b40).stop 2023-12-01T22:14:45.152 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:45.150+0000 7f1c29627700 1 -- 172.21.15.73:0/3586877439 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c240f59d0 msgr2=0x7f1c241066c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:14:45.152 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:45.150+0000 7f1c29627700 1 --2- 172.21.15.73:0/3586877439 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c240f59d0 0x7f1c241066c0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7f1c14013510 tx=0x7f1c1400b040).stop 2023-12-01T22:14:45.152 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:45.150+0000 7f1c29627700 1 -- 172.21.15.73:0/3586877439 shutdown_connections 2023-12-01T22:14:45.152 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:45.150+0000 7f1c29627700 1 --2- 172.21.15.73:0/3586877439 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f1c0c02ff60 0x7f1c0c032420 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:14:45.152 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:45.150+0000 7f1c29627700 1 --2- 172.21.15.73:0/3586877439 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c240f59d0 0x7f1c241066c0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:14:45.152 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:45.150+0000 7f1c29627700 1 -- 172.21.15.73:0/3586877439 >> 172.21.15.73:0/3586877439 conn(0x7f1c240f02e0 msgr2=0x7f1c240f4f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:14:45.152 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:45.150+0000 7f1c29627700 1 -- 172.21.15.73:0/3586877439 shutdown_connections 2023-12-01T22:14:45.152 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:45.150+0000 7f1c29627700 1 -- 172.21.15.73:0/3586877439 wait complete. 2023-12-01T22:14:45.153 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:14:46.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:14:45 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:14:44.651439+0000 mgr.smithi029.loxrfu (mgr.14164) 107 : cluster [DBG] pgmap v100: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:14:46.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:14:45 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:14:45.149647+0000 mon.smithi029 (mon.0) 189 : audit [DBG] from='client.? 172.21.15.73:0/3586877439' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:14:46.526 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:14:46.526 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:14:47.711 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:47.705+0000 7f7e4c69c700 1 -- 172.21.15.73:0/3966848453 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e440f5410 msgr2=0x7f7e440f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:14:47.711 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:47.705+0000 7f7e4c69c700 1 --2- 172.21.15.73:0/3966848453 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e440f5410 0x7f7e440f5830 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7f7e38009fb0 tx=0x7f7e38009b30).stop 2023-12-01T22:14:47.712 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:47.705+0000 7f7e4c69c700 1 -- 172.21.15.73:0/3966848453 shutdown_connections 2023-12-01T22:14:47.712 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:47.705+0000 7f7e4c69c700 1 --2- 172.21.15.73:0/3966848453 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e440f5410 0x7f7e440f5830 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:14:47.712 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:47.705+0000 7f7e4c69c700 1 -- 172.21.15.73:0/3966848453 >> 172.21.15.73:0/3966848453 conn(0x7f7e440f0b70 msgr2=0x7f7e440f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:14:47.712 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:47.706+0000 7f7e4c69c700 1 -- 172.21.15.73:0/3966848453 shutdown_connections 2023-12-01T22:14:47.712 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:47.706+0000 7f7e4c69c700 1 -- 172.21.15.73:0/3966848453 wait complete. 2023-12-01T22:14:47.712 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:47.706+0000 7f7e4c69c700 1 Processor -- start 2023-12-01T22:14:47.712 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:47.706+0000 7f7e4c69c700 1 -- start start 2023-12-01T22:14:47.713 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:47.706+0000 7f7e4c69c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e440f5410 0x7f7e44109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:14:47.713 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:47.706+0000 7f7e4c69c700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e44060240 con 0x7f7e440f5410 2023-12-01T22:14:47.713 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:47.706+0000 7f7e4a438700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e440f5410 0x7f7e44109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:14:47.713 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:47.706+0000 7f7e4a438700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e440f5410 0x7f7e44109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:58856/0 (socket says 172.21.15.73:58856) 2023-12-01T22:14:47.713 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:47.706+0000 7f7e4a438700 1 -- 172.21.15.73:0/870832183 learned_addr learned my addr 172.21.15.73:0/870832183 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:14:47.713 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:47.707+0000 7f7e4a438700 1 -- 172.21.15.73:0/870832183 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e38005040 con 0x7f7e440f5410 2023-12-01T22:14:47.714 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:47.707+0000 7f7e4a438700 1 --2- 172.21.15.73:0/870832183 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e440f5410 0x7f7e44109430 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7f7e38013510 tx=0x7f7e3800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:14:47.714 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:47.707+0000 7f7e36ffd700 1 -- 172.21.15.73:0/870832183 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7e38009320 con 0x7f7e440f5410 2023-12-01T22:14:47.714 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:47.707+0000 7f7e36ffd700 1 -- 172.21.15.73:0/870832183 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7e38013f00 con 0x7f7e440f5410 2023-12-01T22:14:47.714 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:47.707+0000 7f7e36ffd700 1 -- 172.21.15.73:0/870832183 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7e38014d90 con 0x7f7e440f5410 2023-12-01T22:14:47.714 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:47.707+0000 7f7e4c69c700 1 -- 172.21.15.73:0/870832183 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e44060440 con 0x7f7e440f5410 2023-12-01T22:14:47.714 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:47.707+0000 7f7e4c69c700 1 -- 172.21.15.73:0/870832183 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e44109ae0 con 0x7f7e440f5410 2023-12-01T22:14:47.715 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:47.708+0000 7f7e36ffd700 1 -- 172.21.15.73:0/870832183 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7e380095f0 con 0x7f7e440f5410 2023-12-01T22:14:47.715 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:47.708+0000 7f7e4c69c700 1 -- 172.21.15.73:0/870832183 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e44045ad0 con 0x7f7e440f5410 2023-12-01T22:14:47.715 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:47.708+0000 7f7e36ffd700 1 --2- 172.21.15.73:0/870832183 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f7e3002ff60 0x7f7e30032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:14:47.715 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:47.708+0000 7f7e36ffd700 1 -- 172.21.15.73:0/870832183 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7e380410c0 con 0x7f7e440f5410 2023-12-01T22:14:47.715 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:47.708+0000 7f7e49c37700 1 --2- 172.21.15.73:0/870832183 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f7e3002ff60 0x7f7e30032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:14:47.715 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:47.709+0000 7f7e49c37700 1 --2- 172.21.15.73:0/870832183 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f7e3002ff60 0x7f7e30032420 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f7e400096a0 tx=0x7f7e40006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:14:47.716 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:47.713+0000 7f7e36ffd700 1 -- 172.21.15.73:0/870832183 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7e3801d7f0 con 0x7f7e440f5410 2023-12-01T22:14:47.921 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:47.918+0000 7f7e4c69c700 1 -- 172.21.15.73:0/870832183 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7e440fae60 con 0x7f7e440f5410 2023-12-01T22:14:47.921 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:47.919+0000 7f7e36ffd700 1 -- 172.21.15.73:0/870832183 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7e38010230 con 0x7f7e440f5410 2023-12-01T22:14:47.921 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:14:47.922 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:14:47.923 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:47.920+0000 7f7e4c69c700 1 -- 172.21.15.73:0/870832183 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f7e3002ff60 msgr2=0x7f7e30032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:14:47.923 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:47.920+0000 7f7e4c69c700 1 --2- 172.21.15.73:0/870832183 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f7e3002ff60 0x7f7e30032420 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f7e400096a0 tx=0x7f7e40006b40).stop 2023-12-01T22:14:47.923 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:47.920+0000 7f7e4c69c700 1 -- 172.21.15.73:0/870832183 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e440f5410 msgr2=0x7f7e44109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:14:47.923 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:47.920+0000 7f7e4c69c700 1 --2- 172.21.15.73:0/870832183 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e440f5410 0x7f7e44109430 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7f7e38013510 tx=0x7f7e3800b040).stop 2023-12-01T22:14:47.924 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:47.920+0000 7f7e4c69c700 1 -- 172.21.15.73:0/870832183 shutdown_connections 2023-12-01T22:14:47.924 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:47.920+0000 7f7e4c69c700 1 --2- 172.21.15.73:0/870832183 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f7e3002ff60 0x7f7e30032420 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:14:47.924 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:47.920+0000 7f7e4c69c700 1 --2- 172.21.15.73:0/870832183 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e440f5410 0x7f7e44109430 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:14:47.924 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:47.920+0000 7f7e4c69c700 1 -- 172.21.15.73:0/870832183 >> 172.21.15.73:0/870832183 conn(0x7f7e440f0b70 msgr2=0x7f7e440f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:14:47.924 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:47.920+0000 7f7e4c69c700 1 -- 172.21.15.73:0/870832183 shutdown_connections 2023-12-01T22:14:47.924 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:47.921+0000 7f7e4c69c700 1 -- 172.21.15.73:0/870832183 wait complete. 2023-12-01T22:14:47.925 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:14:48.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:14:47 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:14:46.651786+0000 mgr.smithi029.loxrfu (mgr.14164) 108 : cluster [DBG] pgmap v101: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:14:49.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:14:48 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:14:47.920120+0000 mon.smithi029 (mon.0) 190 : audit [DBG] from='client.? 172.21.15.73:0/870832183' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:14:49.308 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:14:49.308 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:14:50.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:14:49 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:14:48.652152+0000 mgr.smithi029.loxrfu (mgr.14164) 109 : cluster [DBG] pgmap v102: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:14:50.484 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:50.482+0000 7fd41a4d3700 1 -- 172.21.15.73:0/3128569494 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4140f6510 msgr2=0x7fd4140f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:14:50.485 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:50.482+0000 7fd41a4d3700 1 --2- 172.21.15.73:0/3128569494 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4140f6510 0x7fd4140f6930 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7fd408009fb0 tx=0x7fd408009b30).stop 2023-12-01T22:14:50.485 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:50.482+0000 7fd41a4d3700 1 -- 172.21.15.73:0/3128569494 shutdown_connections 2023-12-01T22:14:50.485 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:50.482+0000 7fd41a4d3700 1 --2- 172.21.15.73:0/3128569494 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4140f6510 0x7fd4140f6930 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:14:50.485 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:50.482+0000 7fd41a4d3700 1 -- 172.21.15.73:0/3128569494 >> 172.21.15.73:0/3128569494 conn(0x7fd4140f1c70 msgr2=0x7fd4140f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:14:50.485 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:50.482+0000 7fd41a4d3700 1 -- 172.21.15.73:0/3128569494 shutdown_connections 2023-12-01T22:14:50.485 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:50.482+0000 7fd41a4d3700 1 -- 172.21.15.73:0/3128569494 wait complete. 2023-12-01T22:14:50.485 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:50.482+0000 7fd41a4d3700 1 Processor -- start 2023-12-01T22:14:50.485 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:50.482+0000 7fd41a4d3700 1 -- start start 2023-12-01T22:14:50.485 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:50.483+0000 7fd41a4d3700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4140f6510 0x7fd414106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:14:50.485 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:50.483+0000 7fd41a4d3700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd414106c20 con 0x7fd4140f6510 2023-12-01T22:14:50.485 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:50.483+0000 7fd413fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4140f6510 0x7fd414106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:14:50.485 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:50.483+0000 7fd413fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4140f6510 0x7fd414106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:58882/0 (socket says 172.21.15.73:58882) 2023-12-01T22:14:50.485 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:50.483+0000 7fd413fff700 1 -- 172.21.15.73:0/406609370 learned_addr learned my addr 172.21.15.73:0/406609370 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:14:50.486 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:50.483+0000 7fd413fff700 1 -- 172.21.15.73:0/406609370 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd408005040 con 0x7fd4140f6510 2023-12-01T22:14:50.487 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:50.483+0000 7fd413fff700 1 --2- 172.21.15.73:0/406609370 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4140f6510 0x7fd414106730 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7fd408013510 tx=0x7fd40800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:14:50.487 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:50.483+0000 7fd410ff9700 1 -- 172.21.15.73:0/406609370 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd408009320 con 0x7fd4140f6510 2023-12-01T22:14:50.487 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:50.484+0000 7fd410ff9700 1 -- 172.21.15.73:0/406609370 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd408013f00 con 0x7fd4140f6510 2023-12-01T22:14:50.487 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:50.484+0000 7fd41a4d3700 1 -- 172.21.15.73:0/406609370 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd414106e20 con 0x7fd4140f6510 2023-12-01T22:14:50.487 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:50.484+0000 7fd410ff9700 1 -- 172.21.15.73:0/406609370 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd408014d90 con 0x7fd4140f6510 2023-12-01T22:14:50.487 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:50.484+0000 7fd41a4d3700 1 -- 172.21.15.73:0/406609370 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4141072b0 con 0x7fd4140f6510 2023-12-01T22:14:50.487 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:50.484+0000 7fd410ff9700 1 -- 172.21.15.73:0/406609370 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd408014420 con 0x7fd4140f6510 2023-12-01T22:14:50.487 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:50.484+0000 7fd410ff9700 1 --2- 172.21.15.73:0/406609370 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fd3fc02ff60 0x7fd3fc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:14:50.488 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:50.484+0000 7fd41a4d3700 1 -- 172.21.15.73:0/406609370 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd414045ad0 con 0x7fd4140f6510 2023-12-01T22:14:50.488 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:50.485+0000 7fd410ff9700 1 -- 172.21.15.73:0/406609370 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd408041000 con 0x7fd4140f6510 2023-12-01T22:14:50.488 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:50.485+0000 7fd4137fe700 1 --2- 172.21.15.73:0/406609370 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fd3fc02ff60 0x7fd3fc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:14:50.488 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:50.485+0000 7fd4137fe700 1 --2- 172.21.15.73:0/406609370 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fd3fc02ff60 0x7fd3fc032420 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7fd4040096a0 tx=0x7fd404006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:14:50.491 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:50.489+0000 7fd410ff9700 1 -- 172.21.15.73:0/406609370 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd40806d2b0 con 0x7fd4140f6510 2023-12-01T22:14:50.698 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:50.695+0000 7fd41a4d3700 1 -- 172.21.15.73:0/406609370 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd4140fc050 con 0x7fd4140f6510 2023-12-01T22:14:50.698 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:50.696+0000 7fd410ff9700 1 -- 172.21.15.73:0/406609370 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd40801dd50 con 0x7fd4140f6510 2023-12-01T22:14:50.698 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:14:50.698 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:14:50.699 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:50.697+0000 7fd41a4d3700 1 -- 172.21.15.73:0/406609370 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fd3fc02ff60 msgr2=0x7fd3fc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:14:50.699 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:50.697+0000 7fd41a4d3700 1 --2- 172.21.15.73:0/406609370 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fd3fc02ff60 0x7fd3fc032420 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7fd4040096a0 tx=0x7fd404006b40).stop 2023-12-01T22:14:50.699 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:50.697+0000 7fd41a4d3700 1 -- 172.21.15.73:0/406609370 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4140f6510 msgr2=0x7fd414106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:14:50.700 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:50.697+0000 7fd41a4d3700 1 --2- 172.21.15.73:0/406609370 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4140f6510 0x7fd414106730 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7fd408013510 tx=0x7fd40800b040).stop 2023-12-01T22:14:50.700 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:50.697+0000 7fd41a4d3700 1 -- 172.21.15.73:0/406609370 shutdown_connections 2023-12-01T22:14:50.700 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:50.697+0000 7fd41a4d3700 1 --2- 172.21.15.73:0/406609370 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fd3fc02ff60 0x7fd3fc032420 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:14:50.700 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:50.697+0000 7fd41a4d3700 1 --2- 172.21.15.73:0/406609370 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4140f6510 0x7fd414106730 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:14:50.700 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:50.697+0000 7fd41a4d3700 1 -- 172.21.15.73:0/406609370 >> 172.21.15.73:0/406609370 conn(0x7fd4140f1c70 msgr2=0x7fd4140f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:14:50.700 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:50.698+0000 7fd41a4d3700 1 -- 172.21.15.73:0/406609370 shutdown_connections 2023-12-01T22:14:50.700 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:50.698+0000 7fd41a4d3700 1 -- 172.21.15.73:0/406609370 wait complete. 2023-12-01T22:14:50.700 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:14:51.776 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:14:51 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:14:50.652473+0000 mgr.smithi029.loxrfu (mgr.14164) 110 : cluster [DBG] pgmap v103: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:14:51.776 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:14:51 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:14:50.697327+0000 mon.smithi029 (mon.0) 191 : audit [DBG] from='client.? 172.21.15.73:0/406609370' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:14:52.071 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:14:52.071 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:14:53.209 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:53.206+0000 7f1b28878700 1 -- 172.21.15.73:0/3115983755 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b240f5410 msgr2=0x7f1b240f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:14:53.210 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:53.206+0000 7f1b28878700 1 --2- 172.21.15.73:0/3115983755 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b240f5410 0x7f1b240f5830 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7f1b18009fb0 tx=0x7f1b18009b30).stop 2023-12-01T22:14:53.210 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:53.206+0000 7f1b28878700 1 -- 172.21.15.73:0/3115983755 shutdown_connections 2023-12-01T22:14:53.210 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:53.206+0000 7f1b28878700 1 --2- 172.21.15.73:0/3115983755 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b240f5410 0x7f1b240f5830 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:14:53.210 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:53.206+0000 7f1b28878700 1 -- 172.21.15.73:0/3115983755 >> 172.21.15.73:0/3115983755 conn(0x7f1b240f0b70 msgr2=0x7f1b240f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:14:53.210 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:53.206+0000 7f1b28878700 1 -- 172.21.15.73:0/3115983755 shutdown_connections 2023-12-01T22:14:53.210 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:53.206+0000 7f1b28878700 1 -- 172.21.15.73:0/3115983755 wait complete. 2023-12-01T22:14:53.210 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:53.207+0000 7f1b28878700 1 Processor -- start 2023-12-01T22:14:53.210 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:53.207+0000 7f1b28878700 1 -- start start 2023-12-01T22:14:53.210 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:53.207+0000 7f1b28878700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b240f5410 0x7f1b24109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:14:53.210 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:53.207+0000 7f1b28878700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b24060240 con 0x7f1b240f5410 2023-12-01T22:14:53.210 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:53.207+0000 7f1b2259c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b240f5410 0x7f1b24109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:14:53.211 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:53.207+0000 7f1b2259c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b240f5410 0x7f1b24109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:58900/0 (socket says 172.21.15.73:58900) 2023-12-01T22:14:53.211 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:53.207+0000 7f1b2259c700 1 -- 172.21.15.73:0/1851189378 learned_addr learned my addr 172.21.15.73:0/1851189378 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:14:53.211 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:53.207+0000 7f1b2259c700 1 -- 172.21.15.73:0/1851189378 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b18005040 con 0x7f1b240f5410 2023-12-01T22:14:53.211 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:53.207+0000 7f1b2259c700 1 --2- 172.21.15.73:0/1851189378 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b240f5410 0x7f1b24109430 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7f1b18013510 tx=0x7f1b1800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:14:53.211 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:53.208+0000 7f1b12ffd700 1 -- 172.21.15.73:0/1851189378 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1b18009320 con 0x7f1b240f5410 2023-12-01T22:14:53.211 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:53.208+0000 7f1b28878700 1 -- 172.21.15.73:0/1851189378 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b24060440 con 0x7f1b240f5410 2023-12-01T22:14:53.211 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:53.208+0000 7f1b12ffd700 1 -- 172.21.15.73:0/1851189378 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1b18013f00 con 0x7f1b240f5410 2023-12-01T22:14:53.211 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:53.208+0000 7f1b12ffd700 1 -- 172.21.15.73:0/1851189378 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1b18014d90 con 0x7f1b240f5410 2023-12-01T22:14:53.211 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:53.208+0000 7f1b28878700 1 -- 172.21.15.73:0/1851189378 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b24109ae0 con 0x7f1b240f5410 2023-12-01T22:14:53.212 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:53.209+0000 7f1b12ffd700 1 -- 172.21.15.73:0/1851189378 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1b18014420 con 0x7f1b240f5410 2023-12-01T22:14:53.212 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:53.209+0000 7f1b28878700 1 -- 172.21.15.73:0/1851189378 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b24045ad0 con 0x7f1b240f5410 2023-12-01T22:14:53.212 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:53.209+0000 7f1b12ffd700 1 --2- 172.21.15.73:0/1851189378 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f1b0c02ff60 0x7f1b0c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:14:53.212 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:53.209+0000 7f1b12ffd700 1 -- 172.21.15.73:0/1851189378 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1b18040fb0 con 0x7f1b240f5410 2023-12-01T22:14:53.212 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:53.209+0000 7f1b21d9b700 1 --2- 172.21.15.73:0/1851189378 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f1b0c02ff60 0x7f1b0c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:14:53.212 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:53.210+0000 7f1b21d9b700 1 --2- 172.21.15.73:0/1851189378 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f1b0c02ff60 0x7f1b0c032420 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f1b140096a0 tx=0x7f1b14006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:14:53.215 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:53.214+0000 7f1b12ffd700 1 -- 172.21.15.73:0/1851189378 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1b180095f0 con 0x7f1b240f5410 2023-12-01T22:14:53.424 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:53.422+0000 7f1b28878700 1 -- 172.21.15.73:0/1851189378 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1b240fae60 con 0x7f1b240f5410 2023-12-01T22:14:53.424 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:53.423+0000 7f1b12ffd700 1 -- 172.21.15.73:0/1851189378 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1b1801db70 con 0x7f1b240f5410 2023-12-01T22:14:53.425 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:14:53.425 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:14:53.426 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:53.424+0000 7f1b28878700 1 -- 172.21.15.73:0/1851189378 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f1b0c02ff60 msgr2=0x7f1b0c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:14:53.426 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:53.424+0000 7f1b28878700 1 --2- 172.21.15.73:0/1851189378 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f1b0c02ff60 0x7f1b0c032420 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f1b140096a0 tx=0x7f1b14006b40).stop 2023-12-01T22:14:53.426 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:53.424+0000 7f1b28878700 1 -- 172.21.15.73:0/1851189378 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b240f5410 msgr2=0x7f1b24109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:14:53.426 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:53.424+0000 7f1b28878700 1 --2- 172.21.15.73:0/1851189378 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b240f5410 0x7f1b24109430 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7f1b18013510 tx=0x7f1b1800b040).stop 2023-12-01T22:14:53.426 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:53.424+0000 7f1b28878700 1 -- 172.21.15.73:0/1851189378 shutdown_connections 2023-12-01T22:14:53.426 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:53.424+0000 7f1b28878700 1 --2- 172.21.15.73:0/1851189378 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f1b0c02ff60 0x7f1b0c032420 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:14:53.427 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:53.424+0000 7f1b28878700 1 --2- 172.21.15.73:0/1851189378 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b240f5410 0x7f1b24109430 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:14:53.427 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:53.424+0000 7f1b28878700 1 -- 172.21.15.73:0/1851189378 >> 172.21.15.73:0/1851189378 conn(0x7f1b240f0b70 msgr2=0x7f1b240f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:14:53.427 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:53.425+0000 7f1b28878700 1 -- 172.21.15.73:0/1851189378 shutdown_connections 2023-12-01T22:14:53.427 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:53.425+0000 7f1b28878700 1 -- 172.21.15.73:0/1851189378 wait complete. 2023-12-01T22:14:53.427 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:14:54.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:14:53 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:14:52.652721+0000 mgr.smithi029.loxrfu (mgr.14164) 111 : cluster [DBG] pgmap v104: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:14:54.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:14:53 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:14:53.424132+0000 mon.smithi029 (mon.0) 192 : audit [DBG] from='client.? 172.21.15.73:0/1851189378' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:14:54.822 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:14:54.822 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:14:55.976 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:55.973+0000 7fc12650b700 1 -- 172.21.15.73:0/1653545578 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc1200f5410 msgr2=0x7fc1200f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:14:55.976 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:55.973+0000 7fc12650b700 1 --2- 172.21.15.73:0/1653545578 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc1200f5410 0x7fc1200f5830 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7fc110009fb0 tx=0x7fc110009b30).stop 2023-12-01T22:14:55.976 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:55.973+0000 7fc12650b700 1 -- 172.21.15.73:0/1653545578 shutdown_connections 2023-12-01T22:14:55.977 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:55.973+0000 7fc12650b700 1 --2- 172.21.15.73:0/1653545578 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc1200f5410 0x7fc1200f5830 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:14:55.977 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:55.973+0000 7fc12650b700 1 -- 172.21.15.73:0/1653545578 >> 172.21.15.73:0/1653545578 conn(0x7fc1200f0b70 msgr2=0x7fc1200f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:14:55.977 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:55.973+0000 7fc12650b700 1 -- 172.21.15.73:0/1653545578 shutdown_connections 2023-12-01T22:14:55.977 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:55.973+0000 7fc12650b700 1 -- 172.21.15.73:0/1653545578 wait complete. 2023-12-01T22:14:55.977 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:55.973+0000 7fc12650b700 1 Processor -- start 2023-12-01T22:14:55.977 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:55.974+0000 7fc12650b700 1 -- start start 2023-12-01T22:14:55.977 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:55.974+0000 7fc12650b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc1200f5410 0x7fc12010b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:14:55.977 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:55.974+0000 7fc12650b700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc120107240 con 0x7fc1200f5410 2023-12-01T22:14:55.977 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:55.974+0000 7fc11ffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc1200f5410 0x7fc12010b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:14:55.977 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:55.974+0000 7fc11ffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc1200f5410 0x7fc12010b850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:40922/0 (socket says 172.21.15.73:40922) 2023-12-01T22:14:55.977 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:55.974+0000 7fc11ffff700 1 -- 172.21.15.73:0/989204219 learned_addr learned my addr 172.21.15.73:0/989204219 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:14:55.977 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:55.974+0000 7fc11ffff700 1 -- 172.21.15.73:0/989204219 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc110005040 con 0x7fc1200f5410 2023-12-01T22:14:55.977 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:55.974+0000 7fc11ffff700 1 --2- 172.21.15.73:0/989204219 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc1200f5410 0x7fc12010b850 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7fc110013510 tx=0x7fc11000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:14:55.977 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:55.974+0000 7fc11d7fa700 1 -- 172.21.15.73:0/989204219 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc110009320 con 0x7fc1200f5410 2023-12-01T22:14:55.978 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:55.975+0000 7fc11d7fa700 1 -- 172.21.15.73:0/989204219 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc110013f00 con 0x7fc1200f5410 2023-12-01T22:14:55.978 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:55.975+0000 7fc12650b700 1 -- 172.21.15.73:0/989204219 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc120060220 con 0x7fc1200f5410 2023-12-01T22:14:55.978 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:55.975+0000 7fc11d7fa700 1 -- 172.21.15.73:0/989204219 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc110014d90 con 0x7fc1200f5410 2023-12-01T22:14:55.978 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:55.975+0000 7fc12650b700 1 -- 172.21.15.73:0/989204219 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc12010bd40 con 0x7fc1200f5410 2023-12-01T22:14:55.979 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:55.975+0000 7fc11d7fa700 1 -- 172.21.15.73:0/989204219 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc1100095f0 con 0x7fc1200f5410 2023-12-01T22:14:55.979 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:55.975+0000 7fc12650b700 1 -- 172.21.15.73:0/989204219 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc120045ad0 con 0x7fc1200f5410 2023-12-01T22:14:55.979 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:55.976+0000 7fc11d7fa700 1 --2- 172.21.15.73:0/989204219 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fc10c02ffb0 0x7fc10c032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:14:55.979 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:55.976+0000 7fc11d7fa700 1 -- 172.21.15.73:0/989204219 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc110040f80 con 0x7fc1200f5410 2023-12-01T22:14:55.979 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:55.976+0000 7fc117fff700 1 --2- 172.21.15.73:0/989204219 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fc10c02ffb0 0x7fc10c032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:14:55.979 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:55.976+0000 7fc117fff700 1 --2- 172.21.15.73:0/989204219 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fc10c02ffb0 0x7fc10c032470 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7fc1080096a0 tx=0x7fc108006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:14:55.982 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:55.981+0000 7fc11d7fa700 1 -- 172.21.15.73:0/989204219 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc110014420 con 0x7fc1200f5410 2023-12-01T22:14:56.029 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:14:55 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:14:54.653046+0000 mgr.smithi029.loxrfu (mgr.14164) 112 : cluster [DBG] pgmap v105: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:14:56.188 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:56.186+0000 7fc12650b700 1 -- 172.21.15.73:0/989204219 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc1200fae60 con 0x7fc1200f5410 2023-12-01T22:14:56.189 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:56.187+0000 7fc11d7fa700 1 -- 172.21.15.73:0/989204219 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc11001d7f0 con 0x7fc1200f5410 2023-12-01T22:14:56.189 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:14:56.189 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:14:56.190 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:56.188+0000 7fc12650b700 1 -- 172.21.15.73:0/989204219 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fc10c02ffb0 msgr2=0x7fc10c032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:14:56.190 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:56.188+0000 7fc12650b700 1 --2- 172.21.15.73:0/989204219 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fc10c02ffb0 0x7fc10c032470 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7fc1080096a0 tx=0x7fc108006b40).stop 2023-12-01T22:14:56.190 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:56.188+0000 7fc12650b700 1 -- 172.21.15.73:0/989204219 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc1200f5410 msgr2=0x7fc12010b850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:14:56.190 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:56.188+0000 7fc12650b700 1 --2- 172.21.15.73:0/989204219 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc1200f5410 0x7fc12010b850 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7fc110013510 tx=0x7fc11000b040).stop 2023-12-01T22:14:56.191 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:56.189+0000 7fc12650b700 1 -- 172.21.15.73:0/989204219 shutdown_connections 2023-12-01T22:14:56.191 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:56.189+0000 7fc12650b700 1 --2- 172.21.15.73:0/989204219 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fc10c02ffb0 0x7fc10c032470 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:14:56.191 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:56.189+0000 7fc12650b700 1 --2- 172.21.15.73:0/989204219 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc1200f5410 0x7fc12010b850 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:14:56.191 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:56.189+0000 7fc12650b700 1 -- 172.21.15.73:0/989204219 >> 172.21.15.73:0/989204219 conn(0x7fc1200f0b70 msgr2=0x7fc1200f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:14:56.191 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:56.189+0000 7fc12650b700 1 -- 172.21.15.73:0/989204219 shutdown_connections 2023-12-01T22:14:56.191 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:56.189+0000 7fc12650b700 1 -- 172.21.15.73:0/989204219 wait complete. 2023-12-01T22:14:56.191 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:14:57.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:14:56 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:14:56.188381+0000 mon.smithi029 (mon.0) 193 : audit [DBG] from='client.? 172.21.15.73:0/989204219' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:14:57.573 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:14:57.573 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:14:58.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:14:57 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:14:56.653417+0000 mgr.smithi029.loxrfu (mgr.14164) 113 : cluster [DBG] pgmap v106: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:14:58.715 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:58.711+0000 7ff3e210f700 1 -- 172.21.15.73:0/2325428389 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3dc0f51a0 msgr2=0x7ff3dc0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:14:58.715 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:58.711+0000 7ff3e210f700 1 --2- 172.21.15.73:0/2325428389 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3dc0f51a0 0x7ff3dc0f55c0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0x7ff3cc009fb0 tx=0x7ff3cc009b30).stop 2023-12-01T22:14:58.715 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:58.712+0000 7ff3e210f700 1 -- 172.21.15.73:0/2325428389 shutdown_connections 2023-12-01T22:14:58.715 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:58.712+0000 7ff3e210f700 1 --2- 172.21.15.73:0/2325428389 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3dc0f51a0 0x7ff3dc0f55c0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:14:58.715 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:58.712+0000 7ff3e210f700 1 -- 172.21.15.73:0/2325428389 >> 172.21.15.73:0/2325428389 conn(0x7ff3dc0f09a0 msgr2=0x7ff3dc0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:14:58.715 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:58.712+0000 7ff3e210f700 1 -- 172.21.15.73:0/2325428389 shutdown_connections 2023-12-01T22:14:58.715 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:58.712+0000 7ff3e210f700 1 -- 172.21.15.73:0/2325428389 wait complete. 2023-12-01T22:14:58.715 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:58.712+0000 7ff3e210f700 1 Processor -- start 2023-12-01T22:14:58.716 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:58.712+0000 7ff3e210f700 1 -- start start 2023-12-01T22:14:58.716 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:58.712+0000 7ff3e210f700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3dc0f51a0 0x7ff3dc1091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:14:58.716 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:58.713+0000 7ff3e210f700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3dc060240 con 0x7ff3dc0f51a0 2023-12-01T22:14:58.716 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:58.713+0000 7ff3e110d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3dc0f51a0 0x7ff3dc1091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:14:58.716 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:58.713+0000 7ff3e110d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3dc0f51a0 0x7ff3dc1091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:40944/0 (socket says 172.21.15.73:40944) 2023-12-01T22:14:58.716 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:58.713+0000 7ff3e110d700 1 -- 172.21.15.73:0/3340988152 learned_addr learned my addr 172.21.15.73:0/3340988152 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:14:58.716 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:58.713+0000 7ff3e110d700 1 -- 172.21.15.73:0/3340988152 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3cc005040 con 0x7ff3dc0f51a0 2023-12-01T22:14:58.716 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:58.713+0000 7ff3e110d700 1 --2- 172.21.15.73:0/3340988152 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3dc0f51a0 0x7ff3dc1091b0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7ff3cc013510 tx=0x7ff3cc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:14:58.716 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:58.713+0000 7ff3d1ffb700 1 -- 172.21.15.73:0/3340988152 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff3cc009320 con 0x7ff3dc0f51a0 2023-12-01T22:14:58.716 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:58.713+0000 7ff3e210f700 1 -- 172.21.15.73:0/3340988152 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3dc060440 con 0x7ff3dc0f51a0 2023-12-01T22:14:58.716 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:58.713+0000 7ff3d1ffb700 1 -- 172.21.15.73:0/3340988152 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff3cc013f00 con 0x7ff3dc0f51a0 2023-12-01T22:14:58.716 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:58.713+0000 7ff3d1ffb700 1 -- 172.21.15.73:0/3340988152 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff3cc014dc0 con 0x7ff3dc0f51a0 2023-12-01T22:14:58.717 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:58.714+0000 7ff3e210f700 1 -- 172.21.15.73:0/3340988152 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3dc109860 con 0x7ff3dc0f51a0 2023-12-01T22:14:58.717 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:58.714+0000 7ff3d1ffb700 1 -- 172.21.15.73:0/3340988152 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff3cc0095f0 con 0x7ff3dc0f51a0 2023-12-01T22:14:58.717 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:58.714+0000 7ff3e210f700 1 -- 172.21.15.73:0/3340988152 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3dc045ad0 con 0x7ff3dc0f51a0 2023-12-01T22:14:58.718 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:58.715+0000 7ff3d1ffb700 1 --2- 172.21.15.73:0/3340988152 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7ff3c802ff60 0x7ff3c8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:14:58.718 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:58.715+0000 7ff3d1ffb700 1 -- 172.21.15.73:0/3340988152 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff3cc041040 con 0x7ff3dc0f51a0 2023-12-01T22:14:58.718 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:58.715+0000 7ff3e090c700 1 --2- 172.21.15.73:0/3340988152 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7ff3c802ff60 0x7ff3c8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:14:58.718 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:58.715+0000 7ff3e090c700 1 --2- 172.21.15.73:0/3340988152 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7ff3c802ff60 0x7ff3c8032420 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7ff3d80096a0 tx=0x7ff3d8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:14:58.721 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:58.719+0000 7ff3d1ffb700 1 -- 172.21.15.73:0/3340988152 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff3cc014420 con 0x7ff3dc0f51a0 2023-12-01T22:14:58.928 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:58.926+0000 7ff3e210f700 1 -- 172.21.15.73:0/3340988152 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff3dc10a080 con 0x7ff3dc0f51a0 2023-12-01T22:14:58.928 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:58.926+0000 7ff3d1ffb700 1 -- 172.21.15.73:0/3340988152 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff3cc010230 con 0x7ff3dc0f51a0 2023-12-01T22:14:58.928 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:14:58.929 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:14:58.930 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:58.928+0000 7ff3e210f700 1 -- 172.21.15.73:0/3340988152 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7ff3c802ff60 msgr2=0x7ff3c8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:14:58.930 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:58.928+0000 7ff3e210f700 1 --2- 172.21.15.73:0/3340988152 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7ff3c802ff60 0x7ff3c8032420 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7ff3d80096a0 tx=0x7ff3d8006b40).stop 2023-12-01T22:14:58.930 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:58.928+0000 7ff3e210f700 1 -- 172.21.15.73:0/3340988152 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3dc0f51a0 msgr2=0x7ff3dc1091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:14:58.930 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:58.928+0000 7ff3e210f700 1 --2- 172.21.15.73:0/3340988152 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3dc0f51a0 0x7ff3dc1091b0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7ff3cc013510 tx=0x7ff3cc00b040).stop 2023-12-01T22:14:58.930 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:58.928+0000 7ff3e210f700 1 -- 172.21.15.73:0/3340988152 shutdown_connections 2023-12-01T22:14:58.930 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:58.928+0000 7ff3e210f700 1 --2- 172.21.15.73:0/3340988152 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7ff3c802ff60 0x7ff3c8032420 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:14:58.930 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:58.928+0000 7ff3e210f700 1 --2- 172.21.15.73:0/3340988152 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3dc0f51a0 0x7ff3dc1091b0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:14:58.930 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:58.928+0000 7ff3e210f700 1 -- 172.21.15.73:0/3340988152 >> 172.21.15.73:0/3340988152 conn(0x7ff3dc0f09a0 msgr2=0x7ff3dc0f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:14:58.930 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:58.928+0000 7ff3e210f700 1 -- 172.21.15.73:0/3340988152 shutdown_connections 2023-12-01T22:14:58.930 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:14:58.928+0000 7ff3e210f700 1 -- 172.21.15.73:0/3340988152 wait complete. 2023-12-01T22:14:58.931 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:15:00.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:14:59 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:14:58.653757+0000 mgr.smithi029.loxrfu (mgr.14164) 114 : cluster [DBG] pgmap v107: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:15:00.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:14:59 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:14:58.927805+0000 mon.smithi029 (mon.0) 194 : audit [DBG] from='client.? 172.21.15.73:0/3340988152' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:15:00.315 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:15:00.315 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:15:01.508 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:01.505+0000 7fc024be4700 1 -- 172.21.15.73:0/1881099101 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0200f5410 msgr2=0x7fc0200f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:15:01.509 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:01.505+0000 7fc024be4700 1 --2- 172.21.15.73:0/1881099101 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0200f5410 0x7fc0200f5830 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7fc008009fb0 tx=0x7fc008009b30).stop 2023-12-01T22:15:01.509 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:01.505+0000 7fc024be4700 1 -- 172.21.15.73:0/1881099101 shutdown_connections 2023-12-01T22:15:01.509 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:01.505+0000 7fc024be4700 1 --2- 172.21.15.73:0/1881099101 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0200f5410 0x7fc0200f5830 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:15:01.509 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:01.505+0000 7fc024be4700 1 -- 172.21.15.73:0/1881099101 >> 172.21.15.73:0/1881099101 conn(0x7fc0200f0b70 msgr2=0x7fc0200f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:15:01.509 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:01.505+0000 7fc024be4700 1 -- 172.21.15.73:0/1881099101 shutdown_connections 2023-12-01T22:15:01.509 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:01.505+0000 7fc024be4700 1 -- 172.21.15.73:0/1881099101 wait complete. 2023-12-01T22:15:01.509 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:01.506+0000 7fc024be4700 1 Processor -- start 2023-12-01T22:15:01.509 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:01.506+0000 7fc024be4700 1 -- start start 2023-12-01T22:15:01.509 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:01.506+0000 7fc024be4700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0200f5410 0x7fc0201093f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:15:01.509 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:01.506+0000 7fc024be4700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc020060280 con 0x7fc0200f5410 2023-12-01T22:15:01.509 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:01.506+0000 7fc01e59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0200f5410 0x7fc0201093f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:15:01.509 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:01.506+0000 7fc01e59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0200f5410 0x7fc0201093f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:40962/0 (socket says 172.21.15.73:40962) 2023-12-01T22:15:01.509 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:01.506+0000 7fc01e59c700 1 -- 172.21.15.73:0/1103518274 learned_addr learned my addr 172.21.15.73:0/1103518274 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:15:01.510 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:01.506+0000 7fc01e59c700 1 -- 172.21.15.73:0/1103518274 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc008005040 con 0x7fc0200f5410 2023-12-01T22:15:01.510 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:01.507+0000 7fc01e59c700 1 --2- 172.21.15.73:0/1103518274 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0200f5410 0x7fc0201093f0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7fc008013510 tx=0x7fc00800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:15:01.510 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:01.507+0000 7fc016ffd700 1 -- 172.21.15.73:0/1103518274 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc008009320 con 0x7fc0200f5410 2023-12-01T22:15:01.510 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:01.507+0000 7fc024be4700 1 -- 172.21.15.73:0/1103518274 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc020060480 con 0x7fc0200f5410 2023-12-01T22:15:01.510 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:01.507+0000 7fc016ffd700 1 -- 172.21.15.73:0/1103518274 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc008013f00 con 0x7fc0200f5410 2023-12-01T22:15:01.510 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:01.507+0000 7fc016ffd700 1 -- 172.21.15.73:0/1103518274 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc008014d90 con 0x7fc0200f5410 2023-12-01T22:15:01.510 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:01.507+0000 7fc024be4700 1 -- 172.21.15.73:0/1103518274 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc020109b20 con 0x7fc0200f5410 2023-12-01T22:15:01.511 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:01.508+0000 7fc024be4700 1 -- 172.21.15.73:0/1103518274 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc020045ad0 con 0x7fc0200f5410 2023-12-01T22:15:01.511 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:01.508+0000 7fc016ffd700 1 -- 172.21.15.73:0/1103518274 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc0080095f0 con 0x7fc0200f5410 2023-12-01T22:15:01.511 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:01.508+0000 7fc016ffd700 1 --2- 172.21.15.73:0/1103518274 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fc00c02ff60 0x7fc00c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:15:01.511 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:01.508+0000 7fc016ffd700 1 -- 172.21.15.73:0/1103518274 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc008040fc0 con 0x7fc0200f5410 2023-12-01T22:15:01.511 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:01.508+0000 7fc01dd9b700 1 --2- 172.21.15.73:0/1103518274 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fc00c02ff60 0x7fc00c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:15:01.511 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:01.508+0000 7fc01dd9b700 1 --2- 172.21.15.73:0/1103518274 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fc00c02ff60 0x7fc00c032420 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7fc0100096a0 tx=0x7fc010006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:15:01.514 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:01.513+0000 7fc016ffd700 1 -- 172.21.15.73:0/1103518274 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc008014420 con 0x7fc0200f5410 2023-12-01T22:15:01.720 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:01.718+0000 7fc024be4700 1 -- 172.21.15.73:0/1103518274 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc02010a370 con 0x7fc0200f5410 2023-12-01T22:15:01.721 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:01.719+0000 7fc016ffd700 1 -- 172.21.15.73:0/1103518274 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc008010230 con 0x7fc0200f5410 2023-12-01T22:15:01.721 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:15:01.721 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:15:01.722 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:01.720+0000 7fc024be4700 1 -- 172.21.15.73:0/1103518274 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fc00c02ff60 msgr2=0x7fc00c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:15:01.722 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:01.720+0000 7fc024be4700 1 --2- 172.21.15.73:0/1103518274 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fc00c02ff60 0x7fc00c032420 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7fc0100096a0 tx=0x7fc010006b40).stop 2023-12-01T22:15:01.722 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:01.720+0000 7fc024be4700 1 -- 172.21.15.73:0/1103518274 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0200f5410 msgr2=0x7fc0201093f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:15:01.723 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:01.720+0000 7fc024be4700 1 --2- 172.21.15.73:0/1103518274 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0200f5410 0x7fc0201093f0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7fc008013510 tx=0x7fc00800b040).stop 2023-12-01T22:15:01.723 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:01.721+0000 7fc024be4700 1 -- 172.21.15.73:0/1103518274 shutdown_connections 2023-12-01T22:15:01.723 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:01.721+0000 7fc024be4700 1 --2- 172.21.15.73:0/1103518274 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fc00c02ff60 0x7fc00c032420 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:15:01.723 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:01.721+0000 7fc024be4700 1 --2- 172.21.15.73:0/1103518274 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0200f5410 0x7fc0201093f0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:15:01.723 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:01.721+0000 7fc024be4700 1 -- 172.21.15.73:0/1103518274 >> 172.21.15.73:0/1103518274 conn(0x7fc0200f0b70 msgr2=0x7fc0200f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:15:01.723 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:01.721+0000 7fc024be4700 1 -- 172.21.15.73:0/1103518274 shutdown_connections 2023-12-01T22:15:01.723 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:01.721+0000 7fc024be4700 1 -- 172.21.15.73:0/1103518274 wait complete. 2023-12-01T22:15:01.723 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:15:01.776 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:15:01 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:15:00.654052+0000 mgr.smithi029.loxrfu (mgr.14164) 115 : cluster [DBG] pgmap v108: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:15:02.776 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:15:02 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:15:01.720471+0000 mon.smithi029 (mon.0) 195 : audit [DBG] from='client.? 172.21.15.73:0/1103518274' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:15:03.117 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:15:03.117 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:15:03.776 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:15:03 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:15:02.654373+0000 mgr.smithi029.loxrfu (mgr.14164) 116 : cluster [DBG] pgmap v109: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:15:04.291 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:04.289+0000 7fb28f2be700 1 -- 172.21.15.73:0/3304973181 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2880f5410 msgr2=0x7fb2880f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:15:04.291 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:04.289+0000 7fb28f2be700 1 --2- 172.21.15.73:0/3304973181 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2880f5410 0x7fb2880f5830 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7fb278009fb0 tx=0x7fb278009b30).stop 2023-12-01T22:15:04.291 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:04.289+0000 7fb28f2be700 1 -- 172.21.15.73:0/3304973181 shutdown_connections 2023-12-01T22:15:04.291 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:04.289+0000 7fb28f2be700 1 --2- 172.21.15.73:0/3304973181 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2880f5410 0x7fb2880f5830 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:15:04.291 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:04.289+0000 7fb28f2be700 1 -- 172.21.15.73:0/3304973181 >> 172.21.15.73:0/3304973181 conn(0x7fb2880f0b70 msgr2=0x7fb2880f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:15:04.291 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:04.289+0000 7fb28f2be700 1 -- 172.21.15.73:0/3304973181 shutdown_connections 2023-12-01T22:15:04.291 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:04.289+0000 7fb28f2be700 1 -- 172.21.15.73:0/3304973181 wait complete. 2023-12-01T22:15:04.291 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:04.290+0000 7fb28f2be700 1 Processor -- start 2023-12-01T22:15:04.291 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:04.290+0000 7fb28f2be700 1 -- start start 2023-12-01T22:15:04.291 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:04.290+0000 7fb28f2be700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2880f5410 0x7fb2880fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:15:04.292 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:04.290+0000 7fb28f2be700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2880fb820 con 0x7fb2880f5410 2023-12-01T22:15:04.292 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:04.290+0000 7fb28d05a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2880f5410 0x7fb2880fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:15:04.292 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:04.290+0000 7fb28d05a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2880f5410 0x7fb2880fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:40980/0 (socket says 172.21.15.73:40980) 2023-12-01T22:15:04.292 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:04.290+0000 7fb28d05a700 1 -- 172.21.15.73:0/2069723973 learned_addr learned my addr 172.21.15.73:0/2069723973 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:15:04.292 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:04.290+0000 7fb28d05a700 1 -- 172.21.15.73:0/2069723973 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb278005040 con 0x7fb2880f5410 2023-12-01T22:15:04.292 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:04.290+0000 7fb28d05a700 1 --2- 172.21.15.73:0/2069723973 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2880f5410 0x7fb2880fd3a0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7fb278013510 tx=0x7fb27800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:15:04.293 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:04.290+0000 7fb27dffb700 1 -- 172.21.15.73:0/2069723973 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb278009320 con 0x7fb2880f5410 2023-12-01T22:15:04.293 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:04.291+0000 7fb28f2be700 1 -- 172.21.15.73:0/2069723973 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2880fb9c0 con 0x7fb2880f5410 2023-12-01T22:15:04.293 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:04.291+0000 7fb27dffb700 1 -- 172.21.15.73:0/2069723973 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb278013f00 con 0x7fb2880f5410 2023-12-01T22:15:04.294 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:04.291+0000 7fb27dffb700 1 -- 172.21.15.73:0/2069723973 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb278014d90 con 0x7fb2880f5410 2023-12-01T22:15:04.294 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:04.291+0000 7fb28f2be700 1 -- 172.21.15.73:0/2069723973 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2880fbe50 con 0x7fb2880f5410 2023-12-01T22:15:04.294 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:04.292+0000 7fb27dffb700 1 -- 172.21.15.73:0/2069723973 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb2780095f0 con 0x7fb2880f5410 2023-12-01T22:15:04.294 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:04.292+0000 7fb28f2be700 1 -- 172.21.15.73:0/2069723973 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb288059100 con 0x7fb2880f5410 2023-12-01T22:15:04.294 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:04.292+0000 7fb27dffb700 1 --2- 172.21.15.73:0/2069723973 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fb27402ff60 0x7fb274032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:15:04.294 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:04.292+0000 7fb27dffb700 1 -- 172.21.15.73:0/2069723973 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb278040fe0 con 0x7fb2880f5410 2023-12-01T22:15:04.294 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:04.292+0000 7fb28c859700 1 --2- 172.21.15.73:0/2069723973 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fb27402ff60 0x7fb274032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:15:04.294 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:04.292+0000 7fb28c859700 1 --2- 172.21.15.73:0/2069723973 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fb27402ff60 0x7fb274032420 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7fb2840096a0 tx=0x7fb284006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:15:04.297 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:04.297+0000 7fb27dffb700 1 -- 172.21.15.73:0/2069723973 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb278014420 con 0x7fb2880f5410 2023-12-01T22:15:04.503 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:04.502+0000 7fb28f2be700 1 -- 172.21.15.73:0/2069723973 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb288045ad0 con 0x7fb2880f5410 2023-12-01T22:15:04.504 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:04.503+0000 7fb27dffb700 1 -- 172.21.15.73:0/2069723973 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb278010230 con 0x7fb2880f5410 2023-12-01T22:15:04.504 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:15:04.505 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:15:04.506 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:04.505+0000 7fb28f2be700 1 -- 172.21.15.73:0/2069723973 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fb27402ff60 msgr2=0x7fb274032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:15:04.506 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:04.505+0000 7fb28f2be700 1 --2- 172.21.15.73:0/2069723973 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fb27402ff60 0x7fb274032420 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7fb2840096a0 tx=0x7fb284006b40).stop 2023-12-01T22:15:04.506 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:04.505+0000 7fb28f2be700 1 -- 172.21.15.73:0/2069723973 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2880f5410 msgr2=0x7fb2880fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:15:04.506 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:04.505+0000 7fb28f2be700 1 --2- 172.21.15.73:0/2069723973 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2880f5410 0x7fb2880fd3a0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7fb278013510 tx=0x7fb27800b040).stop 2023-12-01T22:15:04.506 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:04.505+0000 7fb28f2be700 1 -- 172.21.15.73:0/2069723973 shutdown_connections 2023-12-01T22:15:04.506 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:04.505+0000 7fb28f2be700 1 --2- 172.21.15.73:0/2069723973 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fb27402ff60 0x7fb274032420 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:15:04.506 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:04.505+0000 7fb28f2be700 1 --2- 172.21.15.73:0/2069723973 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2880f5410 0x7fb2880fd3a0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:15:04.506 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:04.505+0000 7fb28f2be700 1 -- 172.21.15.73:0/2069723973 >> 172.21.15.73:0/2069723973 conn(0x7fb2880f0b70 msgr2=0x7fb28805fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:15:04.507 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:04.505+0000 7fb28f2be700 1 -- 172.21.15.73:0/2069723973 shutdown_connections 2023-12-01T22:15:04.507 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:04.505+0000 7fb28f2be700 1 -- 172.21.15.73:0/2069723973 wait complete. 2023-12-01T22:15:04.507 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:15:05.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:15:04 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:15:04.503508+0000 mon.smithi029 (mon.0) 196 : audit [DBG] from='client.? 172.21.15.73:0/2069723973' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:15:05.869 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:15:05.870 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:15:06.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:15:05 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:15:04.654706+0000 mgr.smithi029.loxrfu (mgr.14164) 117 : cluster [DBG] pgmap v110: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:15:06.989 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:06.987+0000 7f51ad870700 1 -- 172.21.15.73:0/4172801535 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51a80f5410 msgr2=0x7f51a80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:15:06.989 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:06.987+0000 7f51ad870700 1 --2- 172.21.15.73:0/4172801535 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51a80f5410 0x7f51a80f5830 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0x7f519c009fb0 tx=0x7f519c009b30).stop 2023-12-01T22:15:06.989 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:06.987+0000 7f51ad870700 1 -- 172.21.15.73:0/4172801535 shutdown_connections 2023-12-01T22:15:06.989 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:06.987+0000 7f51ad870700 1 --2- 172.21.15.73:0/4172801535 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51a80f5410 0x7f51a80f5830 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:15:06.989 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:06.987+0000 7f51ad870700 1 -- 172.21.15.73:0/4172801535 >> 172.21.15.73:0/4172801535 conn(0x7f51a80f0b70 msgr2=0x7f51a80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:15:06.989 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:06.987+0000 7f51ad870700 1 -- 172.21.15.73:0/4172801535 shutdown_connections 2023-12-01T22:15:06.989 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:06.987+0000 7f51ad870700 1 -- 172.21.15.73:0/4172801535 wait complete. 2023-12-01T22:15:06.989 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:06.987+0000 7f51ad870700 1 Processor -- start 2023-12-01T22:15:06.990 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:06.988+0000 7f51ad870700 1 -- start start 2023-12-01T22:15:06.990 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:06.988+0000 7f51ad870700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51a80f5410 0x7f51a8109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:15:06.990 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:06.988+0000 7f51ad870700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51a8060240 con 0x7f51a80f5410 2023-12-01T22:15:06.990 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:06.988+0000 7f51a6ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51a80f5410 0x7f51a8109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:15:06.990 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:06.988+0000 7f51a6ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51a80f5410 0x7f51a8109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:55376/0 (socket says 172.21.15.73:55376) 2023-12-01T22:15:06.990 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:06.988+0000 7f51a6ffd700 1 -- 172.21.15.73:0/520421949 learned_addr learned my addr 172.21.15.73:0/520421949 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:15:06.990 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:06.988+0000 7f51a6ffd700 1 -- 172.21.15.73:0/520421949 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f519c005040 con 0x7f51a80f5410 2023-12-01T22:15:06.990 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:06.988+0000 7f51a6ffd700 1 --2- 172.21.15.73:0/520421949 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51a80f5410 0x7f51a8109430 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7f519c014000 tx=0x7f519c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:15:06.991 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:06.989+0000 7f518f7fe700 1 -- 172.21.15.73:0/520421949 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f519c0095f0 con 0x7f51a80f5410 2023-12-01T22:15:06.991 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:06.989+0000 7f518f7fe700 1 -- 172.21.15.73:0/520421949 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f519c007e70 con 0x7f51a80f5410 2023-12-01T22:15:06.992 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:06.989+0000 7f51ad870700 1 -- 172.21.15.73:0/520421949 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51a8060440 con 0x7f51a80f5410 2023-12-01T22:15:06.992 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:06.989+0000 7f518f7fe700 1 -- 172.21.15.73:0/520421949 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f519c014ea0 con 0x7f51a80f5410 2023-12-01T22:15:06.992 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:06.989+0000 7f51ad870700 1 -- 172.21.15.73:0/520421949 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51a8109ae0 con 0x7f51a80f5410 2023-12-01T22:15:06.992 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:06.989+0000 7f518f7fe700 1 -- 172.21.15.73:0/520421949 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f519c025070 con 0x7f51a80f5410 2023-12-01T22:15:06.992 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:06.989+0000 7f51ad870700 1 -- 172.21.15.73:0/520421949 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51a8045ad0 con 0x7f51a80f5410 2023-12-01T22:15:06.992 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:06.990+0000 7f518f7fe700 1 --2- 172.21.15.73:0/520421949 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f519002ff60 0x7f5190032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:15:06.992 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:06.990+0000 7f518f7fe700 1 -- 172.21.15.73:0/520421949 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f519c0409f0 con 0x7f51a80f5410 2023-12-01T22:15:06.992 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:06.990+0000 7f51a67fc700 1 --2- 172.21.15.73:0/520421949 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f519002ff60 0x7f5190032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:15:06.992 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:06.990+0000 7f51a67fc700 1 --2- 172.21.15.73:0/520421949 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f519002ff60 0x7f5190032420 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f51980096a0 tx=0x7f5198006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:15:06.995 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:06.994+0000 7f518f7fe700 1 -- 172.21.15.73:0/520421949 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f519c0079c0 con 0x7f51a80f5410 2023-12-01T22:15:07.202 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:07.201+0000 7f51ad870700 1 -- 172.21.15.73:0/520421949 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f51a80fae60 con 0x7f51a80f5410 2023-12-01T22:15:07.202 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:07.201+0000 7f518f7fe700 1 -- 172.21.15.73:0/520421949 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f519c00f300 con 0x7f51a80f5410 2023-12-01T22:15:07.203 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:15:07.203 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:15:07.204 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:07.203+0000 7f51ad870700 1 -- 172.21.15.73:0/520421949 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f519002ff60 msgr2=0x7f5190032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:15:07.204 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:07.203+0000 7f51ad870700 1 --2- 172.21.15.73:0/520421949 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f519002ff60 0x7f5190032420 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f51980096a0 tx=0x7f5198006b40).stop 2023-12-01T22:15:07.204 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:07.203+0000 7f51ad870700 1 -- 172.21.15.73:0/520421949 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51a80f5410 msgr2=0x7f51a8109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:15:07.204 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:07.203+0000 7f51ad870700 1 --2- 172.21.15.73:0/520421949 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51a80f5410 0x7f51a8109430 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7f519c014000 tx=0x7f519c00b040).stop 2023-12-01T22:15:07.204 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:07.203+0000 7f51ad870700 1 -- 172.21.15.73:0/520421949 shutdown_connections 2023-12-01T22:15:07.204 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:07.203+0000 7f51ad870700 1 --2- 172.21.15.73:0/520421949 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f519002ff60 0x7f5190032420 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:15:07.204 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:07.203+0000 7f51ad870700 1 --2- 172.21.15.73:0/520421949 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51a80f5410 0x7f51a8109430 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:15:07.205 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:07.203+0000 7f51ad870700 1 -- 172.21.15.73:0/520421949 >> 172.21.15.73:0/520421949 conn(0x7f51a80f0b70 msgr2=0x7f51a80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:15:07.205 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:07.203+0000 7f51ad870700 1 -- 172.21.15.73:0/520421949 shutdown_connections 2023-12-01T22:15:07.205 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:07.203+0000 7f51ad870700 1 -- 172.21.15.73:0/520421949 wait complete. 2023-12-01T22:15:07.205 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:15:08.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:15:07 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:15:06.655049+0000 mgr.smithi029.loxrfu (mgr.14164) 118 : cluster [DBG] pgmap v111: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:15:08.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:15:07 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:15:07.201652+0000 mon.smithi029 (mon.0) 197 : audit [DBG] from='client.? 172.21.15.73:0/520421949' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:15:08.586 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:15:08.587 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:15:09.720 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:09.718+0000 7fc77c52b700 1 -- 172.21.15.73:0/2208973617 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7740f5410 msgr2=0x7fc7740f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:15:09.721 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:09.718+0000 7fc77c52b700 1 --2- 172.21.15.73:0/2208973617 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7740f5410 0x7fc7740f5830 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7fc764009fb0 tx=0x7fc764009b30).stop 2023-12-01T22:15:09.721 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:09.719+0000 7fc77c52b700 1 -- 172.21.15.73:0/2208973617 shutdown_connections 2023-12-01T22:15:09.721 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:09.719+0000 7fc77c52b700 1 --2- 172.21.15.73:0/2208973617 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7740f5410 0x7fc7740f5830 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:15:09.721 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:09.719+0000 7fc77c52b700 1 -- 172.21.15.73:0/2208973617 >> 172.21.15.73:0/2208973617 conn(0x7fc7740f0b70 msgr2=0x7fc7740f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:15:09.721 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:09.719+0000 7fc77c52b700 1 -- 172.21.15.73:0/2208973617 shutdown_connections 2023-12-01T22:15:09.721 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:09.719+0000 7fc77c52b700 1 -- 172.21.15.73:0/2208973617 wait complete. 2023-12-01T22:15:09.721 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:09.719+0000 7fc77c52b700 1 Processor -- start 2023-12-01T22:15:09.721 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:09.719+0000 7fc77c52b700 1 -- start start 2023-12-01T22:15:09.721 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:09.719+0000 7fc77c52b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7740f5410 0x7fc7741093f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:15:09.721 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:09.719+0000 7fc77c52b700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc774060280 con 0x7fc7740f5410 2023-12-01T22:15:09.721 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:09.719+0000 7fc77a2c7700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7740f5410 0x7fc7741093f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:15:09.721 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:09.719+0000 7fc77a2c7700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7740f5410 0x7fc7741093f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:55390/0 (socket says 172.21.15.73:55390) 2023-12-01T22:15:09.722 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:09.719+0000 7fc77a2c7700 1 -- 172.21.15.73:0/128808217 learned_addr learned my addr 172.21.15.73:0/128808217 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:15:09.722 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:09.720+0000 7fc77a2c7700 1 -- 172.21.15.73:0/128808217 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc764005040 con 0x7fc7740f5410 2023-12-01T22:15:09.722 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:09.720+0000 7fc77a2c7700 1 --2- 172.21.15.73:0/128808217 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7740f5410 0x7fc7741093f0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7fc764013510 tx=0x7fc76400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:15:09.722 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:09.720+0000 7fc76affd700 1 -- 172.21.15.73:0/128808217 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc764009320 con 0x7fc7740f5410 2023-12-01T22:15:09.723 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:09.720+0000 7fc77c52b700 1 -- 172.21.15.73:0/128808217 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc774060480 con 0x7fc7740f5410 2023-12-01T22:15:09.723 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:09.720+0000 7fc76affd700 1 -- 172.21.15.73:0/128808217 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc764013f00 con 0x7fc7740f5410 2023-12-01T22:15:09.723 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:09.720+0000 7fc76affd700 1 -- 172.21.15.73:0/128808217 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc764014d90 con 0x7fc7740f5410 2023-12-01T22:15:09.723 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:09.720+0000 7fc77c52b700 1 -- 172.21.15.73:0/128808217 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc774109b20 con 0x7fc7740f5410 2023-12-01T22:15:09.723 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:09.721+0000 7fc76affd700 1 -- 172.21.15.73:0/128808217 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc7640095f0 con 0x7fc7740f5410 2023-12-01T22:15:09.723 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:09.721+0000 7fc77c52b700 1 -- 172.21.15.73:0/128808217 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc774045ad0 con 0x7fc7740f5410 2023-12-01T22:15:09.723 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:09.721+0000 7fc76affd700 1 --2- 172.21.15.73:0/128808217 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fc76002ff60 0x7fc760032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:15:09.723 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:09.721+0000 7fc76affd700 1 -- 172.21.15.73:0/128808217 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc764041000 con 0x7fc7740f5410 2023-12-01T22:15:09.723 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:09.722+0000 7fc779ac6700 1 --2- 172.21.15.73:0/128808217 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fc76002ff60 0x7fc760032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:15:09.723 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:09.722+0000 7fc779ac6700 1 --2- 172.21.15.73:0/128808217 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fc76002ff60 0x7fc760032420 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7fc7700096a0 tx=0x7fc770006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:15:09.727 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:09.726+0000 7fc76affd700 1 -- 172.21.15.73:0/128808217 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc764014420 con 0x7fc7740f5410 2023-12-01T22:15:09.931 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:09.930+0000 7fc77c52b700 1 -- 172.21.15.73:0/128808217 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc77410a340 con 0x7fc7740f5410 2023-12-01T22:15:09.932 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:09.931+0000 7fc76affd700 1 -- 172.21.15.73:0/128808217 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc764010230 con 0x7fc7740f5410 2023-12-01T22:15:09.932 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:15:09.932 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:15:09.934 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:09.933+0000 7fc77c52b700 1 -- 172.21.15.73:0/128808217 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fc76002ff60 msgr2=0x7fc760032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:15:09.934 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:09.933+0000 7fc77c52b700 1 --2- 172.21.15.73:0/128808217 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fc76002ff60 0x7fc760032420 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7fc7700096a0 tx=0x7fc770006b40).stop 2023-12-01T22:15:09.934 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:09.933+0000 7fc77c52b700 1 -- 172.21.15.73:0/128808217 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7740f5410 msgr2=0x7fc7741093f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:15:09.934 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:09.933+0000 7fc77c52b700 1 --2- 172.21.15.73:0/128808217 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7740f5410 0x7fc7741093f0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7fc764013510 tx=0x7fc76400b040).stop 2023-12-01T22:15:09.934 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:09.933+0000 7fc77c52b700 1 -- 172.21.15.73:0/128808217 shutdown_connections 2023-12-01T22:15:09.934 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:09.933+0000 7fc77c52b700 1 --2- 172.21.15.73:0/128808217 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fc76002ff60 0x7fc760032420 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:15:09.934 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:09.933+0000 7fc77c52b700 1 --2- 172.21.15.73:0/128808217 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7740f5410 0x7fc7741093f0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:15:09.934 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:09.933+0000 7fc77c52b700 1 -- 172.21.15.73:0/128808217 >> 172.21.15.73:0/128808217 conn(0x7fc7740f0b70 msgr2=0x7fc7740f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:15:09.934 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:09.933+0000 7fc77c52b700 1 -- 172.21.15.73:0/128808217 shutdown_connections 2023-12-01T22:15:09.934 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:09.933+0000 7fc77c52b700 1 -- 172.21.15.73:0/128808217 wait complete. 2023-12-01T22:15:09.935 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:15:10.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:15:09 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:15:08.655396+0000 mgr.smithi029.loxrfu (mgr.14164) 119 : cluster [DBG] pgmap v112: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:15:11.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:15:10 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:15:09.931591+0000 mon.smithi029 (mon.0) 198 : audit [DBG] from='client.? 172.21.15.73:0/128808217' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:15:11.359 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:15:11.359 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:15:12.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:15:11 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:15:10.655736+0000 mgr.smithi029.loxrfu (mgr.14164) 120 : cluster [DBG] pgmap v113: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:15:12.482 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:12.480+0000 7f97f9cab700 1 -- 172.21.15.73:0/2083171216 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97f40f47c0 msgr2=0x7f97f40f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:15:12.483 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:12.480+0000 7f97f9cab700 1 --2- 172.21.15.73:0/2083171216 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97f40f47c0 0x7f97f40f4be0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7f97e8009fb0 tx=0x7f97e8009b30).stop 2023-12-01T22:15:12.483 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:12.481+0000 7f97f9cab700 1 -- 172.21.15.73:0/2083171216 shutdown_connections 2023-12-01T22:15:12.483 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:12.481+0000 7f97f9cab700 1 --2- 172.21.15.73:0/2083171216 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97f40f47c0 0x7f97f40f4be0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:15:12.483 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:12.481+0000 7f97f9cab700 1 -- 172.21.15.73:0/2083171216 >> 172.21.15.73:0/2083171216 conn(0x7f97f40f02e0 msgr2=0x7f97f40f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:15:12.483 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:12.481+0000 7f97f9cab700 1 -- 172.21.15.73:0/2083171216 shutdown_connections 2023-12-01T22:15:12.483 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:12.481+0000 7f97f9cab700 1 -- 172.21.15.73:0/2083171216 wait complete. 2023-12-01T22:15:12.484 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:12.481+0000 7f97f9cab700 1 Processor -- start 2023-12-01T22:15:12.484 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:12.481+0000 7f97f9cab700 1 -- start start 2023-12-01T22:15:12.485 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:12.482+0000 7f97f9cab700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97f41067d0 0x7f97f4106bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:15:12.485 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:12.482+0000 7f97f9cab700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97f41070e0 con 0x7f97f41067d0 2023-12-01T22:15:12.485 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:12.482+0000 7f97f37fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97f41067d0 0x7f97f4106bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:15:12.485 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:12.482+0000 7f97f37fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97f41067d0 0x7f97f4106bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:55394/0 (socket says 172.21.15.73:55394) 2023-12-01T22:15:12.485 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:12.482+0000 7f97f37fe700 1 -- 172.21.15.73:0/2881683643 learned_addr learned my addr 172.21.15.73:0/2881683643 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:15:12.485 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:12.482+0000 7f97f37fe700 1 -- 172.21.15.73:0/2881683643 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97e8005040 con 0x7f97f41067d0 2023-12-01T22:15:12.485 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:12.483+0000 7f97f37fe700 1 --2- 172.21.15.73:0/2881683643 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97f41067d0 0x7f97f4106bf0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7f97e8009f80 tx=0x7f97e8009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:15:12.485 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:12.483+0000 7f97dbfff700 1 -- 172.21.15.73:0/2881683643 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f97e80136c0 con 0x7f97f41067d0 2023-12-01T22:15:12.485 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:12.483+0000 7f97f9cab700 1 -- 172.21.15.73:0/2881683643 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97f41072e0 con 0x7f97f41067d0 2023-12-01T22:15:12.485 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:12.483+0000 7f97dbfff700 1 -- 172.21.15.73:0/2881683643 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f97e80095f0 con 0x7f97f41067d0 2023-12-01T22:15:12.485 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:12.483+0000 7f97dbfff700 1 -- 172.21.15.73:0/2881683643 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f97e8014420 con 0x7f97f41067d0 2023-12-01T22:15:12.485 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:12.483+0000 7f97f9cab700 1 -- 172.21.15.73:0/2881683643 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97f4107e30 con 0x7f97f41067d0 2023-12-01T22:15:12.486 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:12.483+0000 7f97f9cab700 1 -- 172.21.15.73:0/2881683643 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97f4059100 con 0x7f97f41067d0 2023-12-01T22:15:12.486 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:12.484+0000 7f97dbfff700 1 -- 172.21.15.73:0/2881683643 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f97e8025060 con 0x7f97f41067d0 2023-12-01T22:15:12.486 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:12.484+0000 7f97dbfff700 1 --2- 172.21.15.73:0/2881683643 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f97dc02ff60 0x7f97dc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:15:12.486 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:12.484+0000 7f97dbfff700 1 -- 172.21.15.73:0/2881683643 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f97e8040d80 con 0x7f97f41067d0 2023-12-01T22:15:12.486 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:12.484+0000 7f97f2ffd700 1 --2- 172.21.15.73:0/2881683643 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f97dc02ff60 0x7f97dc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:15:12.487 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:12.484+0000 7f97f2ffd700 1 --2- 172.21.15.73:0/2881683643 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f97dc02ff60 0x7f97dc032420 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f97e40096a0 tx=0x7f97e4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:15:12.490 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:12.489+0000 7f97dbfff700 1 -- 172.21.15.73:0/2881683643 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f97e801cb60 con 0x7f97f41067d0 2023-12-01T22:15:12.697 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:12.695+0000 7f97f9cab700 1 -- 172.21.15.73:0/2881683643 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f97f41085f0 con 0x7f97f41067d0 2023-12-01T22:15:12.697 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:12.696+0000 7f97dbfff700 1 -- 172.21.15.73:0/2881683643 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f97e8042070 con 0x7f97f41067d0 2023-12-01T22:15:12.698 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:15:12.698 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:15:12.699 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:12.697+0000 7f97f9cab700 1 -- 172.21.15.73:0/2881683643 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f97dc02ff60 msgr2=0x7f97dc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:15:12.699 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:12.697+0000 7f97f9cab700 1 --2- 172.21.15.73:0/2881683643 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f97dc02ff60 0x7f97dc032420 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f97e40096a0 tx=0x7f97e4006b40).stop 2023-12-01T22:15:12.699 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:12.697+0000 7f97f9cab700 1 -- 172.21.15.73:0/2881683643 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97f41067d0 msgr2=0x7f97f4106bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:15:12.699 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:12.697+0000 7f97f9cab700 1 --2- 172.21.15.73:0/2881683643 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97f41067d0 0x7f97f4106bf0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7f97e8009f80 tx=0x7f97e8009b30).stop 2023-12-01T22:15:12.699 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:12.697+0000 7f97f9cab700 1 -- 172.21.15.73:0/2881683643 shutdown_connections 2023-12-01T22:15:12.700 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:12.697+0000 7f97f9cab700 1 --2- 172.21.15.73:0/2881683643 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f97dc02ff60 0x7f97dc032420 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:15:12.700 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:12.697+0000 7f97f9cab700 1 --2- 172.21.15.73:0/2881683643 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97f41067d0 0x7f97f4106bf0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:15:12.700 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:12.697+0000 7f97f9cab700 1 -- 172.21.15.73:0/2881683643 >> 172.21.15.73:0/2881683643 conn(0x7f97f40f02e0 msgr2=0x7f97f40f0ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:15:12.700 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:12.698+0000 7f97f9cab700 1 -- 172.21.15.73:0/2881683643 shutdown_connections 2023-12-01T22:15:12.700 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:12.698+0000 7f97f9cab700 1 -- 172.21.15.73:0/2881683643 wait complete. 2023-12-01T22:15:12.700 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:15:13.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:15:12 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:15:12.697255+0000 mon.smithi029 (mon.0) 199 : audit [DBG] from='client.? 172.21.15.73:0/2881683643' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:15:14.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:15:13 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:15:12.656111+0000 mgr.smithi029.loxrfu (mgr.14164) 121 : cluster [DBG] pgmap v114: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:15:14.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:15:13 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:15:12.779543+0000 mon.smithi029 (mon.0) 200 : audit [INF] from='mgr.14164 172.21.15.29:0/3195927111' entity='mgr.smithi029.loxrfu' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi029.loxrfu/mirror_snapshot_schedule"}]: dispatch 2023-12-01T22:15:14.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:15:13 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:15:12.783275+0000 mon.smithi029 (mon.0) 201 : audit [INF] from='mgr.14164 172.21.15.29:0/3195927111' entity='mgr.smithi029.loxrfu' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi029.loxrfu/trash_purge_schedule"}]: dispatch 2023-12-01T22:15:14.077 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:15:14.077 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:15:15.219 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:15.216+0000 7f10824ce700 1 -- 172.21.15.73:0/2101944777 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f107c0601e0 msgr2=0x7f107c0f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:15:15.219 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:15.216+0000 7f10824ce700 1 --2- 172.21.15.73:0/2101944777 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f107c0601e0 0x7f107c0f8f20 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0x7f106c009fb0 tx=0x7f106c009b30).stop 2023-12-01T22:15:15.219 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:15.217+0000 7f10824ce700 1 -- 172.21.15.73:0/2101944777 shutdown_connections 2023-12-01T22:15:15.219 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:15.217+0000 7f10824ce700 1 --2- 172.21.15.73:0/2101944777 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f107c0601e0 0x7f107c0f8f20 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:15:15.219 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:15.217+0000 7f10824ce700 1 -- 172.21.15.73:0/2101944777 >> 172.21.15.73:0/2101944777 conn(0x7f107c0f09a0 msgr2=0x7f107c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:15:15.219 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:15.217+0000 7f10824ce700 1 -- 172.21.15.73:0/2101944777 shutdown_connections 2023-12-01T22:15:15.220 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:15.217+0000 7f10824ce700 1 -- 172.21.15.73:0/2101944777 wait complete. 2023-12-01T22:15:15.220 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:15.217+0000 7f10824ce700 1 Processor -- start 2023-12-01T22:15:15.220 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:15.217+0000 7f10824ce700 1 -- start start 2023-12-01T22:15:15.220 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:15.217+0000 7f10824ce700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f107c0601e0 0x7f107c0f69a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:15:15.220 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:15.217+0000 7f10824ce700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f107c0f4e20 con 0x7f107c0601e0 2023-12-01T22:15:15.220 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:15.217+0000 7f10814cc700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f107c0601e0 0x7f107c0f69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:15:15.220 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:15.217+0000 7f10814cc700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f107c0601e0 0x7f107c0f69a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:47034/0 (socket says 172.21.15.73:47034) 2023-12-01T22:15:15.220 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:15.217+0000 7f10814cc700 1 -- 172.21.15.73:0/4187682508 learned_addr learned my addr 172.21.15.73:0/4187682508 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:15:15.222 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:15.218+0000 7f10814cc700 1 -- 172.21.15.73:0/4187682508 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f106c005040 con 0x7f107c0601e0 2023-12-01T22:15:15.222 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:15.218+0000 7f10814cc700 1 --2- 172.21.15.73:0/4187682508 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f107c0601e0 0x7f107c0f69a0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7f106c013510 tx=0x7f106c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:15:15.222 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:15.218+0000 7f1071ffb700 1 -- 172.21.15.73:0/4187682508 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f106c009320 con 0x7f107c0601e0 2023-12-01T22:15:15.222 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:15.218+0000 7f10824ce700 1 -- 172.21.15.73:0/4187682508 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f107c0f4fc0 con 0x7f107c0601e0 2023-12-01T22:15:15.222 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:15.218+0000 7f1071ffb700 1 -- 172.21.15.73:0/4187682508 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f106c013f00 con 0x7f107c0601e0 2023-12-01T22:15:15.222 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:15.218+0000 7f1071ffb700 1 -- 172.21.15.73:0/4187682508 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f106c014d90 con 0x7f107c0601e0 2023-12-01T22:15:15.222 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:15.218+0000 7f10824ce700 1 -- 172.21.15.73:0/4187682508 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f107c0f5450 con 0x7f107c0601e0 2023-12-01T22:15:15.222 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:15.219+0000 7f10824ce700 1 -- 172.21.15.73:0/4187682508 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f107c059100 con 0x7f107c0601e0 2023-12-01T22:15:15.222 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:15.219+0000 7f1071ffb700 1 -- 172.21.15.73:0/4187682508 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f106c0095f0 con 0x7f107c0601e0 2023-12-01T22:15:15.222 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:15.219+0000 7f1071ffb700 1 --2- 172.21.15.73:0/4187682508 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f106802ff60 0x7f1068032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:15:15.222 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:15.219+0000 7f1071ffb700 1 -- 172.21.15.73:0/4187682508 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f106c040fc0 con 0x7f107c0601e0 2023-12-01T22:15:15.222 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:15.219+0000 7f1080ccb700 1 --2- 172.21.15.73:0/4187682508 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f106802ff60 0x7f1068032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:15:15.223 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:15.220+0000 7f1080ccb700 1 --2- 172.21.15.73:0/4187682508 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f106802ff60 0x7f1068032420 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f10780096a0 tx=0x7f1078006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:15:15.225 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:15.224+0000 7f1071ffb700 1 -- 172.21.15.73:0/4187682508 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f106c014420 con 0x7f107c0601e0 2023-12-01T22:15:15.434 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:15.433+0000 7f10824ce700 1 -- 172.21.15.73:0/4187682508 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f107c045ad0 con 0x7f107c0601e0 2023-12-01T22:15:15.435 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:15.433+0000 7f1071ffb700 1 -- 172.21.15.73:0/4187682508 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f106c010230 con 0x7f107c0601e0 2023-12-01T22:15:15.435 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:15:15.435 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:15:15.437 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:15.435+0000 7f10824ce700 1 -- 172.21.15.73:0/4187682508 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f106802ff60 msgr2=0x7f1068032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:15:15.437 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:15.435+0000 7f10824ce700 1 --2- 172.21.15.73:0/4187682508 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f106802ff60 0x7f1068032420 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f10780096a0 tx=0x7f1078006b40).stop 2023-12-01T22:15:15.437 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:15.435+0000 7f10824ce700 1 -- 172.21.15.73:0/4187682508 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f107c0601e0 msgr2=0x7f107c0f69a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:15:15.437 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:15.435+0000 7f10824ce700 1 --2- 172.21.15.73:0/4187682508 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f107c0601e0 0x7f107c0f69a0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7f106c013510 tx=0x7f106c00b040).stop 2023-12-01T22:15:15.437 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:15.435+0000 7f10824ce700 1 -- 172.21.15.73:0/4187682508 shutdown_connections 2023-12-01T22:15:15.437 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:15.435+0000 7f10824ce700 1 --2- 172.21.15.73:0/4187682508 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f106802ff60 0x7f1068032420 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:15:15.437 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:15.435+0000 7f10824ce700 1 --2- 172.21.15.73:0/4187682508 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f107c0601e0 0x7f107c0f69a0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:15:15.437 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:15.435+0000 7f10824ce700 1 -- 172.21.15.73:0/4187682508 >> 172.21.15.73:0/4187682508 conn(0x7f107c0f09a0 msgr2=0x7f107c0f1420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:15:15.438 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:15.435+0000 7f10824ce700 1 -- 172.21.15.73:0/4187682508 shutdown_connections 2023-12-01T22:15:15.438 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:15.435+0000 7f10824ce700 1 -- 172.21.15.73:0/4187682508 wait complete. 2023-12-01T22:15:15.438 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:15:16.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:15:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:15:14.656397+0000 mgr.smithi029.loxrfu (mgr.14164) 122 : cluster [DBG] pgmap v115: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:15:16.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:15:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:15:15.434945+0000 mon.smithi029 (mon.0) 202 : audit [DBG] from='client.? 172.21.15.73:0/4187682508' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:15:16.827 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:15:16.828 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:15:18.003 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:17.999+0000 7f3f6e287700 1 -- 172.21.15.73:0/2023997108 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3f680601e0 msgr2=0x7f3f680f9190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:15:18.003 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:17.999+0000 7f3f6e287700 1 --2- 172.21.15.73:0/2023997108 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3f680601e0 0x7f3f680f9190 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0x7f3f58009fb0 tx=0x7f3f58009b30).stop 2023-12-01T22:15:18.003 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:17.999+0000 7f3f6e287700 1 -- 172.21.15.73:0/2023997108 shutdown_connections 2023-12-01T22:15:18.003 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:17.999+0000 7f3f6e287700 1 --2- 172.21.15.73:0/2023997108 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3f680601e0 0x7f3f680f9190 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:15:18.003 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:17.999+0000 7f3f6e287700 1 -- 172.21.15.73:0/2023997108 >> 172.21.15.73:0/2023997108 conn(0x7f3f680f0b70 msgr2=0x7f3f680f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:15:18.003 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:18.000+0000 7f3f6e287700 1 -- 172.21.15.73:0/2023997108 shutdown_connections 2023-12-01T22:15:18.003 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:18.000+0000 7f3f6e287700 1 -- 172.21.15.73:0/2023997108 wait complete. 2023-12-01T22:15:18.003 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:18.000+0000 7f3f6e287700 1 Processor -- start 2023-12-01T22:15:18.003 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:18.000+0000 7f3f6e287700 1 -- start start 2023-12-01T22:15:18.003 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:18.000+0000 7f3f6e287700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3f680601e0 0x7f3f681066b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:15:18.003 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:18.000+0000 7f3f6e287700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f68106ba0 con 0x7f3f680601e0 2023-12-01T22:15:18.004 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:18.001+0000 7f3f67fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3f680601e0 0x7f3f681066b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:15:18.004 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:18.001+0000 7f3f67fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3f680601e0 0x7f3f681066b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:47058/0 (socket says 172.21.15.73:47058) 2023-12-01T22:15:18.004 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:18.001+0000 7f3f67fff700 1 -- 172.21.15.73:0/345912802 learned_addr learned my addr 172.21.15.73:0/345912802 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:15:18.004 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:18.001+0000 7f3f67fff700 1 -- 172.21.15.73:0/345912802 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f58005040 con 0x7f3f680601e0 2023-12-01T22:15:18.004 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:18.001+0000 7f3f67fff700 1 --2- 172.21.15.73:0/345912802 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3f680601e0 0x7f3f681066b0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7f3f58004dd0 tx=0x7f3f58009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:15:18.004 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:18.001+0000 7f3f64ff9700 1 -- 172.21.15.73:0/345912802 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3f58018070 con 0x7f3f680601e0 2023-12-01T22:15:18.004 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:18.001+0000 7f3f6e287700 1 -- 172.21.15.73:0/345912802 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f68106da0 con 0x7f3f680601e0 2023-12-01T22:15:18.004 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:18.001+0000 7f3f64ff9700 1 -- 172.21.15.73:0/345912802 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3f58013410 con 0x7f3f680601e0 2023-12-01T22:15:18.004 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:18.001+0000 7f3f64ff9700 1 -- 172.21.15.73:0/345912802 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3f5801c3c0 con 0x7f3f680601e0 2023-12-01T22:15:18.004 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:18.001+0000 7f3f6e287700 1 -- 172.21.15.73:0/345912802 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f68107230 con 0x7f3f680601e0 2023-12-01T22:15:18.005 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:18.002+0000 7f3f64ff9700 1 -- 172.21.15.73:0/345912802 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3f5801d420 con 0x7f3f680601e0 2023-12-01T22:15:18.005 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:18.002+0000 7f3f6e287700 1 -- 172.21.15.73:0/345912802 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f68045ad0 con 0x7f3f680601e0 2023-12-01T22:15:18.005 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:18.002+0000 7f3f64ff9700 1 --2- 172.21.15.73:0/345912802 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f3f5002ff60 0x7f3f50032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:15:18.005 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:18.002+0000 7f3f64ff9700 1 -- 172.21.15.73:0/345912802 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3f58041510 con 0x7f3f680601e0 2023-12-01T22:15:18.005 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:18.002+0000 7f3f677fe700 1 --2- 172.21.15.73:0/345912802 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f3f5002ff60 0x7f3f50032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:15:18.005 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:18.003+0000 7f3f677fe700 1 --2- 172.21.15.73:0/345912802 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f3f5002ff60 0x7f3f50032420 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f3f5c0096a0 tx=0x7f3f5c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:15:18.008 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:18.007+0000 7f3f64ff9700 1 -- 172.21.15.73:0/345912802 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3f5801d820 con 0x7f3f680601e0 2023-12-01T22:15:18.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:15:17 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:15:16.656705+0000 mgr.smithi029.loxrfu (mgr.14164) 123 : cluster [DBG] pgmap v116: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:15:18.218 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:18.216+0000 7f3f6e287700 1 -- 172.21.15.73:0/345912802 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3f681080e0 con 0x7f3f680601e0 2023-12-01T22:15:18.218 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:18.216+0000 7f3f64ff9700 1 -- 172.21.15.73:0/345912802 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3f5801b370 con 0x7f3f680601e0 2023-12-01T22:15:18.218 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:15:18.218 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:15:18.220 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:18.218+0000 7f3f6e287700 1 -- 172.21.15.73:0/345912802 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f3f5002ff60 msgr2=0x7f3f50032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:15:18.220 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:18.218+0000 7f3f6e287700 1 --2- 172.21.15.73:0/345912802 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f3f5002ff60 0x7f3f50032420 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f3f5c0096a0 tx=0x7f3f5c006b40).stop 2023-12-01T22:15:18.220 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:18.218+0000 7f3f6e287700 1 -- 172.21.15.73:0/345912802 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3f680601e0 msgr2=0x7f3f681066b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:15:18.220 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:18.218+0000 7f3f6e287700 1 --2- 172.21.15.73:0/345912802 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3f680601e0 0x7f3f681066b0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7f3f58004dd0 tx=0x7f3f58009b30).stop 2023-12-01T22:15:18.220 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:18.218+0000 7f3f6e287700 1 -- 172.21.15.73:0/345912802 shutdown_connections 2023-12-01T22:15:18.220 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:18.218+0000 7f3f6e287700 1 --2- 172.21.15.73:0/345912802 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f3f5002ff60 0x7f3f50032420 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:15:18.220 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:18.218+0000 7f3f6e287700 1 --2- 172.21.15.73:0/345912802 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3f680601e0 0x7f3f681066b0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:15:18.221 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:18.218+0000 7f3f6e287700 1 -- 172.21.15.73:0/345912802 >> 172.21.15.73:0/345912802 conn(0x7f3f680f0b70 msgr2=0x7f3f680f1650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:15:18.221 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:18.218+0000 7f3f6e287700 1 -- 172.21.15.73:0/345912802 shutdown_connections 2023-12-01T22:15:18.221 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:18.218+0000 7f3f6e287700 1 -- 172.21.15.73:0/345912802 wait complete. 2023-12-01T22:15:18.221 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:15:19.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:15:18 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:15:18.217900+0000 mon.smithi029 (mon.0) 203 : audit [DBG] from='client.? 172.21.15.73:0/345912802' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:15:19.597 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:15:19.598 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:15:20.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:15:19 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:15:18.657119+0000 mgr.smithi029.loxrfu (mgr.14164) 124 : cluster [DBG] pgmap v117: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:15:20.804 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:20.800+0000 7fb2854a3700 1 -- 172.21.15.73:0/3365709646 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2800f4600 msgr2=0x7fb2800f4a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:15:20.804 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:20.800+0000 7fb2854a3700 1 --2- 172.21.15.73:0/3365709646 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2800f4600 0x7fb2800f4a20 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7fb270009fb0 tx=0x7fb270009b30).stop 2023-12-01T22:15:20.804 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:20.800+0000 7fb2854a3700 1 -- 172.21.15.73:0/3365709646 shutdown_connections 2023-12-01T22:15:20.804 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:20.800+0000 7fb2854a3700 1 --2- 172.21.15.73:0/3365709646 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2800f4600 0x7fb2800f4a20 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:15:20.804 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:20.800+0000 7fb2854a3700 1 -- 172.21.15.73:0/3365709646 >> 172.21.15.73:0/3365709646 conn(0x7fb2800f0120 msgr2=0x7fb2800f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:15:20.804 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:20.801+0000 7fb2854a3700 1 -- 172.21.15.73:0/3365709646 shutdown_connections 2023-12-01T22:15:20.804 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:20.801+0000 7fb2854a3700 1 -- 172.21.15.73:0/3365709646 wait complete. 2023-12-01T22:15:20.805 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:20.801+0000 7fb2854a3700 1 Processor -- start 2023-12-01T22:15:20.805 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:20.801+0000 7fb2854a3700 1 -- start start 2023-12-01T22:15:20.805 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:20.801+0000 7fb2854a3700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2800f4600 0x7fb280106500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:15:20.805 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:20.801+0000 7fb2854a3700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2801069f0 con 0x7fb2800f4600 2023-12-01T22:15:20.805 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:20.802+0000 7fb27effd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2800f4600 0x7fb280106500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:15:20.805 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:20.802+0000 7fb27effd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2800f4600 0x7fb280106500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:47076/0 (socket says 172.21.15.73:47076) 2023-12-01T22:15:20.805 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:20.802+0000 7fb27effd700 1 -- 172.21.15.73:0/1087988726 learned_addr learned my addr 172.21.15.73:0/1087988726 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:15:20.805 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:20.802+0000 7fb27effd700 1 -- 172.21.15.73:0/1087988726 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb270005040 con 0x7fb2800f4600 2023-12-01T22:15:20.805 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:20.802+0000 7fb27effd700 1 --2- 172.21.15.73:0/1087988726 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2800f4600 0x7fb280106500 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7fb270013510 tx=0x7fb27000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:15:20.805 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:20.802+0000 7fb2677fe700 1 -- 172.21.15.73:0/1087988726 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb270009320 con 0x7fb2800f4600 2023-12-01T22:15:20.805 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:20.802+0000 7fb2854a3700 1 -- 172.21.15.73:0/1087988726 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb280106bf0 con 0x7fb2800f4600 2023-12-01T22:15:20.805 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:20.802+0000 7fb2677fe700 1 -- 172.21.15.73:0/1087988726 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb270013f00 con 0x7fb2800f4600 2023-12-01T22:15:20.805 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:20.802+0000 7fb2854a3700 1 -- 172.21.15.73:0/1087988726 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb280107080 con 0x7fb2800f4600 2023-12-01T22:15:20.805 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:20.802+0000 7fb2677fe700 1 -- 172.21.15.73:0/1087988726 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb270014d90 con 0x7fb2800f4600 2023-12-01T22:15:20.806 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:20.803+0000 7fb2677fe700 1 -- 172.21.15.73:0/1087988726 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb2700263a0 con 0x7fb2800f4600 2023-12-01T22:15:20.806 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:20.803+0000 7fb2854a3700 1 -- 172.21.15.73:0/1087988726 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb280045ad0 con 0x7fb2800f4600 2023-12-01T22:15:20.807 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:20.803+0000 7fb2677fe700 1 --2- 172.21.15.73:0/1087988726 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fb26802ff60 0x7fb268032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:15:20.807 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:20.803+0000 7fb2677fe700 1 -- 172.21.15.73:0/1087988726 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb270040ff0 con 0x7fb2800f4600 2023-12-01T22:15:20.807 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:20.803+0000 7fb27e7fc700 1 --2- 172.21.15.73:0/1087988726 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fb26802ff60 0x7fb268032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:15:20.807 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:20.804+0000 7fb27e7fc700 1 --2- 172.21.15.73:0/1087988726 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fb26802ff60 0x7fb268032420 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7fb2740096a0 tx=0x7fb274006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:15:20.810 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:20.808+0000 7fb2677fe700 1 -- 172.21.15.73:0/1087988726 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb2700095f0 con 0x7fb2800f4600 2023-12-01T22:15:21.013 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:21.012+0000 7fb2854a3700 1 -- 172.21.15.73:0/1087988726 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb280107e90 con 0x7fb2800f4600 2023-12-01T22:15:21.014 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:21.013+0000 7fb2677fe700 1 -- 172.21.15.73:0/1087988726 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb27001d9d0 con 0x7fb2800f4600 2023-12-01T22:15:21.014 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:15:21.014 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:15:21.016 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:21.014+0000 7fb2854a3700 1 -- 172.21.15.73:0/1087988726 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fb26802ff60 msgr2=0x7fb268032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:15:21.016 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:21.014+0000 7fb2854a3700 1 --2- 172.21.15.73:0/1087988726 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fb26802ff60 0x7fb268032420 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7fb2740096a0 tx=0x7fb274006b40).stop 2023-12-01T22:15:21.016 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:21.014+0000 7fb2854a3700 1 -- 172.21.15.73:0/1087988726 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2800f4600 msgr2=0x7fb280106500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:15:21.016 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:21.014+0000 7fb2854a3700 1 --2- 172.21.15.73:0/1087988726 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2800f4600 0x7fb280106500 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7fb270013510 tx=0x7fb27000b040).stop 2023-12-01T22:15:21.016 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:21.014+0000 7fb2854a3700 1 -- 172.21.15.73:0/1087988726 shutdown_connections 2023-12-01T22:15:21.016 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:21.014+0000 7fb2854a3700 1 --2- 172.21.15.73:0/1087988726 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fb26802ff60 0x7fb268032420 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:15:21.016 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:21.014+0000 7fb2854a3700 1 --2- 172.21.15.73:0/1087988726 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2800f4600 0x7fb280106500 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:15:21.016 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:21.014+0000 7fb2854a3700 1 -- 172.21.15.73:0/1087988726 >> 172.21.15.73:0/1087988726 conn(0x7fb2800f0120 msgr2=0x7fb2800f0c10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:15:21.017 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:21.014+0000 7fb2854a3700 1 -- 172.21.15.73:0/1087988726 shutdown_connections 2023-12-01T22:15:21.017 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:21.014+0000 7fb2854a3700 1 -- 172.21.15.73:0/1087988726 wait complete. 2023-12-01T22:15:21.017 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:15:21.777 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:15:21 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:15:20.657419+0000 mgr.smithi029.loxrfu (mgr.14164) 125 : cluster [DBG] pgmap v118: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:15:21.777 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:15:21 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:15:21.013831+0000 mon.smithi029 (mon.0) 204 : audit [DBG] from='client.? 172.21.15.73:0/1087988726' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:15:22.379 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:15:22.379 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:15:23.628 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:23.624+0000 7fa174ff4700 1 -- 172.21.15.73:0/1863865429 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa1700f6510 msgr2=0x7fa1700f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:15:23.628 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:23.624+0000 7fa174ff4700 1 --2- 172.21.15.73:0/1863865429 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa1700f6510 0x7fa1700f6930 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0x7fa164009fb0 tx=0x7fa164009b30).stop 2023-12-01T22:15:23.628 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:23.624+0000 7fa174ff4700 1 -- 172.21.15.73:0/1863865429 shutdown_connections 2023-12-01T22:15:23.628 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:23.624+0000 7fa174ff4700 1 --2- 172.21.15.73:0/1863865429 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa1700f6510 0x7fa1700f6930 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:15:23.628 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:23.624+0000 7fa174ff4700 1 -- 172.21.15.73:0/1863865429 >> 172.21.15.73:0/1863865429 conn(0x7fa1700f1c70 msgr2=0x7fa1700f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:15:23.628 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:23.624+0000 7fa174ff4700 1 -- 172.21.15.73:0/1863865429 shutdown_connections 2023-12-01T22:15:23.628 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:23.624+0000 7fa174ff4700 1 -- 172.21.15.73:0/1863865429 wait complete. 2023-12-01T22:15:23.628 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:23.625+0000 7fa174ff4700 1 Processor -- start 2023-12-01T22:15:23.628 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:23.625+0000 7fa174ff4700 1 -- start start 2023-12-01T22:15:23.628 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:23.625+0000 7fa174ff4700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa1700f6510 0x7fa170106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:15:23.628 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:23.625+0000 7fa174ff4700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa170106c20 con 0x7fa1700f6510 2023-12-01T22:15:23.629 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:23.625+0000 7fa16e59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa1700f6510 0x7fa170106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:15:23.629 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:23.625+0000 7fa16e59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa1700f6510 0x7fa170106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:47082/0 (socket says 172.21.15.73:47082) 2023-12-01T22:15:23.629 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:23.625+0000 7fa16e59c700 1 -- 172.21.15.73:0/266564814 learned_addr learned my addr 172.21.15.73:0/266564814 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:15:23.629 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:23.625+0000 7fa16e59c700 1 -- 172.21.15.73:0/266564814 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa164005040 con 0x7fa1700f6510 2023-12-01T22:15:23.629 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:23.625+0000 7fa16e59c700 1 --2- 172.21.15.73:0/266564814 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa1700f6510 0x7fa170106730 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7fa164013510 tx=0x7fa16400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:15:23.629 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:23.626+0000 7fa15effd700 1 -- 172.21.15.73:0/266564814 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa164009320 con 0x7fa1700f6510 2023-12-01T22:15:23.629 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:23.626+0000 7fa174ff4700 1 -- 172.21.15.73:0/266564814 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa170106e20 con 0x7fa1700f6510 2023-12-01T22:15:23.629 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:23.626+0000 7fa15effd700 1 -- 172.21.15.73:0/266564814 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa164013f00 con 0x7fa1700f6510 2023-12-01T22:15:23.629 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:23.626+0000 7fa15effd700 1 -- 172.21.15.73:0/266564814 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa164014d90 con 0x7fa1700f6510 2023-12-01T22:15:23.629 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:23.626+0000 7fa174ff4700 1 -- 172.21.15.73:0/266564814 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1701072b0 con 0x7fa1700f6510 2023-12-01T22:15:23.630 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:23.626+0000 7fa15effd700 1 -- 172.21.15.73:0/266564814 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa164014420 con 0x7fa1700f6510 2023-12-01T22:15:23.630 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:23.626+0000 7fa174ff4700 1 -- 172.21.15.73:0/266564814 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa170045ad0 con 0x7fa1700f6510 2023-12-01T22:15:23.631 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:23.627+0000 7fa15effd700 1 --2- 172.21.15.73:0/266564814 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fa15802ff60 0x7fa158032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:15:23.631 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:23.627+0000 7fa15effd700 1 -- 172.21.15.73:0/266564814 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa1640410c0 con 0x7fa1700f6510 2023-12-01T22:15:23.631 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:23.627+0000 7fa16dd9b700 1 --2- 172.21.15.73:0/266564814 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fa15802ff60 0x7fa158032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:15:23.631 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:23.627+0000 7fa16dd9b700 1 --2- 172.21.15.73:0/266564814 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fa15802ff60 0x7fa158032420 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7fa1600096a0 tx=0x7fa160006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:15:23.633 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:23.631+0000 7fa15effd700 1 -- 172.21.15.73:0/266564814 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa1640095f0 con 0x7fa1700f6510 2023-12-01T22:15:23.839 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:23.838+0000 7fa174ff4700 1 -- 172.21.15.73:0/266564814 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa1700fbf60 con 0x7fa1700f6510 2023-12-01T22:15:23.840 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:23.839+0000 7fa15effd700 1 -- 172.21.15.73:0/266564814 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa16401db70 con 0x7fa1700f6510 2023-12-01T22:15:23.840 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:15:23.840 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:15:23.842 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:23.840+0000 7fa174ff4700 1 -- 172.21.15.73:0/266564814 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fa15802ff60 msgr2=0x7fa158032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:15:23.842 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:23.840+0000 7fa174ff4700 1 --2- 172.21.15.73:0/266564814 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fa15802ff60 0x7fa158032420 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7fa1600096a0 tx=0x7fa160006b40).stop 2023-12-01T22:15:23.842 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:23.840+0000 7fa174ff4700 1 -- 172.21.15.73:0/266564814 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa1700f6510 msgr2=0x7fa170106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:15:23.842 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:23.840+0000 7fa174ff4700 1 --2- 172.21.15.73:0/266564814 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa1700f6510 0x7fa170106730 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7fa164013510 tx=0x7fa16400b040).stop 2023-12-01T22:15:23.842 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:23.840+0000 7fa174ff4700 1 -- 172.21.15.73:0/266564814 shutdown_connections 2023-12-01T22:15:23.842 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:23.840+0000 7fa174ff4700 1 --2- 172.21.15.73:0/266564814 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fa15802ff60 0x7fa158032420 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:15:23.842 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:23.840+0000 7fa174ff4700 1 --2- 172.21.15.73:0/266564814 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa1700f6510 0x7fa170106730 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:15:23.842 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:23.840+0000 7fa174ff4700 1 -- 172.21.15.73:0/266564814 >> 172.21.15.73:0/266564814 conn(0x7fa1700f1c70 msgr2=0x7fa1700f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:15:23.842 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:23.840+0000 7fa174ff4700 1 -- 172.21.15.73:0/266564814 shutdown_connections 2023-12-01T22:15:23.843 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:23.840+0000 7fa174ff4700 1 -- 172.21.15.73:0/266564814 wait complete. 2023-12-01T22:15:23.843 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:15:24.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:15:23 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:15:22.657692+0000 mgr.smithi029.loxrfu (mgr.14164) 126 : cluster [DBG] pgmap v119: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:15:25.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:15:24 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:15:23.839957+0000 mon.smithi029 (mon.0) 205 : audit [DBG] from='client.? 172.21.15.73:0/266564814' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:15:25.185 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:15:25.185 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:15:26.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:15:25 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:15:24.658038+0000 mgr.smithi029.loxrfu (mgr.14164) 127 : cluster [DBG] pgmap v120: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:15:26.277 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:26.274+0000 7f9696b39700 1 -- 172.21.15.73:0/2075323279 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96900f5410 msgr2=0x7f96900f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:15:26.278 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:26.274+0000 7f9696b39700 1 --2- 172.21.15.73:0/2075323279 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96900f5410 0x7f96900f5830 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0x7f9684009fb0 tx=0x7f9684009b30).stop 2023-12-01T22:15:26.278 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:26.274+0000 7f9696b39700 1 -- 172.21.15.73:0/2075323279 shutdown_connections 2023-12-01T22:15:26.278 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:26.274+0000 7f9696b39700 1 --2- 172.21.15.73:0/2075323279 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96900f5410 0x7f96900f5830 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:15:26.278 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:26.274+0000 7f9696b39700 1 -- 172.21.15.73:0/2075323279 >> 172.21.15.73:0/2075323279 conn(0x7f96900f0b70 msgr2=0x7f96900f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:15:26.278 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:26.274+0000 7f9696b39700 1 -- 172.21.15.73:0/2075323279 shutdown_connections 2023-12-01T22:15:26.278 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:26.274+0000 7f9696b39700 1 -- 172.21.15.73:0/2075323279 wait complete. 2023-12-01T22:15:26.278 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:26.274+0000 7f9696b39700 1 Processor -- start 2023-12-01T22:15:26.278 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:26.274+0000 7f9696b39700 1 -- start start 2023-12-01T22:15:26.278 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:26.275+0000 7f9696b39700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96900f5410 0x7f9690108920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:15:26.279 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:26.275+0000 7f9696b39700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9690108e10 con 0x7f96900f5410 2023-12-01T22:15:26.279 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:26.275+0000 7f96948d5700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96900f5410 0x7f9690108920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:15:26.279 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:26.275+0000 7f96948d5700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96900f5410 0x7f9690108920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:37566/0 (socket says 172.21.15.73:37566) 2023-12-01T22:15:26.279 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:26.275+0000 7f96948d5700 1 -- 172.21.15.73:0/760243365 learned_addr learned my addr 172.21.15.73:0/760243365 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:15:26.279 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:26.275+0000 7f96948d5700 1 -- 172.21.15.73:0/760243365 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9684005040 con 0x7f96900f5410 2023-12-01T22:15:26.279 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:26.275+0000 7f96948d5700 1 --2- 172.21.15.73:0/760243365 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96900f5410 0x7f9690108920 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7f9684013510 tx=0x7f968400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:15:26.279 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:26.276+0000 7f968d7fa700 1 -- 172.21.15.73:0/760243365 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9684009320 con 0x7f96900f5410 2023-12-01T22:15:26.279 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:26.276+0000 7f9696b39700 1 -- 172.21.15.73:0/760243365 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9690109010 con 0x7f96900f5410 2023-12-01T22:15:26.279 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:26.276+0000 7f968d7fa700 1 -- 172.21.15.73:0/760243365 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9684013f00 con 0x7f96900f5410 2023-12-01T22:15:26.279 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:26.276+0000 7f968d7fa700 1 -- 172.21.15.73:0/760243365 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9684014d90 con 0x7f96900f5410 2023-12-01T22:15:26.279 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:26.276+0000 7f9696b39700 1 -- 172.21.15.73:0/760243365 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96901094a0 con 0x7f96900f5410 2023-12-01T22:15:26.280 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:26.276+0000 7f968d7fa700 1 -- 172.21.15.73:0/760243365 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9684014420 con 0x7f96900f5410 2023-12-01T22:15:26.280 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:26.276+0000 7f9696b39700 1 -- 172.21.15.73:0/760243365 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9690045ad0 con 0x7f96900f5410 2023-12-01T22:15:26.280 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:26.277+0000 7f968d7fa700 1 --2- 172.21.15.73:0/760243365 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f968002ff60 0x7f9680032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:15:26.281 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:26.277+0000 7f968d7fa700 1 -- 172.21.15.73:0/760243365 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f96840410c0 con 0x7f96900f5410 2023-12-01T22:15:26.281 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:26.277+0000 7f968ffff700 1 --2- 172.21.15.73:0/760243365 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f968002ff60 0x7f9680032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:15:26.281 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:26.277+0000 7f968ffff700 1 --2- 172.21.15.73:0/760243365 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f968002ff60 0x7f9680032420 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f967c0096a0 tx=0x7f967c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:15:26.283 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:26.282+0000 7f968d7fa700 1 -- 172.21.15.73:0/760243365 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f96840095f0 con 0x7f96900f5410 2023-12-01T22:15:26.490 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:26.488+0000 7f9696b39700 1 -- 172.21.15.73:0/760243365 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f969010a350 con 0x7f96900f5410 2023-12-01T22:15:26.490 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:26.489+0000 7f968d7fa700 1 -- 172.21.15.73:0/760243365 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f968401db70 con 0x7f96900f5410 2023-12-01T22:15:26.490 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:15:26.491 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:15:26.492 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:26.490+0000 7f9696b39700 1 -- 172.21.15.73:0/760243365 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f968002ff60 msgr2=0x7f9680032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:15:26.492 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:26.490+0000 7f9696b39700 1 --2- 172.21.15.73:0/760243365 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f968002ff60 0x7f9680032420 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f967c0096a0 tx=0x7f967c006b40).stop 2023-12-01T22:15:26.493 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:26.490+0000 7f9696b39700 1 -- 172.21.15.73:0/760243365 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96900f5410 msgr2=0x7f9690108920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:15:26.493 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:26.490+0000 7f9696b39700 1 --2- 172.21.15.73:0/760243365 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96900f5410 0x7f9690108920 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7f9684013510 tx=0x7f968400b040).stop 2023-12-01T22:15:26.493 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:26.490+0000 7f9696b39700 1 -- 172.21.15.73:0/760243365 shutdown_connections 2023-12-01T22:15:26.493 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:26.490+0000 7f9696b39700 1 --2- 172.21.15.73:0/760243365 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f968002ff60 0x7f9680032420 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:15:26.493 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:26.490+0000 7f9696b39700 1 --2- 172.21.15.73:0/760243365 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96900f5410 0x7f9690108920 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:15:26.493 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:26.490+0000 7f9696b39700 1 -- 172.21.15.73:0/760243365 >> 172.21.15.73:0/760243365 conn(0x7f96900f0b70 msgr2=0x7f969005fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:15:26.493 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:26.490+0000 7f9696b39700 1 -- 172.21.15.73:0/760243365 shutdown_connections 2023-12-01T22:15:26.493 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:26.490+0000 7f9696b39700 1 -- 172.21.15.73:0/760243365 wait complete. 2023-12-01T22:15:26.493 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:15:27.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:15:26 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:15:26.489911+0000 mon.smithi029 (mon.0) 206 : audit [DBG] from='client.? 172.21.15.73:0/760243365' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:15:27.867 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:15:27.867 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:15:28.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:15:27 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:15:26.658423+0000 mgr.smithi029.loxrfu (mgr.14164) 128 : cluster [DBG] pgmap v121: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:15:29.033 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:29.031+0000 7f7e82a6e700 1 -- 172.21.15.73:0/686707933 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e7c0f5410 msgr2=0x7f7e7c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:15:29.034 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:29.031+0000 7f7e82a6e700 1 --2- 172.21.15.73:0/686707933 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e7c0f5410 0x7f7e7c0f5830 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0x7f7e70009fb0 tx=0x7f7e70009b30).stop 2023-12-01T22:15:29.034 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:29.031+0000 7f7e82a6e700 1 -- 172.21.15.73:0/686707933 shutdown_connections 2023-12-01T22:15:29.034 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:29.031+0000 7f7e82a6e700 1 --2- 172.21.15.73:0/686707933 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e7c0f5410 0x7f7e7c0f5830 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:15:29.034 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:29.031+0000 7f7e82a6e700 1 -- 172.21.15.73:0/686707933 >> 172.21.15.73:0/686707933 conn(0x7f7e7c0f0b70 msgr2=0x7f7e7c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:15:29.034 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:29.031+0000 7f7e82a6e700 1 -- 172.21.15.73:0/686707933 shutdown_connections 2023-12-01T22:15:29.034 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:29.031+0000 7f7e82a6e700 1 -- 172.21.15.73:0/686707933 wait complete. 2023-12-01T22:15:29.034 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:29.031+0000 7f7e82a6e700 1 Processor -- start 2023-12-01T22:15:29.034 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:29.031+0000 7f7e82a6e700 1 -- start start 2023-12-01T22:15:29.034 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:29.032+0000 7f7e82a6e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e7c0f5410 0x7f7e7c0fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:15:29.034 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:29.032+0000 7f7e82a6e700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e7c0fb820 con 0x7f7e7c0f5410 2023-12-01T22:15:29.034 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:29.032+0000 7f7e8080a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e7c0f5410 0x7f7e7c0fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:15:29.034 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:29.032+0000 7f7e8080a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e7c0f5410 0x7f7e7c0fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:37582/0 (socket says 172.21.15.73:37582) 2023-12-01T22:15:29.035 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:29.032+0000 7f7e8080a700 1 -- 172.21.15.73:0/2508051291 learned_addr learned my addr 172.21.15.73:0/2508051291 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:15:29.036 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:29.032+0000 7f7e8080a700 1 -- 172.21.15.73:0/2508051291 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e70005040 con 0x7f7e7c0f5410 2023-12-01T22:15:29.036 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:29.032+0000 7f7e8080a700 1 --2- 172.21.15.73:0/2508051291 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e7c0f5410 0x7f7e7c0fd3a0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7f7e70013510 tx=0x7f7e7000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:15:29.036 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:29.032+0000 7f7e797fa700 1 -- 172.21.15.73:0/2508051291 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7e70009320 con 0x7f7e7c0f5410 2023-12-01T22:15:29.036 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:29.032+0000 7f7e82a6e700 1 -- 172.21.15.73:0/2508051291 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e7c0fb9c0 con 0x7f7e7c0f5410 2023-12-01T22:15:29.036 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:29.032+0000 7f7e797fa700 1 -- 172.21.15.73:0/2508051291 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7e70013f00 con 0x7f7e7c0f5410 2023-12-01T22:15:29.036 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:29.032+0000 7f7e797fa700 1 -- 172.21.15.73:0/2508051291 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7e70014d90 con 0x7f7e7c0f5410 2023-12-01T22:15:29.036 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:29.033+0000 7f7e82a6e700 1 -- 172.21.15.73:0/2508051291 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e7c0fbe50 con 0x7f7e7c0f5410 2023-12-01T22:15:29.036 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:29.033+0000 7f7e797fa700 1 -- 172.21.15.73:0/2508051291 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7e70014420 con 0x7f7e7c0f5410 2023-12-01T22:15:29.036 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:29.033+0000 7f7e82a6e700 1 -- 172.21.15.73:0/2508051291 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e7c059100 con 0x7f7e7c0f5410 2023-12-01T22:15:29.037 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:29.033+0000 7f7e797fa700 1 --2- 172.21.15.73:0/2508051291 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f7e6c02ff60 0x7f7e6c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:15:29.037 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:29.033+0000 7f7e797fa700 1 -- 172.21.15.73:0/2508051291 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7e70041070 con 0x7f7e7c0f5410 2023-12-01T22:15:29.037 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:29.034+0000 7f7e7bfff700 1 --2- 172.21.15.73:0/2508051291 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f7e6c02ff60 0x7f7e6c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:15:29.037 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:29.034+0000 7f7e7bfff700 1 --2- 172.21.15.73:0/2508051291 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f7e6c02ff60 0x7f7e6c032420 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f7e680096a0 tx=0x7f7e68006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:15:29.040 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:29.038+0000 7f7e797fa700 1 -- 172.21.15.73:0/2508051291 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7e700095f0 con 0x7f7e7c0f5410 2023-12-01T22:15:29.247 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:29.245+0000 7f7e82a6e700 1 -- 172.21.15.73:0/2508051291 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7e7c0fae60 con 0x7f7e7c0f5410 2023-12-01T22:15:29.247 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:29.246+0000 7f7e797fa700 1 -- 172.21.15.73:0/2508051291 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7e7001db70 con 0x7f7e7c0f5410 2023-12-01T22:15:29.248 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:15:29.248 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:15:29.249 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:29.247+0000 7f7e82a6e700 1 -- 172.21.15.73:0/2508051291 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f7e6c02ff60 msgr2=0x7f7e6c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:15:29.249 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:29.247+0000 7f7e82a6e700 1 --2- 172.21.15.73:0/2508051291 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f7e6c02ff60 0x7f7e6c032420 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f7e680096a0 tx=0x7f7e68006b40).stop 2023-12-01T22:15:29.249 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:29.247+0000 7f7e82a6e700 1 -- 172.21.15.73:0/2508051291 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e7c0f5410 msgr2=0x7f7e7c0fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:15:29.250 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:29.247+0000 7f7e82a6e700 1 --2- 172.21.15.73:0/2508051291 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e7c0f5410 0x7f7e7c0fd3a0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7f7e70013510 tx=0x7f7e7000b040).stop 2023-12-01T22:15:29.250 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:29.247+0000 7f7e82a6e700 1 -- 172.21.15.73:0/2508051291 shutdown_connections 2023-12-01T22:15:29.250 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:29.247+0000 7f7e82a6e700 1 --2- 172.21.15.73:0/2508051291 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f7e6c02ff60 0x7f7e6c032420 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:15:29.250 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:29.247+0000 7f7e82a6e700 1 --2- 172.21.15.73:0/2508051291 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e7c0f5410 0x7f7e7c0fd3a0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:15:29.250 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:29.247+0000 7f7e82a6e700 1 -- 172.21.15.73:0/2508051291 >> 172.21.15.73:0/2508051291 conn(0x7f7e7c0f0b70 msgr2=0x7f7e7c05fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:15:29.250 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:29.248+0000 7f7e82a6e700 1 -- 172.21.15.73:0/2508051291 shutdown_connections 2023-12-01T22:15:29.250 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:29.248+0000 7f7e82a6e700 1 -- 172.21.15.73:0/2508051291 wait complete. 2023-12-01T22:15:29.250 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:15:30.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:15:29 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:15:28.658807+0000 mgr.smithi029.loxrfu (mgr.14164) 129 : cluster [DBG] pgmap v122: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:15:30.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:15:29 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:15:29.247184+0000 mon.smithi029 (mon.0) 207 : audit [DBG] from='client.? 172.21.15.73:0/2508051291' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:15:30.620 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:15:30.621 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:15:31.777 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:15:31 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:15:30.659203+0000 mgr.smithi029.loxrfu (mgr.14164) 130 : cluster [DBG] pgmap v123: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:15:31.834 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:31.832+0000 7faa95c7b700 1 -- 172.21.15.73:0/569233051 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa90060220 msgr2=0x7faa900f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:15:31.835 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:31.832+0000 7faa95c7b700 1 --2- 172.21.15.73:0/569233051 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa90060220 0x7faa900f9150 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0x7faa84009fb0 tx=0x7faa84009b30).stop 2023-12-01T22:15:31.835 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:31.832+0000 7faa95c7b700 1 -- 172.21.15.73:0/569233051 shutdown_connections 2023-12-01T22:15:31.835 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:31.832+0000 7faa95c7b700 1 --2- 172.21.15.73:0/569233051 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa90060220 0x7faa900f9150 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:15:31.835 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:31.832+0000 7faa95c7b700 1 -- 172.21.15.73:0/569233051 >> 172.21.15.73:0/569233051 conn(0x7faa900f0b70 msgr2=0x7faa900f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:15:31.835 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:31.832+0000 7faa95c7b700 1 -- 172.21.15.73:0/569233051 shutdown_connections 2023-12-01T22:15:31.835 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:31.832+0000 7faa95c7b700 1 -- 172.21.15.73:0/569233051 wait complete. 2023-12-01T22:15:31.835 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:31.833+0000 7faa95c7b700 1 Processor -- start 2023-12-01T22:15:31.835 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:31.833+0000 7faa95c7b700 1 -- start start 2023-12-01T22:15:31.835 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:31.833+0000 7faa95c7b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa90060220 0x7faa901066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:15:31.835 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:31.833+0000 7faa95c7b700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa90106bd0 con 0x7faa90060220 2023-12-01T22:15:31.837 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:31.833+0000 7faa8f7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa90060220 0x7faa901066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:15:31.837 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:31.833+0000 7faa8f7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa90060220 0x7faa901066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:37594/0 (socket says 172.21.15.73:37594) 2023-12-01T22:15:31.837 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:31.833+0000 7faa8f7fe700 1 -- 172.21.15.73:0/2323679521 learned_addr learned my addr 172.21.15.73:0/2323679521 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:15:31.837 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:31.834+0000 7faa8f7fe700 1 -- 172.21.15.73:0/2323679521 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa84005040 con 0x7faa90060220 2023-12-01T22:15:31.837 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:31.834+0000 7faa8f7fe700 1 --2- 172.21.15.73:0/2323679521 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa90060220 0x7faa901066e0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7faa84013510 tx=0x7faa8400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:15:31.837 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:31.834+0000 7faa77fff700 1 -- 172.21.15.73:0/2323679521 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faa84009320 con 0x7faa90060220 2023-12-01T22:15:31.837 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:31.834+0000 7faa95c7b700 1 -- 172.21.15.73:0/2323679521 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa90106dd0 con 0x7faa90060220 2023-12-01T22:15:31.837 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:31.834+0000 7faa77fff700 1 -- 172.21.15.73:0/2323679521 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7faa84013f00 con 0x7faa90060220 2023-12-01T22:15:31.837 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:31.834+0000 7faa77fff700 1 -- 172.21.15.73:0/2323679521 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faa84014dc0 con 0x7faa90060220 2023-12-01T22:15:31.838 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:31.834+0000 7faa95c7b700 1 -- 172.21.15.73:0/2323679521 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa90107260 con 0x7faa90060220 2023-12-01T22:15:31.838 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:31.835+0000 7faa77fff700 1 -- 172.21.15.73:0/2323679521 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7faa8401d3d0 con 0x7faa90060220 2023-12-01T22:15:31.838 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:31.835+0000 7faa77fff700 1 --2- 172.21.15.73:0/2323679521 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7faa7802ff60 0x7faa78032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:15:31.838 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:31.835+0000 7faa95c7b700 1 -- 172.21.15.73:0/2323679521 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa90045ad0 con 0x7faa90060220 2023-12-01T22:15:31.838 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:31.835+0000 7faa77fff700 1 -- 172.21.15.73:0/2323679521 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7faa84041010 con 0x7faa90060220 2023-12-01T22:15:31.838 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:31.835+0000 7faa8effd700 1 --2- 172.21.15.73:0/2323679521 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7faa7802ff60 0x7faa78032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:15:31.838 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:31.835+0000 7faa8effd700 1 --2- 172.21.15.73:0/2323679521 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7faa7802ff60 0x7faa78032420 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7faa800096a0 tx=0x7faa80006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:15:31.841 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:31.840+0000 7faa77fff700 1 -- 172.21.15.73:0/2323679521 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7faa8406d2b0 con 0x7faa90060220 2023-12-01T22:15:32.045 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:32.044+0000 7faa95c7b700 1 -- 172.21.15.73:0/2323679521 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7faa90108110 con 0x7faa90060220 2023-12-01T22:15:32.046 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:32.045+0000 7faa77fff700 1 -- 172.21.15.73:0/2323679521 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7faa84012330 con 0x7faa90060220 2023-12-01T22:15:32.046 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:15:32.046 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:15:32.048 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:32.046+0000 7faa95c7b700 1 -- 172.21.15.73:0/2323679521 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7faa7802ff60 msgr2=0x7faa78032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:15:32.048 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:32.046+0000 7faa95c7b700 1 --2- 172.21.15.73:0/2323679521 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7faa7802ff60 0x7faa78032420 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7faa800096a0 tx=0x7faa80006b40).stop 2023-12-01T22:15:32.048 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:32.046+0000 7faa95c7b700 1 -- 172.21.15.73:0/2323679521 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa90060220 msgr2=0x7faa901066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:15:32.048 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:32.046+0000 7faa95c7b700 1 --2- 172.21.15.73:0/2323679521 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa90060220 0x7faa901066e0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7faa84013510 tx=0x7faa8400b040).stop 2023-12-01T22:15:32.048 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:32.046+0000 7faa95c7b700 1 -- 172.21.15.73:0/2323679521 shutdown_connections 2023-12-01T22:15:32.048 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:32.046+0000 7faa95c7b700 1 --2- 172.21.15.73:0/2323679521 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7faa7802ff60 0x7faa78032420 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:15:32.049 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:32.046+0000 7faa95c7b700 1 --2- 172.21.15.73:0/2323679521 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa90060220 0x7faa901066e0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:15:32.049 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:32.046+0000 7faa95c7b700 1 -- 172.21.15.73:0/2323679521 >> 172.21.15.73:0/2323679521 conn(0x7faa900f0b70 msgr2=0x7faa900f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:15:32.049 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:32.046+0000 7faa95c7b700 1 -- 172.21.15.73:0/2323679521 shutdown_connections 2023-12-01T22:15:32.049 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:32.046+0000 7faa95c7b700 1 -- 172.21.15.73:0/2323679521 wait complete. 2023-12-01T22:15:32.049 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:15:32.776 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:15:32 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:15:32.045938+0000 mon.smithi029 (mon.0) 208 : audit [DBG] from='client.? 172.21.15.73:0/2323679521' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:15:33.431 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:15:33.431 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:15:33.776 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:15:33 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:15:32.659601+0000 mgr.smithi029.loxrfu (mgr.14164) 131 : cluster [DBG] pgmap v124: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:15:34.599 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:34.597+0000 7ff95d817700 1 -- 172.21.15.73:0/3467109945 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9580f47c0 msgr2=0x7ff9580f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:15:34.599 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:34.597+0000 7ff95d817700 1 --2- 172.21.15.73:0/3467109945 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9580f47c0 0x7ff9580f4be0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0x7ff94c009fb0 tx=0x7ff94c009b30).stop 2023-12-01T22:15:34.599 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:34.597+0000 7ff95d817700 1 -- 172.21.15.73:0/3467109945 shutdown_connections 2023-12-01T22:15:34.599 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:34.597+0000 7ff95d817700 1 --2- 172.21.15.73:0/3467109945 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9580f47c0 0x7ff9580f4be0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:15:34.600 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:34.597+0000 7ff95d817700 1 -- 172.21.15.73:0/3467109945 >> 172.21.15.73:0/3467109945 conn(0x7ff9580f02e0 msgr2=0x7ff9580f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:15:34.600 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:34.597+0000 7ff95d817700 1 -- 172.21.15.73:0/3467109945 shutdown_connections 2023-12-01T22:15:34.600 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:34.597+0000 7ff95d817700 1 -- 172.21.15.73:0/3467109945 wait complete. 2023-12-01T22:15:34.602 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:34.598+0000 7ff95d817700 1 Processor -- start 2023-12-01T22:15:34.602 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:34.598+0000 7ff95d817700 1 -- start start 2023-12-01T22:15:34.602 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:34.598+0000 7ff95d817700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9580f47c0 0x7ff958106700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:15:34.602 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:34.598+0000 7ff95d817700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff958106bf0 con 0x7ff9580f47c0 2023-12-01T22:15:34.602 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:34.598+0000 7ff956ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9580f47c0 0x7ff958106700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:15:34.602 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:34.598+0000 7ff956ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9580f47c0 0x7ff958106700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:60608/0 (socket says 172.21.15.73:60608) 2023-12-01T22:15:34.602 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:34.598+0000 7ff956ffd700 1 -- 172.21.15.73:0/1740563352 learned_addr learned my addr 172.21.15.73:0/1740563352 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:15:34.602 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:34.599+0000 7ff956ffd700 1 -- 172.21.15.73:0/1740563352 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff94c005040 con 0x7ff9580f47c0 2023-12-01T22:15:34.602 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:34.599+0000 7ff956ffd700 1 --2- 172.21.15.73:0/1740563352 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9580f47c0 0x7ff958106700 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7ff94c014000 tx=0x7ff94c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:15:34.602 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:34.599+0000 7ff93f7fe700 1 -- 172.21.15.73:0/1740563352 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff94c0095f0 con 0x7ff9580f47c0 2023-12-01T22:15:34.603 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:34.599+0000 7ff95d817700 1 -- 172.21.15.73:0/1740563352 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff958106df0 con 0x7ff9580f47c0 2023-12-01T22:15:34.603 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:34.599+0000 7ff93f7fe700 1 -- 172.21.15.73:0/1740563352 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff94c007e70 con 0x7ff9580f47c0 2023-12-01T22:15:34.603 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:34.599+0000 7ff93f7fe700 1 -- 172.21.15.73:0/1740563352 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff94c014ea0 con 0x7ff9580f47c0 2023-12-01T22:15:34.603 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:34.599+0000 7ff95d817700 1 -- 172.21.15.73:0/1740563352 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff958107280 con 0x7ff9580f47c0 2023-12-01T22:15:34.603 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:34.600+0000 7ff95d817700 1 -- 172.21.15.73:0/1740563352 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff958045ad0 con 0x7ff9580f47c0 2023-12-01T22:15:34.603 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:34.600+0000 7ff93f7fe700 1 -- 172.21.15.73:0/1740563352 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff94c025070 con 0x7ff9580f47c0 2023-12-01T22:15:34.603 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:34.600+0000 7ff93f7fe700 1 --2- 172.21.15.73:0/1740563352 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7ff94002ff60 0x7ff940032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:15:34.603 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:34.600+0000 7ff93f7fe700 1 -- 172.21.15.73:0/1740563352 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff94c0409c0 con 0x7ff9580f47c0 2023-12-01T22:15:34.604 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:34.600+0000 7ff9567fc700 1 --2- 172.21.15.73:0/1740563352 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7ff94002ff60 0x7ff940032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:15:34.604 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:34.601+0000 7ff9567fc700 1 --2- 172.21.15.73:0/1740563352 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7ff94002ff60 0x7ff940032420 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7ff9480096a0 tx=0x7ff948006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:15:34.607 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:34.605+0000 7ff93f7fe700 1 -- 172.21.15.73:0/1740563352 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff94c0079c0 con 0x7ff9580f47c0 2023-12-01T22:15:34.811 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:34.809+0000 7ff95d817700 1 -- 172.21.15.73:0/1740563352 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff9581080d0 con 0x7ff9580f47c0 2023-12-01T22:15:34.812 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:34.810+0000 7ff93f7fe700 1 -- 172.21.15.73:0/1740563352 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff94c00f300 con 0x7ff9580f47c0 2023-12-01T22:15:34.812 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:15:34.812 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:15:34.814 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:34.812+0000 7ff95d817700 1 -- 172.21.15.73:0/1740563352 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7ff94002ff60 msgr2=0x7ff940032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:15:34.814 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:34.812+0000 7ff95d817700 1 --2- 172.21.15.73:0/1740563352 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7ff94002ff60 0x7ff940032420 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7ff9480096a0 tx=0x7ff948006b40).stop 2023-12-01T22:15:34.814 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:34.812+0000 7ff95d817700 1 -- 172.21.15.73:0/1740563352 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9580f47c0 msgr2=0x7ff958106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:15:34.814 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:34.812+0000 7ff95d817700 1 --2- 172.21.15.73:0/1740563352 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9580f47c0 0x7ff958106700 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7ff94c014000 tx=0x7ff94c00b040).stop 2023-12-01T22:15:34.814 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:34.812+0000 7ff95d817700 1 -- 172.21.15.73:0/1740563352 shutdown_connections 2023-12-01T22:15:34.815 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:34.812+0000 7ff95d817700 1 --2- 172.21.15.73:0/1740563352 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7ff94002ff60 0x7ff940032420 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:15:34.815 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:34.812+0000 7ff95d817700 1 --2- 172.21.15.73:0/1740563352 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9580f47c0 0x7ff958106700 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:15:34.815 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:34.812+0000 7ff95d817700 1 -- 172.21.15.73:0/1740563352 >> 172.21.15.73:0/1740563352 conn(0x7ff9580f02e0 msgr2=0x7ff9580f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:15:34.815 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:34.812+0000 7ff95d817700 1 -- 172.21.15.73:0/1740563352 shutdown_connections 2023-12-01T22:15:34.815 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:34.812+0000 7ff95d817700 1 -- 172.21.15.73:0/1740563352 wait complete. 2023-12-01T22:15:34.815 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:15:36.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:15:35 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:15:34.659947+0000 mgr.smithi029.loxrfu (mgr.14164) 132 : cluster [DBG] pgmap v125: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:15:36.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:15:35 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:15:34.811638+0000 mon.smithi029 (mon.0) 209 : audit [DBG] from='client.? 172.21.15.73:0/1740563352' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:15:36.199 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:15:36.200 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:15:37.370 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:37.367+0000 7f976bdba700 1 -- 172.21.15.73:0/3611465416 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97640f5410 msgr2=0x7f97640f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:15:37.370 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:37.367+0000 7f976bdba700 1 --2- 172.21.15.73:0/3611465416 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97640f5410 0x7f97640f5830 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 rx=0x7f9754009fb0 tx=0x7f9754009b30).stop 2023-12-01T22:15:37.370 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:37.367+0000 7f976bdba700 1 -- 172.21.15.73:0/3611465416 shutdown_connections 2023-12-01T22:15:37.370 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:37.367+0000 7f976bdba700 1 --2- 172.21.15.73:0/3611465416 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97640f5410 0x7f97640f5830 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:15:37.370 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:37.367+0000 7f976bdba700 1 -- 172.21.15.73:0/3611465416 >> 172.21.15.73:0/3611465416 conn(0x7f97640f0b70 msgr2=0x7f97640f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:15:37.370 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:37.367+0000 7f976bdba700 1 -- 172.21.15.73:0/3611465416 shutdown_connections 2023-12-01T22:15:37.370 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:37.367+0000 7f976bdba700 1 -- 172.21.15.73:0/3611465416 wait complete. 2023-12-01T22:15:37.370 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:37.368+0000 7f976bdba700 1 Processor -- start 2023-12-01T22:15:37.370 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:37.368+0000 7f976bdba700 1 -- start start 2023-12-01T22:15:37.371 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:37.368+0000 7f976bdba700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97640f5410 0x7f9764106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:15:37.371 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:37.368+0000 7f976bdba700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9764106c10 con 0x7f97640f5410 2023-12-01T22:15:37.371 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:37.368+0000 7f9769b56700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97640f5410 0x7f9764106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:15:37.371 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:37.368+0000 7f9769b56700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97640f5410 0x7f9764106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:60634/0 (socket says 172.21.15.73:60634) 2023-12-01T22:15:37.371 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:37.368+0000 7f9769b56700 1 -- 172.21.15.73:0/446799289 learned_addr learned my addr 172.21.15.73:0/446799289 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:15:37.372 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:37.368+0000 7f9769b56700 1 -- 172.21.15.73:0/446799289 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9754005040 con 0x7f97640f5410 2023-12-01T22:15:37.372 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:37.369+0000 7f9769b56700 1 --2- 172.21.15.73:0/446799289 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97640f5410 0x7f9764106720 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7f9754013510 tx=0x7f975400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:15:37.372 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:37.369+0000 7f975a7fc700 1 -- 172.21.15.73:0/446799289 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9754009320 con 0x7f97640f5410 2023-12-01T22:15:37.372 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:37.369+0000 7f976bdba700 1 -- 172.21.15.73:0/446799289 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9764106e10 con 0x7f97640f5410 2023-12-01T22:15:37.372 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:37.369+0000 7f975a7fc700 1 -- 172.21.15.73:0/446799289 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9754013f00 con 0x7f97640f5410 2023-12-01T22:15:37.372 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:37.369+0000 7f975a7fc700 1 -- 172.21.15.73:0/446799289 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9754014d90 con 0x7f97640f5410 2023-12-01T22:15:37.373 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:37.369+0000 7f976bdba700 1 -- 172.21.15.73:0/446799289 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97641072a0 con 0x7f97640f5410 2023-12-01T22:15:37.373 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:37.370+0000 7f976bdba700 1 -- 172.21.15.73:0/446799289 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9764045ad0 con 0x7f97640f5410 2023-12-01T22:15:37.373 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:37.370+0000 7f975a7fc700 1 -- 172.21.15.73:0/446799289 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f97540095f0 con 0x7f97640f5410 2023-12-01T22:15:37.373 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:37.370+0000 7f975a7fc700 1 --2- 172.21.15.73:0/446799289 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f975002ff60 0x7f9750032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:15:37.373 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:37.370+0000 7f975a7fc700 1 -- 172.21.15.73:0/446799289 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9754040fe0 con 0x7f97640f5410 2023-12-01T22:15:37.373 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:37.370+0000 7f9769355700 1 --2- 172.21.15.73:0/446799289 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f975002ff60 0x7f9750032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:15:37.373 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:37.371+0000 7f9769355700 1 --2- 172.21.15.73:0/446799289 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f975002ff60 0x7f9750032420 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f97600096a0 tx=0x7f9760006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:15:37.376 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:37.375+0000 7f975a7fc700 1 -- 172.21.15.73:0/446799289 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f975401d7f0 con 0x7f97640f5410 2023-12-01T22:15:37.582 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:37.581+0000 7f976bdba700 1 -- 172.21.15.73:0/446799289 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9764108150 con 0x7f97640f5410 2023-12-01T22:15:37.583 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:37.581+0000 7f975a7fc700 1 -- 172.21.15.73:0/446799289 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9754010230 con 0x7f97640f5410 2023-12-01T22:15:37.583 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:15:37.583 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:15:37.584 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:37.583+0000 7f976bdba700 1 -- 172.21.15.73:0/446799289 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f975002ff60 msgr2=0x7f9750032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:15:37.585 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:37.583+0000 7f976bdba700 1 --2- 172.21.15.73:0/446799289 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f975002ff60 0x7f9750032420 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f97600096a0 tx=0x7f9760006b40).stop 2023-12-01T22:15:37.585 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:37.583+0000 7f976bdba700 1 -- 172.21.15.73:0/446799289 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97640f5410 msgr2=0x7f9764106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:15:37.585 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:37.583+0000 7f976bdba700 1 --2- 172.21.15.73:0/446799289 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97640f5410 0x7f9764106720 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7f9754013510 tx=0x7f975400b040).stop 2023-12-01T22:15:37.585 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:37.583+0000 7f976bdba700 1 -- 172.21.15.73:0/446799289 shutdown_connections 2023-12-01T22:15:37.585 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:37.583+0000 7f976bdba700 1 --2- 172.21.15.73:0/446799289 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f975002ff60 0x7f9750032420 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:15:37.585 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:37.583+0000 7f976bdba700 1 --2- 172.21.15.73:0/446799289 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97640f5410 0x7f9764106720 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:15:37.585 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:37.583+0000 7f976bdba700 1 -- 172.21.15.73:0/446799289 >> 172.21.15.73:0/446799289 conn(0x7f97640f0b70 msgr2=0x7f976405fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:15:37.585 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:37.583+0000 7f976bdba700 1 -- 172.21.15.73:0/446799289 shutdown_connections 2023-12-01T22:15:37.585 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:37.583+0000 7f976bdba700 1 -- 172.21.15.73:0/446799289 wait complete. 2023-12-01T22:15:37.585 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:15:38.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:15:37 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:15:36.660272+0000 mgr.smithi029.loxrfu (mgr.14164) 133 : cluster [DBG] pgmap v126: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:15:38.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:15:37 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:15:37.582597+0000 mon.smithi029 (mon.0) 210 : audit [DBG] from='client.? 172.21.15.73:0/446799289' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:15:38.959 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:15:38.959 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:15:40.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:15:39 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:15:38.660656+0000 mgr.smithi029.loxrfu (mgr.14164) 134 : cluster [DBG] pgmap v127: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:15:40.084 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:40.082+0000 7fb399cd9700 1 -- 172.21.15.73:0/1608874793 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3940f5410 msgr2=0x7fb3940f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:15:40.084 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:40.082+0000 7fb399cd9700 1 --2- 172.21.15.73:0/1608874793 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3940f5410 0x7fb3940f5830 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0x7fb384009fb0 tx=0x7fb384009b30).stop 2023-12-01T22:15:40.084 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:40.082+0000 7fb399cd9700 1 -- 172.21.15.73:0/1608874793 shutdown_connections 2023-12-01T22:15:40.084 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:40.082+0000 7fb399cd9700 1 --2- 172.21.15.73:0/1608874793 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3940f5410 0x7fb3940f5830 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:15:40.085 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:40.082+0000 7fb399cd9700 1 -- 172.21.15.73:0/1608874793 >> 172.21.15.73:0/1608874793 conn(0x7fb3940f0b70 msgr2=0x7fb3940f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:15:40.085 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:40.082+0000 7fb399cd9700 1 -- 172.21.15.73:0/1608874793 shutdown_connections 2023-12-01T22:15:40.085 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:40.082+0000 7fb399cd9700 1 -- 172.21.15.73:0/1608874793 wait complete. 2023-12-01T22:15:40.085 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:40.083+0000 7fb399cd9700 1 Processor -- start 2023-12-01T22:15:40.086 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:40.083+0000 7fb399cd9700 1 -- start start 2023-12-01T22:15:40.086 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:40.083+0000 7fb399cd9700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3940f5410 0x7fb394106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:15:40.086 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:40.083+0000 7fb399cd9700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb394106c10 con 0x7fb3940f5410 2023-12-01T22:15:40.086 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:40.083+0000 7fb3937fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3940f5410 0x7fb394106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:15:40.086 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:40.083+0000 7fb3937fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3940f5410 0x7fb394106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:60656/0 (socket says 172.21.15.73:60656) 2023-12-01T22:15:40.086 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:40.083+0000 7fb3937fe700 1 -- 172.21.15.73:0/2296554286 learned_addr learned my addr 172.21.15.73:0/2296554286 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:15:40.087 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:40.084+0000 7fb3937fe700 1 -- 172.21.15.73:0/2296554286 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb384005040 con 0x7fb3940f5410 2023-12-01T22:15:40.087 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:40.084+0000 7fb3937fe700 1 --2- 172.21.15.73:0/2296554286 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3940f5410 0x7fb394106720 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7fb384013510 tx=0x7fb38400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:15:40.087 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:40.084+0000 7fb38bfff700 1 -- 172.21.15.73:0/2296554286 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb384009320 con 0x7fb3940f5410 2023-12-01T22:15:40.087 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:40.084+0000 7fb399cd9700 1 -- 172.21.15.73:0/2296554286 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb394106e10 con 0x7fb3940f5410 2023-12-01T22:15:40.087 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:40.084+0000 7fb38bfff700 1 -- 172.21.15.73:0/2296554286 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb384013f00 con 0x7fb3940f5410 2023-12-01T22:15:40.087 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:40.084+0000 7fb38bfff700 1 -- 172.21.15.73:0/2296554286 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb384014d90 con 0x7fb3940f5410 2023-12-01T22:15:40.087 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:40.084+0000 7fb399cd9700 1 -- 172.21.15.73:0/2296554286 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3941072a0 con 0x7fb3940f5410 2023-12-01T22:15:40.087 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:40.085+0000 7fb38bfff700 1 -- 172.21.15.73:0/2296554286 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb3840263a0 con 0x7fb3940f5410 2023-12-01T22:15:40.088 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:40.085+0000 7fb38bfff700 1 --2- 172.21.15.73:0/2296554286 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fb38002ff60 0x7fb380032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:15:40.088 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:40.085+0000 7fb38bfff700 1 -- 172.21.15.73:0/2296554286 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb3840410b0 con 0x7fb3940f5410 2023-12-01T22:15:40.088 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:40.085+0000 7fb399cd9700 1 -- 172.21.15.73:0/2296554286 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb394045ad0 con 0x7fb3940f5410 2023-12-01T22:15:40.088 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:40.085+0000 7fb392ffd700 1 --2- 172.21.15.73:0/2296554286 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fb38002ff60 0x7fb380032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:15:40.088 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:40.086+0000 7fb392ffd700 1 --2- 172.21.15.73:0/2296554286 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fb38002ff60 0x7fb380032420 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7fb37c0096a0 tx=0x7fb37c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:15:40.091 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:40.090+0000 7fb38bfff700 1 -- 172.21.15.73:0/2296554286 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb384012920 con 0x7fb3940f5410 2023-12-01T22:15:40.300 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:40.298+0000 7fb399cd9700 1 -- 172.21.15.73:0/2296554286 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb394107fc0 con 0x7fb3940f5410 2023-12-01T22:15:40.300 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:40.299+0000 7fb38bfff700 1 -- 172.21.15.73:0/2296554286 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb38402b020 con 0x7fb3940f5410 2023-12-01T22:15:40.300 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:15:40.301 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:15:40.302 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:40.300+0000 7fb399cd9700 1 -- 172.21.15.73:0/2296554286 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fb38002ff60 msgr2=0x7fb380032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:15:40.302 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:40.300+0000 7fb399cd9700 1 --2- 172.21.15.73:0/2296554286 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fb38002ff60 0x7fb380032420 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7fb37c0096a0 tx=0x7fb37c006b40).stop 2023-12-01T22:15:40.302 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:40.300+0000 7fb399cd9700 1 -- 172.21.15.73:0/2296554286 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3940f5410 msgr2=0x7fb394106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:15:40.302 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:40.300+0000 7fb399cd9700 1 --2- 172.21.15.73:0/2296554286 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3940f5410 0x7fb394106720 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7fb384013510 tx=0x7fb38400b040).stop 2023-12-01T22:15:40.303 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:40.300+0000 7fb399cd9700 1 -- 172.21.15.73:0/2296554286 shutdown_connections 2023-12-01T22:15:40.303 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:40.300+0000 7fb399cd9700 1 --2- 172.21.15.73:0/2296554286 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fb38002ff60 0x7fb380032420 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:15:40.303 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:40.300+0000 7fb399cd9700 1 --2- 172.21.15.73:0/2296554286 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3940f5410 0x7fb394106720 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:15:40.303 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:40.300+0000 7fb399cd9700 1 -- 172.21.15.73:0/2296554286 >> 172.21.15.73:0/2296554286 conn(0x7fb3940f0b70 msgr2=0x7fb39405fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:15:40.303 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:40.300+0000 7fb399cd9700 1 -- 172.21.15.73:0/2296554286 shutdown_connections 2023-12-01T22:15:40.303 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:40.301+0000 7fb399cd9700 1 -- 172.21.15.73:0/2296554286 wait complete. 2023-12-01T22:15:40.303 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:15:41.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:15:40 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:15:40.300191+0000 mon.smithi029 (mon.0) 211 : audit [DBG] from='client.? 172.21.15.73:0/2296554286' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:15:41.700 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:15:41.700 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:15:42.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:15:41 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:15:40.661038+0000 mgr.smithi029.loxrfu (mgr.14164) 135 : cluster [DBG] pgmap v128: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:15:42.818 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:42.813+0000 7f32782d0700 1 -- 172.21.15.73:0/2041573143 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32700f5410 msgr2=0x7f32700f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:15:42.818 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:42.813+0000 7f32782d0700 1 --2- 172.21.15.73:0/2041573143 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32700f5410 0x7f32700f5830 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 rx=0x7f3260009fb0 tx=0x7f3260009b30).stop 2023-12-01T22:15:42.818 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:42.813+0000 7f32782d0700 1 -- 172.21.15.73:0/2041573143 shutdown_connections 2023-12-01T22:15:42.818 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:42.813+0000 7f32782d0700 1 --2- 172.21.15.73:0/2041573143 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32700f5410 0x7f32700f5830 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:15:42.818 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:42.813+0000 7f32782d0700 1 -- 172.21.15.73:0/2041573143 >> 172.21.15.73:0/2041573143 conn(0x7f32700f0b70 msgr2=0x7f32700f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:15:42.818 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:42.813+0000 7f32782d0700 1 -- 172.21.15.73:0/2041573143 shutdown_connections 2023-12-01T22:15:42.818 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:42.814+0000 7f32782d0700 1 -- 172.21.15.73:0/2041573143 wait complete. 2023-12-01T22:15:42.819 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:42.814+0000 7f32782d0700 1 Processor -- start 2023-12-01T22:15:42.819 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:42.814+0000 7f32782d0700 1 -- start start 2023-12-01T22:15:42.819 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:42.814+0000 7f32782d0700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32700f5410 0x7f3270109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:15:42.819 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:42.814+0000 7f32782d0700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3270060240 con 0x7f32700f5410 2023-12-01T22:15:42.819 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:42.814+0000 7f327606c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32700f5410 0x7f3270109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:15:42.819 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:42.814+0000 7f327606c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32700f5410 0x7f3270109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:60674/0 (socket says 172.21.15.73:60674) 2023-12-01T22:15:42.819 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:42.814+0000 7f327606c700 1 -- 172.21.15.73:0/814929120 learned_addr learned my addr 172.21.15.73:0/814929120 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:15:42.819 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:42.815+0000 7f327606c700 1 -- 172.21.15.73:0/814929120 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3260005040 con 0x7f32700f5410 2023-12-01T22:15:42.819 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:42.815+0000 7f327606c700 1 --2- 172.21.15.73:0/814929120 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32700f5410 0x7f3270109430 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7f3260013510 tx=0x7f326000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:15:42.819 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:42.815+0000 7f3266ffd700 1 -- 172.21.15.73:0/814929120 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3260009320 con 0x7f32700f5410 2023-12-01T22:15:42.819 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:42.815+0000 7f32782d0700 1 -- 172.21.15.73:0/814929120 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3270060440 con 0x7f32700f5410 2023-12-01T22:15:42.820 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:42.815+0000 7f3266ffd700 1 -- 172.21.15.73:0/814929120 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3260013f00 con 0x7f32700f5410 2023-12-01T22:15:42.820 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:42.815+0000 7f3266ffd700 1 -- 172.21.15.73:0/814929120 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3260014d90 con 0x7f32700f5410 2023-12-01T22:15:42.820 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:42.815+0000 7f32782d0700 1 -- 172.21.15.73:0/814929120 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3270109ae0 con 0x7f32700f5410 2023-12-01T22:15:42.820 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:42.816+0000 7f3266ffd700 1 -- 172.21.15.73:0/814929120 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f32600095f0 con 0x7f32700f5410 2023-12-01T22:15:42.820 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:42.816+0000 7f32782d0700 1 -- 172.21.15.73:0/814929120 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3270045ad0 con 0x7f32700f5410 2023-12-01T22:15:42.820 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:42.816+0000 7f3266ffd700 1 --2- 172.21.15.73:0/814929120 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f325c02ff60 0x7f325c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:15:42.820 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:42.816+0000 7f3266ffd700 1 -- 172.21.15.73:0/814929120 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3260041000 con 0x7f32700f5410 2023-12-01T22:15:42.820 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:42.816+0000 7f327586b700 1 --2- 172.21.15.73:0/814929120 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f325c02ff60 0x7f325c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:15:42.820 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:42.817+0000 7f327586b700 1 --2- 172.21.15.73:0/814929120 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f325c02ff60 0x7f325c032420 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f326c0096a0 tx=0x7f326c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:15:42.823 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:42.822+0000 7f3266ffd700 1 -- 172.21.15.73:0/814929120 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3260014420 con 0x7f32700f5410 2023-12-01T22:15:43.030 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:43.028+0000 7f32782d0700 1 -- 172.21.15.73:0/814929120 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f32700fae60 con 0x7f32700f5410 2023-12-01T22:15:43.030 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:43.029+0000 7f3266ffd700 1 -- 172.21.15.73:0/814929120 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3260010230 con 0x7f32700f5410 2023-12-01T22:15:43.031 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:15:43.031 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:15:43.032 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:43.030+0000 7f32782d0700 1 -- 172.21.15.73:0/814929120 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f325c02ff60 msgr2=0x7f325c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:15:43.032 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:43.030+0000 7f32782d0700 1 --2- 172.21.15.73:0/814929120 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f325c02ff60 0x7f325c032420 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f326c0096a0 tx=0x7f326c006b40).stop 2023-12-01T22:15:43.033 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:43.030+0000 7f32782d0700 1 -- 172.21.15.73:0/814929120 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32700f5410 msgr2=0x7f3270109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:15:43.033 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:43.030+0000 7f32782d0700 1 --2- 172.21.15.73:0/814929120 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32700f5410 0x7f3270109430 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7f3260013510 tx=0x7f326000b040).stop 2023-12-01T22:15:43.033 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:43.030+0000 7f32782d0700 1 -- 172.21.15.73:0/814929120 shutdown_connections 2023-12-01T22:15:43.033 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:43.030+0000 7f32782d0700 1 --2- 172.21.15.73:0/814929120 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f325c02ff60 0x7f325c032420 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:15:43.033 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:43.030+0000 7f32782d0700 1 --2- 172.21.15.73:0/814929120 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32700f5410 0x7f3270109430 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:15:43.033 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:43.030+0000 7f32782d0700 1 -- 172.21.15.73:0/814929120 >> 172.21.15.73:0/814929120 conn(0x7f32700f0b70 msgr2=0x7f32700f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:15:43.033 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:43.030+0000 7f32782d0700 1 -- 172.21.15.73:0/814929120 shutdown_connections 2023-12-01T22:15:43.033 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:43.030+0000 7f32782d0700 1 -- 172.21.15.73:0/814929120 wait complete. 2023-12-01T22:15:43.033 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:15:44.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:15:43 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:15:42.661296+0000 mgr.smithi029.loxrfu (mgr.14164) 136 : cluster [DBG] pgmap v129: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:15:44.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:15:43 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:15:43.030044+0000 mon.smithi029 (mon.0) 212 : audit [DBG] from='client.? 172.21.15.73:0/814929120' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:15:44.465 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:15:44.465 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:15:45.628 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:45.626+0000 7fd50b2c1700 1 -- 172.21.15.73:0/3127469458 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd5040f5410 msgr2=0x7fd5040f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:15:45.628 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:45.626+0000 7fd50b2c1700 1 --2- 172.21.15.73:0/3127469458 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd5040f5410 0x7fd5040f5830 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 rx=0x7fd4f4009fb0 tx=0x7fd4f4009b30).stop 2023-12-01T22:15:45.628 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:45.626+0000 7fd50b2c1700 1 -- 172.21.15.73:0/3127469458 shutdown_connections 2023-12-01T22:15:45.628 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:45.626+0000 7fd50b2c1700 1 --2- 172.21.15.73:0/3127469458 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd5040f5410 0x7fd5040f5830 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:15:45.628 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:45.626+0000 7fd50b2c1700 1 -- 172.21.15.73:0/3127469458 >> 172.21.15.73:0/3127469458 conn(0x7fd5040f0b70 msgr2=0x7fd5040f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:15:45.628 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:45.626+0000 7fd50b2c1700 1 -- 172.21.15.73:0/3127469458 shutdown_connections 2023-12-01T22:15:45.629 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:45.626+0000 7fd50b2c1700 1 -- 172.21.15.73:0/3127469458 wait complete. 2023-12-01T22:15:45.629 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:45.626+0000 7fd50b2c1700 1 Processor -- start 2023-12-01T22:15:45.629 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:45.626+0000 7fd50b2c1700 1 -- start start 2023-12-01T22:15:45.629 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:45.626+0000 7fd50b2c1700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd5040f5410 0x7fd504109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:15:45.629 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:45.626+0000 7fd50b2c1700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd504060240 con 0x7fd5040f5410 2023-12-01T22:15:45.629 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:45.627+0000 7fd50905d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd5040f5410 0x7fd504109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:15:45.629 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:45.627+0000 7fd50905d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd5040f5410 0x7fd504109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:59854/0 (socket says 172.21.15.73:59854) 2023-12-01T22:15:45.629 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:45.627+0000 7fd50905d700 1 -- 172.21.15.73:0/2077596576 learned_addr learned my addr 172.21.15.73:0/2077596576 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:15:45.629 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:45.627+0000 7fd50905d700 1 -- 172.21.15.73:0/2077596576 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4f4005040 con 0x7fd5040f5410 2023-12-01T22:15:45.629 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:45.627+0000 7fd50905d700 1 --2- 172.21.15.73:0/2077596576 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd5040f5410 0x7fd504109430 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7fd4f4014000 tx=0x7fd4f400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:15:45.629 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:45.627+0000 7fd4f9ffb700 1 -- 172.21.15.73:0/2077596576 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd4f40095f0 con 0x7fd5040f5410 2023-12-01T22:15:45.629 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:45.627+0000 7fd50b2c1700 1 -- 172.21.15.73:0/2077596576 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd504060440 con 0x7fd5040f5410 2023-12-01T22:15:45.629 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:45.627+0000 7fd4f9ffb700 1 -- 172.21.15.73:0/2077596576 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd4f4007e70 con 0x7fd5040f5410 2023-12-01T22:15:45.629 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:45.627+0000 7fd4f9ffb700 1 -- 172.21.15.73:0/2077596576 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd4f4014ea0 con 0x7fd5040f5410 2023-12-01T22:15:45.630 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:45.627+0000 7fd50b2c1700 1 -- 172.21.15.73:0/2077596576 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd504109ae0 con 0x7fd5040f5410 2023-12-01T22:15:45.630 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:45.628+0000 7fd4f9ffb700 1 -- 172.21.15.73:0/2077596576 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd4f4025070 con 0x7fd5040f5410 2023-12-01T22:15:45.630 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:45.628+0000 7fd4f9ffb700 1 --2- 172.21.15.73:0/2077596576 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fd4f002ff60 0x7fd4f0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:15:45.631 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:45.628+0000 7fd4f9ffb700 1 -- 172.21.15.73:0/2077596576 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd4f4040a60 con 0x7fd5040f5410 2023-12-01T22:15:45.631 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:45.628+0000 7fd50b2c1700 1 -- 172.21.15.73:0/2077596576 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd504045ad0 con 0x7fd5040f5410 2023-12-01T22:15:45.631 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:45.628+0000 7fd50885c700 1 --2- 172.21.15.73:0/2077596576 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fd4f002ff60 0x7fd4f0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:15:45.631 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:45.629+0000 7fd50885c700 1 --2- 172.21.15.73:0/2077596576 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fd4f002ff60 0x7fd4f0032420 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7fd5000096a0 tx=0x7fd500006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:15:45.634 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:45.633+0000 7fd4f9ffb700 1 -- 172.21.15.73:0/2077596576 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd4f4026350 con 0x7fd5040f5410 2023-12-01T22:15:45.841 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:45.841+0000 7fd50b2c1700 1 -- 172.21.15.73:0/2077596576 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd50410a180 con 0x7fd5040f5410 2023-12-01T22:15:45.842 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:45.841+0000 7fd4f9ffb700 1 -- 172.21.15.73:0/2077596576 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd4f4012330 con 0x7fd5040f5410 2023-12-01T22:15:45.842 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:15:45.842 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:15:45.843 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:45.843+0000 7fd50b2c1700 1 -- 172.21.15.73:0/2077596576 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fd4f002ff60 msgr2=0x7fd4f0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:15:45.844 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:45.843+0000 7fd50b2c1700 1 --2- 172.21.15.73:0/2077596576 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fd4f002ff60 0x7fd4f0032420 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7fd5000096a0 tx=0x7fd500006b40).stop 2023-12-01T22:15:45.844 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:45.843+0000 7fd50b2c1700 1 -- 172.21.15.73:0/2077596576 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd5040f5410 msgr2=0x7fd504109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:15:45.844 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:45.843+0000 7fd50b2c1700 1 --2- 172.21.15.73:0/2077596576 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd5040f5410 0x7fd504109430 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7fd4f4014000 tx=0x7fd4f400b040).stop 2023-12-01T22:15:45.844 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:45.843+0000 7fd50b2c1700 1 -- 172.21.15.73:0/2077596576 shutdown_connections 2023-12-01T22:15:45.844 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:45.843+0000 7fd50b2c1700 1 --2- 172.21.15.73:0/2077596576 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fd4f002ff60 0x7fd4f0032420 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:15:45.844 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:45.843+0000 7fd50b2c1700 1 --2- 172.21.15.73:0/2077596576 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd5040f5410 0x7fd504109430 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:15:45.844 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:45.843+0000 7fd50b2c1700 1 -- 172.21.15.73:0/2077596576 >> 172.21.15.73:0/2077596576 conn(0x7fd5040f0b70 msgr2=0x7fd5040f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:15:45.844 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:45.843+0000 7fd50b2c1700 1 -- 172.21.15.73:0/2077596576 shutdown_connections 2023-12-01T22:15:45.844 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:45.843+0000 7fd50b2c1700 1 -- 172.21.15.73:0/2077596576 wait complete. 2023-12-01T22:15:45.844 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:15:46.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:15:45 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:15:44.661551+0000 mgr.smithi029.loxrfu (mgr.14164) 137 : cluster [DBG] pgmap v130: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:15:47.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:15:46 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:15:45.841664+0000 mon.smithi029 (mon.0) 213 : audit [DBG] from='client.? 172.21.15.73:0/2077596576' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:15:47.259 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:15:47.259 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:15:48.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:15:47 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:15:46.661875+0000 mgr.smithi029.loxrfu (mgr.14164) 138 : cluster [DBG] pgmap v131: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:15:48.392 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:48.391+0000 7f17ccac7700 1 -- 172.21.15.73:0/1365566199 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17c80f5410 msgr2=0x7f17c80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:15:48.392 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:48.391+0000 7f17ccac7700 1 --2- 172.21.15.73:0/1365566199 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17c80f5410 0x7f17c80f5830 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 rx=0x7f17b0009fb0 tx=0x7f17b0009b30).stop 2023-12-01T22:15:48.392 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:48.391+0000 7f17ccac7700 1 -- 172.21.15.73:0/1365566199 shutdown_connections 2023-12-01T22:15:48.393 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:48.391+0000 7f17ccac7700 1 --2- 172.21.15.73:0/1365566199 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17c80f5410 0x7f17c80f5830 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:15:48.393 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:48.391+0000 7f17ccac7700 1 -- 172.21.15.73:0/1365566199 >> 172.21.15.73:0/1365566199 conn(0x7f17c80f0b70 msgr2=0x7f17c80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:15:48.393 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:48.391+0000 7f17ccac7700 1 -- 172.21.15.73:0/1365566199 shutdown_connections 2023-12-01T22:15:48.393 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:48.391+0000 7f17ccac7700 1 -- 172.21.15.73:0/1365566199 wait complete. 2023-12-01T22:15:48.393 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:48.391+0000 7f17ccac7700 1 Processor -- start 2023-12-01T22:15:48.393 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:48.391+0000 7f17ccac7700 1 -- start start 2023-12-01T22:15:48.393 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:48.392+0000 7f17ccac7700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17c80f5410 0x7f17c81066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:15:48.393 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:48.392+0000 7f17ccac7700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17c8106bd0 con 0x7f17c80f5410 2023-12-01T22:15:48.395 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:48.392+0000 7f17c659c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17c80f5410 0x7f17c81066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:15:48.395 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:48.392+0000 7f17c659c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17c80f5410 0x7f17c81066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:59862/0 (socket says 172.21.15.73:59862) 2023-12-01T22:15:48.395 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:48.392+0000 7f17c659c700 1 -- 172.21.15.73:0/1089259324 learned_addr learned my addr 172.21.15.73:0/1089259324 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:15:48.395 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:48.392+0000 7f17c659c700 1 -- 172.21.15.73:0/1089259324 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17b0005040 con 0x7f17c80f5410 2023-12-01T22:15:48.395 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:48.392+0000 7f17c659c700 1 --2- 172.21.15.73:0/1089259324 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17c80f5410 0x7f17c81066e0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7f17b0013510 tx=0x7f17b000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:15:48.395 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:48.393+0000 7f17beffd700 1 -- 172.21.15.73:0/1089259324 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f17b0009320 con 0x7f17c80f5410 2023-12-01T22:15:48.395 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:48.393+0000 7f17ccac7700 1 -- 172.21.15.73:0/1089259324 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17c8106dd0 con 0x7f17c80f5410 2023-12-01T22:15:48.395 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:48.393+0000 7f17beffd700 1 -- 172.21.15.73:0/1089259324 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f17b0013f00 con 0x7f17c80f5410 2023-12-01T22:15:48.395 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:48.393+0000 7f17beffd700 1 -- 172.21.15.73:0/1089259324 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f17b0014d90 con 0x7f17c80f5410 2023-12-01T22:15:48.395 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:48.393+0000 7f17ccac7700 1 -- 172.21.15.73:0/1089259324 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17c8107260 con 0x7f17c80f5410 2023-12-01T22:15:48.395 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:48.393+0000 7f17beffd700 1 -- 172.21.15.73:0/1089259324 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f17b00263a0 con 0x7f17c80f5410 2023-12-01T22:15:48.395 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:48.394+0000 7f17beffd700 1 --2- 172.21.15.73:0/1089259324 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f17b402ff60 0x7f17b4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:15:48.395 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:48.394+0000 7f17beffd700 1 -- 172.21.15.73:0/1089259324 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f17b0041080 con 0x7f17c80f5410 2023-12-01T22:15:48.396 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:48.394+0000 7f17ccac7700 1 -- 172.21.15.73:0/1089259324 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17c8045ad0 con 0x7f17c80f5410 2023-12-01T22:15:48.396 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:48.394+0000 7f17c5d9b700 1 --2- 172.21.15.73:0/1089259324 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f17b402ff60 0x7f17b4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:15:48.396 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:48.394+0000 7f17c5d9b700 1 --2- 172.21.15.73:0/1089259324 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f17b402ff60 0x7f17b4032420 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f17b80096a0 tx=0x7f17b8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:15:48.399 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:48.399+0000 7f17beffd700 1 -- 172.21.15.73:0/1089259324 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f17b0012920 con 0x7f17c80f5410 2023-12-01T22:15:48.607 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:48.606+0000 7f17ccac7700 1 -- 172.21.15.73:0/1089259324 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f17c80faf50 con 0x7f17c80f5410 2023-12-01T22:15:48.607 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:48.606+0000 7f17beffd700 1 -- 172.21.15.73:0/1089259324 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f17b002b020 con 0x7f17c80f5410 2023-12-01T22:15:48.607 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:15:48.607 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:15:48.609 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:48.608+0000 7f17ccac7700 1 -- 172.21.15.73:0/1089259324 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f17b402ff60 msgr2=0x7f17b4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:15:48.609 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:48.608+0000 7f17ccac7700 1 --2- 172.21.15.73:0/1089259324 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f17b402ff60 0x7f17b4032420 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f17b80096a0 tx=0x7f17b8006b40).stop 2023-12-01T22:15:48.609 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:48.608+0000 7f17ccac7700 1 -- 172.21.15.73:0/1089259324 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17c80f5410 msgr2=0x7f17c81066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:15:48.609 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:48.608+0000 7f17ccac7700 1 --2- 172.21.15.73:0/1089259324 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17c80f5410 0x7f17c81066e0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7f17b0013510 tx=0x7f17b000b040).stop 2023-12-01T22:15:48.609 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:48.608+0000 7f17ccac7700 1 -- 172.21.15.73:0/1089259324 shutdown_connections 2023-12-01T22:15:48.609 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:48.608+0000 7f17ccac7700 1 --2- 172.21.15.73:0/1089259324 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f17b402ff60 0x7f17b4032420 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:15:48.609 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:48.608+0000 7f17ccac7700 1 --2- 172.21.15.73:0/1089259324 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17c80f5410 0x7f17c81066e0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:15:48.609 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:48.608+0000 7f17ccac7700 1 -- 172.21.15.73:0/1089259324 >> 172.21.15.73:0/1089259324 conn(0x7f17c80f0b70 msgr2=0x7f17c805fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:15:48.609 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:48.608+0000 7f17ccac7700 1 -- 172.21.15.73:0/1089259324 shutdown_connections 2023-12-01T22:15:48.609 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:48.608+0000 7f17ccac7700 1 -- 172.21.15.73:0/1089259324 wait complete. 2023-12-01T22:15:48.610 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:15:49.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:15:48 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:15:48.606967+0000 mon.smithi029 (mon.0) 214 : audit [DBG] from='client.? 172.21.15.73:0/1089259324' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:15:49.990 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:15:49.991 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:15:50.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:15:49 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:15:48.662127+0000 mgr.smithi029.loxrfu (mgr.14164) 139 : cluster [DBG] pgmap v132: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:15:51.126 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:51.123+0000 7fe3f4d7e700 1 -- 172.21.15.73:0/969933267 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3f00f5410 msgr2=0x7fe3f00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:15:51.126 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:51.123+0000 7fe3f4d7e700 1 --2- 172.21.15.73:0/969933267 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3f00f5410 0x7fe3f00f5830 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0x7fe3d8009fb0 tx=0x7fe3d8009b30).stop 2023-12-01T22:15:51.126 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:51.124+0000 7fe3f4d7e700 1 -- 172.21.15.73:0/969933267 shutdown_connections 2023-12-01T22:15:51.126 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:51.124+0000 7fe3f4d7e700 1 --2- 172.21.15.73:0/969933267 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3f00f5410 0x7fe3f00f5830 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:15:51.126 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:51.124+0000 7fe3f4d7e700 1 -- 172.21.15.73:0/969933267 >> 172.21.15.73:0/969933267 conn(0x7fe3f00f0b70 msgr2=0x7fe3f00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:15:51.126 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:51.124+0000 7fe3f4d7e700 1 -- 172.21.15.73:0/969933267 shutdown_connections 2023-12-01T22:15:51.126 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:51.124+0000 7fe3f4d7e700 1 -- 172.21.15.73:0/969933267 wait complete. 2023-12-01T22:15:51.126 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:51.124+0000 7fe3f4d7e700 1 Processor -- start 2023-12-01T22:15:51.127 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:51.124+0000 7fe3f4d7e700 1 -- start start 2023-12-01T22:15:51.127 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:51.125+0000 7fe3f4d7e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3f00f5410 0x7fe3f0105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:15:51.127 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:51.125+0000 7fe3f4d7e700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3f0105b20 con 0x7fe3f00f5410 2023-12-01T22:15:51.127 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:51.125+0000 7fe3ee59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3f00f5410 0x7fe3f0105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:15:51.127 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:51.125+0000 7fe3ee59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3f00f5410 0x7fe3f0105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:59886/0 (socket says 172.21.15.73:59886) 2023-12-01T22:15:51.127 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:51.125+0000 7fe3ee59c700 1 -- 172.21.15.73:0/2546022816 learned_addr learned my addr 172.21.15.73:0/2546022816 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:15:51.127 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:51.125+0000 7fe3ee59c700 1 -- 172.21.15.73:0/2546022816 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3d8005040 con 0x7fe3f00f5410 2023-12-01T22:15:51.127 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:51.125+0000 7fe3ee59c700 1 --2- 172.21.15.73:0/2546022816 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3f00f5410 0x7fe3f0105630 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7fe3d8013510 tx=0x7fe3d800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:15:51.127 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:51.125+0000 7fe3e6ffd700 1 -- 172.21.15.73:0/2546022816 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe3d8009320 con 0x7fe3f00f5410 2023-12-01T22:15:51.127 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:51.125+0000 7fe3f4d7e700 1 -- 172.21.15.73:0/2546022816 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3f0105d20 con 0x7fe3f00f5410 2023-12-01T22:15:51.127 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:51.125+0000 7fe3e6ffd700 1 -- 172.21.15.73:0/2546022816 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe3d8013f00 con 0x7fe3f00f5410 2023-12-01T22:15:51.127 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:51.125+0000 7fe3e6ffd700 1 -- 172.21.15.73:0/2546022816 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe3d8014d90 con 0x7fe3f00f5410 2023-12-01T22:15:51.127 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:51.125+0000 7fe3f4d7e700 1 -- 172.21.15.73:0/2546022816 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3f01061b0 con 0x7fe3f00f5410 2023-12-01T22:15:51.128 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:51.126+0000 7fe3e6ffd700 1 -- 172.21.15.73:0/2546022816 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe3d8014420 con 0x7fe3f00f5410 2023-12-01T22:15:51.128 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:51.126+0000 7fe3f4d7e700 1 -- 172.21.15.73:0/2546022816 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3f0045ad0 con 0x7fe3f00f5410 2023-12-01T22:15:51.128 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:51.126+0000 7fe3e6ffd700 1 --2- 172.21.15.73:0/2546022816 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fe3dc02ff60 0x7fe3dc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:15:51.128 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:51.126+0000 7fe3e6ffd700 1 -- 172.21.15.73:0/2546022816 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe3d80410c0 con 0x7fe3f00f5410 2023-12-01T22:15:51.128 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:51.127+0000 7fe3edd9b700 1 --2- 172.21.15.73:0/2546022816 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fe3dc02ff60 0x7fe3dc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:15:51.128 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:51.127+0000 7fe3edd9b700 1 --2- 172.21.15.73:0/2546022816 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fe3dc02ff60 0x7fe3dc032420 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7fe3e00096a0 tx=0x7fe3e0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:15:51.132 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:51.131+0000 7fe3e6ffd700 1 -- 172.21.15.73:0/2546022816 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe3d80095f0 con 0x7fe3f00f5410 2023-12-01T22:15:51.340 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:51.339+0000 7fe3f4d7e700 1 -- 172.21.15.73:0/2546022816 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe3f00fae60 con 0x7fe3f00f5410 2023-12-01T22:15:51.340 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:51.340+0000 7fe3e6ffd700 1 -- 172.21.15.73:0/2546022816 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe3d801db70 con 0x7fe3f00f5410 2023-12-01T22:15:51.341 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:15:51.341 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:15:51.342 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:51.341+0000 7fe3f4d7e700 1 -- 172.21.15.73:0/2546022816 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fe3dc02ff60 msgr2=0x7fe3dc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:15:51.342 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:51.341+0000 7fe3f4d7e700 1 --2- 172.21.15.73:0/2546022816 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fe3dc02ff60 0x7fe3dc032420 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7fe3e00096a0 tx=0x7fe3e0006b40).stop 2023-12-01T22:15:51.342 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:51.341+0000 7fe3f4d7e700 1 -- 172.21.15.73:0/2546022816 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3f00f5410 msgr2=0x7fe3f0105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:15:51.342 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:51.341+0000 7fe3f4d7e700 1 --2- 172.21.15.73:0/2546022816 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3f00f5410 0x7fe3f0105630 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7fe3d8013510 tx=0x7fe3d800b040).stop 2023-12-01T22:15:51.342 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:51.341+0000 7fe3f4d7e700 1 -- 172.21.15.73:0/2546022816 shutdown_connections 2023-12-01T22:15:51.342 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:51.341+0000 7fe3f4d7e700 1 --2- 172.21.15.73:0/2546022816 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fe3dc02ff60 0x7fe3dc032420 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:15:51.342 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:51.341+0000 7fe3f4d7e700 1 --2- 172.21.15.73:0/2546022816 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3f00f5410 0x7fe3f0105630 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:15:51.342 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:51.341+0000 7fe3f4d7e700 1 -- 172.21.15.73:0/2546022816 >> 172.21.15.73:0/2546022816 conn(0x7fe3f00f0b70 msgr2=0x7fe3f00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:15:51.342 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:51.341+0000 7fe3f4d7e700 1 -- 172.21.15.73:0/2546022816 shutdown_connections 2023-12-01T22:15:51.343 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:51.341+0000 7fe3f4d7e700 1 -- 172.21.15.73:0/2546022816 wait complete. 2023-12-01T22:15:51.343 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:15:51.776 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:15:51 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:15:50.662441+0000 mgr.smithi029.loxrfu (mgr.14164) 140 : cluster [DBG] pgmap v133: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:15:51.776 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:15:51 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:15:51.340153+0000 mon.smithi029 (mon.0) 215 : audit [DBG] from='client.? 172.21.15.73:0/2546022816' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:15:52.737 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:15:52.737 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:15:53.936 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:53.935+0000 7fec16f13700 1 -- 172.21.15.73:0/2097830774 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec100f51a0 msgr2=0x7fec100f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:15:53.936 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:53.935+0000 7fec16f13700 1 --2- 172.21.15.73:0/2097830774 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec100f51a0 0x7fec100f55c0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 rx=0x7fec00009fb0 tx=0x7fec00009b30).stop 2023-12-01T22:15:53.936 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:53.935+0000 7fec16f13700 1 -- 172.21.15.73:0/2097830774 shutdown_connections 2023-12-01T22:15:53.936 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:53.935+0000 7fec16f13700 1 --2- 172.21.15.73:0/2097830774 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec100f51a0 0x7fec100f55c0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:15:53.936 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:53.935+0000 7fec16f13700 1 -- 172.21.15.73:0/2097830774 >> 172.21.15.73:0/2097830774 conn(0x7fec100f09a0 msgr2=0x7fec100f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:15:53.937 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:53.935+0000 7fec16f13700 1 -- 172.21.15.73:0/2097830774 shutdown_connections 2023-12-01T22:15:53.937 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:53.935+0000 7fec16f13700 1 -- 172.21.15.73:0/2097830774 wait complete. 2023-12-01T22:15:53.937 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:53.935+0000 7fec16f13700 1 Processor -- start 2023-12-01T22:15:53.937 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:53.935+0000 7fec16f13700 1 -- start start 2023-12-01T22:15:53.937 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:53.935+0000 7fec16f13700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec100f51a0 0x7fec101064e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:15:53.937 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:53.935+0000 7fec16f13700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec101069d0 con 0x7fec100f51a0 2023-12-01T22:15:53.937 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:53.935+0000 7fec15f11700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec100f51a0 0x7fec101064e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:15:53.937 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:53.936+0000 7fec15f11700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec100f51a0 0x7fec101064e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:59900/0 (socket says 172.21.15.73:59900) 2023-12-01T22:15:53.937 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:53.936+0000 7fec15f11700 1 -- 172.21.15.73:0/347405092 learned_addr learned my addr 172.21.15.73:0/347405092 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:15:53.938 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:53.936+0000 7fec15f11700 1 -- 172.21.15.73:0/347405092 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec00005040 con 0x7fec100f51a0 2023-12-01T22:15:53.939 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:53.936+0000 7fec15f11700 1 --2- 172.21.15.73:0/347405092 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec100f51a0 0x7fec101064e0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7fec00013510 tx=0x7fec0000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:15:53.939 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:53.936+0000 7fec067fc700 1 -- 172.21.15.73:0/347405092 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fec00009320 con 0x7fec100f51a0 2023-12-01T22:15:53.939 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:53.936+0000 7fec16f13700 1 -- 172.21.15.73:0/347405092 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec10106bd0 con 0x7fec100f51a0 2023-12-01T22:15:53.939 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:53.936+0000 7fec067fc700 1 -- 172.21.15.73:0/347405092 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fec00013f00 con 0x7fec100f51a0 2023-12-01T22:15:53.939 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:53.936+0000 7fec067fc700 1 -- 172.21.15.73:0/347405092 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fec00014dc0 con 0x7fec100f51a0 2023-12-01T22:15:53.939 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:53.936+0000 7fec16f13700 1 -- 172.21.15.73:0/347405092 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec10107060 con 0x7fec100f51a0 2023-12-01T22:15:53.939 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:53.937+0000 7fec067fc700 1 -- 172.21.15.73:0/347405092 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fec000095f0 con 0x7fec100f51a0 2023-12-01T22:15:53.939 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:53.937+0000 7fec16f13700 1 -- 172.21.15.73:0/347405092 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec10045ad0 con 0x7fec100f51a0 2023-12-01T22:15:53.939 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:53.937+0000 7fec067fc700 1 --2- 172.21.15.73:0/347405092 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7febfc02ff60 0x7febfc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:15:53.939 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:53.937+0000 7fec067fc700 1 -- 172.21.15.73:0/347405092 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fec00041080 con 0x7fec100f51a0 2023-12-01T22:15:53.939 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:53.937+0000 7fec15710700 1 --2- 172.21.15.73:0/347405092 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7febfc02ff60 0x7febfc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:15:53.939 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:53.938+0000 7fec15710700 1 --2- 172.21.15.73:0/347405092 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7febfc02ff60 0x7febfc032420 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7fec0c0096a0 tx=0x7fec0c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:15:53.942 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:53.942+0000 7fec067fc700 1 -- 172.21.15.73:0/347405092 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fec00014420 con 0x7fec100f51a0 2023-12-01T22:15:54.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:15:53 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:15:52.662795+0000 mgr.smithi029.loxrfu (mgr.14164) 141 : cluster [DBG] pgmap v134: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:15:54.146 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:54.144+0000 7fec16f13700 1 -- 172.21.15.73:0/347405092 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fec10107f10 con 0x7fec100f51a0 2023-12-01T22:15:54.146 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:54.145+0000 7fec067fc700 1 -- 172.21.15.73:0/347405092 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fec00010230 con 0x7fec100f51a0 2023-12-01T22:15:54.146 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:15:54.147 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:15:54.149 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:54.146+0000 7fec16f13700 1 -- 172.21.15.73:0/347405092 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7febfc02ff60 msgr2=0x7febfc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:15:54.149 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:54.146+0000 7fec16f13700 1 --2- 172.21.15.73:0/347405092 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7febfc02ff60 0x7febfc032420 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7fec0c0096a0 tx=0x7fec0c006b40).stop 2023-12-01T22:15:54.149 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:54.146+0000 7fec16f13700 1 -- 172.21.15.73:0/347405092 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec100f51a0 msgr2=0x7fec101064e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:15:54.149 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:54.146+0000 7fec16f13700 1 --2- 172.21.15.73:0/347405092 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec100f51a0 0x7fec101064e0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7fec00013510 tx=0x7fec0000b040).stop 2023-12-01T22:15:54.149 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:54.146+0000 7fec16f13700 1 -- 172.21.15.73:0/347405092 shutdown_connections 2023-12-01T22:15:54.149 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:54.146+0000 7fec16f13700 1 --2- 172.21.15.73:0/347405092 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7febfc02ff60 0x7febfc032420 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:15:54.149 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:54.146+0000 7fec16f13700 1 --2- 172.21.15.73:0/347405092 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec100f51a0 0x7fec101064e0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:15:54.149 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:54.146+0000 7fec16f13700 1 -- 172.21.15.73:0/347405092 >> 172.21.15.73:0/347405092 conn(0x7fec100f09a0 msgr2=0x7fec1005fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:15:54.149 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:54.146+0000 7fec16f13700 1 -- 172.21.15.73:0/347405092 shutdown_connections 2023-12-01T22:15:54.150 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:54.146+0000 7fec16f13700 1 -- 172.21.15.73:0/347405092 wait complete. 2023-12-01T22:15:54.150 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:15:55.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:15:54 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:15:54.146174+0000 mon.smithi029 (mon.0) 216 : audit [DBG] from='client.? 172.21.15.73:0/347405092' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:15:55.520 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:15:55.520 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:15:56.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:15:55 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:15:54.663035+0000 mgr.smithi029.loxrfu (mgr.14164) 142 : cluster [DBG] pgmap v135: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:15:56.721 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:56.719+0000 7f402ab27700 1 -- 172.21.15.73:0/1355156565 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4024060200 msgr2=0x7f40240fb570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:15:56.721 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:56.719+0000 7f402ab27700 1 --2- 172.21.15.73:0/1355156565 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4024060200 0x7f40240fb570 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 rx=0x7f4018009fb0 tx=0x7f4018009b30).stop 2023-12-01T22:15:56.723 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:56.721+0000 7f402ab27700 1 -- 172.21.15.73:0/1355156565 shutdown_connections 2023-12-01T22:15:56.723 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:56.721+0000 7f402ab27700 1 --2- 172.21.15.73:0/1355156565 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4024060200 0x7f40240fb570 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:15:56.723 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:56.721+0000 7f402ab27700 1 -- 172.21.15.73:0/1355156565 >> 172.21.15.73:0/1355156565 conn(0x7f40240f0b70 msgr2=0x7f40240f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:15:56.723 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:56.722+0000 7f402ab27700 1 -- 172.21.15.73:0/1355156565 shutdown_connections 2023-12-01T22:15:56.723 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:56.722+0000 7f402ab27700 1 -- 172.21.15.73:0/1355156565 wait complete. 2023-12-01T22:15:56.724 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:56.722+0000 7f402ab27700 1 Processor -- start 2023-12-01T22:15:56.724 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:56.722+0000 7f402ab27700 1 -- start start 2023-12-01T22:15:56.724 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:56.723+0000 7f402ab27700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4024060200 0x7f40241088d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:15:56.724 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:56.723+0000 7f402ab27700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4024108dc0 con 0x7f4024060200 2023-12-01T22:15:56.725 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:56.723+0000 7f40288c3700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4024060200 0x7f40241088d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:15:56.726 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:56.723+0000 7f40288c3700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4024060200 0x7f40241088d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:33934/0 (socket says 172.21.15.73:33934) 2023-12-01T22:15:56.726 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:56.723+0000 7f40288c3700 1 -- 172.21.15.73:0/881174491 learned_addr learned my addr 172.21.15.73:0/881174491 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:15:56.726 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:56.723+0000 7f40288c3700 1 -- 172.21.15.73:0/881174491 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4018005040 con 0x7f4024060200 2023-12-01T22:15:56.726 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:56.724+0000 7f40288c3700 1 --2- 172.21.15.73:0/881174491 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4024060200 0x7f40241088d0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7f4018009830 tx=0x7f401800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:15:56.726 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:56.724+0000 7f40217fa700 1 -- 172.21.15.73:0/881174491 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4018009320 con 0x7f4024060200 2023-12-01T22:15:56.726 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:56.724+0000 7f402ab27700 1 -- 172.21.15.73:0/881174491 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4024108fc0 con 0x7f4024060200 2023-12-01T22:15:56.726 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:56.724+0000 7f40217fa700 1 -- 172.21.15.73:0/881174491 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f40180136c0 con 0x7f4024060200 2023-12-01T22:15:56.726 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:56.724+0000 7f40217fa700 1 -- 172.21.15.73:0/881174491 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f401801ee90 con 0x7f4024060200 2023-12-01T22:15:56.726 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:56.724+0000 7f402ab27700 1 -- 172.21.15.73:0/881174491 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4024109450 con 0x7f4024060200 2023-12-01T22:15:56.727 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:56.725+0000 7f40217fa700 1 -- 172.21.15.73:0/881174491 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4018014440 con 0x7f4024060200 2023-12-01T22:15:56.727 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:56.725+0000 7f402ab27700 1 -- 172.21.15.73:0/881174491 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4024045ad0 con 0x7f4024060200 2023-12-01T22:15:56.727 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:56.725+0000 7f40217fa700 1 --2- 172.21.15.73:0/881174491 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f401402ff10 0x7f40140323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:15:56.727 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:56.725+0000 7f40217fa700 1 -- 172.21.15.73:0/881174491 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4018041130 con 0x7f4024060200 2023-12-01T22:15:56.727 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:56.725+0000 7f4023fff700 1 --2- 172.21.15.73:0/881174491 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f401402ff10 0x7f40140323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:15:56.727 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:56.725+0000 7f4023fff700 1 --2- 172.21.15.73:0/881174491 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f401402ff10 0x7f40140323d0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f40100096a0 tx=0x7f4010006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:15:56.731 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:56.730+0000 7f40217fa700 1 -- 172.21.15.73:0/881174491 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f401801cb10 con 0x7f4024060200 2023-12-01T22:15:56.935 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:56.933+0000 7f402ab27700 1 -- 172.21.15.73:0/881174491 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f402410a300 con 0x7f4024060200 2023-12-01T22:15:56.936 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:56.934+0000 7f40217fa700 1 -- 172.21.15.73:0/881174491 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f401801cb10 con 0x7f4024060200 2023-12-01T22:15:56.936 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:15:56.936 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:15:56.937 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:56.936+0000 7f402ab27700 1 -- 172.21.15.73:0/881174491 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f401402ff10 msgr2=0x7f40140323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:15:56.938 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:56.936+0000 7f402ab27700 1 --2- 172.21.15.73:0/881174491 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f401402ff10 0x7f40140323d0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f40100096a0 tx=0x7f4010006b40).stop 2023-12-01T22:15:56.938 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:56.936+0000 7f402ab27700 1 -- 172.21.15.73:0/881174491 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4024060200 msgr2=0x7f40241088d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:15:56.938 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:56.936+0000 7f402ab27700 1 --2- 172.21.15.73:0/881174491 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4024060200 0x7f40241088d0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7f4018009830 tx=0x7f401800b040).stop 2023-12-01T22:15:56.938 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:56.936+0000 7f402ab27700 1 -- 172.21.15.73:0/881174491 shutdown_connections 2023-12-01T22:15:56.938 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:56.936+0000 7f402ab27700 1 --2- 172.21.15.73:0/881174491 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f401402ff10 0x7f40140323d0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:15:56.938 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:56.936+0000 7f402ab27700 1 --2- 172.21.15.73:0/881174491 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4024060200 0x7f40241088d0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:15:56.938 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:56.936+0000 7f402ab27700 1 -- 172.21.15.73:0/881174491 >> 172.21.15.73:0/881174491 conn(0x7f40240f0b70 msgr2=0x7f40240f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:15:56.938 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:56.936+0000 7f402ab27700 1 -- 172.21.15.73:0/881174491 shutdown_connections 2023-12-01T22:15:56.938 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:56.936+0000 7f402ab27700 1 -- 172.21.15.73:0/881174491 wait complete. 2023-12-01T22:15:56.939 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:15:58.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:15:57 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:15:56.663378+0000 mgr.smithi029.loxrfu (mgr.14164) 143 : cluster [DBG] pgmap v136: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:15:58.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:15:57 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:15:56.935516+0000 mon.smithi029 (mon.0) 217 : audit [DBG] from='client.? 172.21.15.73:0/881174491' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:15:58.303 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:15:58.303 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:15:59.483 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:59.480+0000 7ff635c50700 1 -- 172.21.15.73:0/1878488494 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff630060280 msgr2=0x7ff6300f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:15:59.483 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:59.480+0000 7ff635c50700 1 --2- 172.21.15.73:0/1878488494 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff630060280 0x7ff6300f90f0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0x7ff618009fb0 tx=0x7ff618009b30).stop 2023-12-01T22:15:59.483 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:59.480+0000 7ff635c50700 1 -- 172.21.15.73:0/1878488494 shutdown_connections 2023-12-01T22:15:59.483 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:59.480+0000 7ff635c50700 1 --2- 172.21.15.73:0/1878488494 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff630060280 0x7ff6300f90f0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:15:59.484 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:59.480+0000 7ff635c50700 1 -- 172.21.15.73:0/1878488494 >> 172.21.15.73:0/1878488494 conn(0x7ff6300f0b70 msgr2=0x7ff6300f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:15:59.484 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:59.480+0000 7ff635c50700 1 -- 172.21.15.73:0/1878488494 shutdown_connections 2023-12-01T22:15:59.484 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:59.480+0000 7ff635c50700 1 -- 172.21.15.73:0/1878488494 wait complete. 2023-12-01T22:15:59.484 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:59.481+0000 7ff635c50700 1 Processor -- start 2023-12-01T22:15:59.484 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:59.481+0000 7ff635c50700 1 -- start start 2023-12-01T22:15:59.484 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:59.481+0000 7ff635c50700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff630060280 0x7ff6301066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:15:59.484 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:59.481+0000 7ff635c50700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff630106bd0 con 0x7ff630060280 2023-12-01T22:15:59.484 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:59.481+0000 7ff62f7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff630060280 0x7ff6301066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:15:59.484 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:59.481+0000 7ff62f7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff630060280 0x7ff6301066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:33950/0 (socket says 172.21.15.73:33950) 2023-12-01T22:15:59.484 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:59.481+0000 7ff62f7fe700 1 -- 172.21.15.73:0/2657128976 learned_addr learned my addr 172.21.15.73:0/2657128976 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:15:59.484 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:59.482+0000 7ff62f7fe700 1 -- 172.21.15.73:0/2657128976 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff618005040 con 0x7ff630060280 2023-12-01T22:15:59.484 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:59.482+0000 7ff62f7fe700 1 --2- 172.21.15.73:0/2657128976 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff630060280 0x7ff6301066e0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7ff618009830 tx=0x7ff61800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:15:59.486 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:59.482+0000 7ff627fff700 1 -- 172.21.15.73:0/2657128976 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff618009320 con 0x7ff630060280 2023-12-01T22:15:59.486 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:59.482+0000 7ff635c50700 1 -- 172.21.15.73:0/2657128976 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff630106dd0 con 0x7ff630060280 2023-12-01T22:15:59.486 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:59.482+0000 7ff627fff700 1 -- 172.21.15.73:0/2657128976 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff6180136c0 con 0x7ff630060280 2023-12-01T22:15:59.486 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:59.482+0000 7ff627fff700 1 -- 172.21.15.73:0/2657128976 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff61801ee90 con 0x7ff630060280 2023-12-01T22:15:59.486 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:59.482+0000 7ff635c50700 1 -- 172.21.15.73:0/2657128976 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff630107260 con 0x7ff630060280 2023-12-01T22:15:59.486 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:59.483+0000 7ff627fff700 1 -- 172.21.15.73:0/2657128976 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff61801e420 con 0x7ff630060280 2023-12-01T22:15:59.486 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:59.483+0000 7ff635c50700 1 -- 172.21.15.73:0/2657128976 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff630045ad0 con 0x7ff630060280 2023-12-01T22:15:59.486 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:59.483+0000 7ff627fff700 1 --2- 172.21.15.73:0/2657128976 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7ff61c02ff10 0x7ff61c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:15:59.486 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:59.483+0000 7ff627fff700 1 -- 172.21.15.73:0/2657128976 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff618040fe0 con 0x7ff630060280 2023-12-01T22:15:59.486 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:59.483+0000 7ff62effd700 1 --2- 172.21.15.73:0/2657128976 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7ff61c02ff10 0x7ff61c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:15:59.486 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:59.483+0000 7ff62effd700 1 --2- 172.21.15.73:0/2657128976 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7ff61c02ff10 0x7ff61c0323d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7ff6200096a0 tx=0x7ff620006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:15:59.489 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:59.487+0000 7ff627fff700 1 -- 172.21.15.73:0/2657128976 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff61801c790 con 0x7ff630060280 2023-12-01T22:15:59.693 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:59.692+0000 7ff635c50700 1 -- 172.21.15.73:0/2657128976 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff630108110 con 0x7ff630060280 2023-12-01T22:15:59.694 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:59.692+0000 7ff627fff700 1 -- 172.21.15.73:0/2657128976 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff61802c2d0 con 0x7ff630060280 2023-12-01T22:15:59.694 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:15:59.694 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:15:59.695 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:59.694+0000 7ff635c50700 1 -- 172.21.15.73:0/2657128976 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7ff61c02ff10 msgr2=0x7ff61c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:15:59.695 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:59.694+0000 7ff635c50700 1 --2- 172.21.15.73:0/2657128976 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7ff61c02ff10 0x7ff61c0323d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7ff6200096a0 tx=0x7ff620006b40).stop 2023-12-01T22:15:59.696 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:59.694+0000 7ff635c50700 1 -- 172.21.15.73:0/2657128976 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff630060280 msgr2=0x7ff6301066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:15:59.696 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:59.694+0000 7ff635c50700 1 --2- 172.21.15.73:0/2657128976 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff630060280 0x7ff6301066e0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7ff618009830 tx=0x7ff61800b040).stop 2023-12-01T22:15:59.696 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:59.694+0000 7ff635c50700 1 -- 172.21.15.73:0/2657128976 shutdown_connections 2023-12-01T22:15:59.696 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:59.694+0000 7ff635c50700 1 --2- 172.21.15.73:0/2657128976 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7ff61c02ff10 0x7ff61c0323d0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:15:59.696 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:59.694+0000 7ff635c50700 1 --2- 172.21.15.73:0/2657128976 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff630060280 0x7ff6301066e0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:15:59.696 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:59.694+0000 7ff635c50700 1 -- 172.21.15.73:0/2657128976 >> 172.21.15.73:0/2657128976 conn(0x7ff6300f0b70 msgr2=0x7ff6300f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:15:59.696 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:59.694+0000 7ff635c50700 1 -- 172.21.15.73:0/2657128976 shutdown_connections 2023-12-01T22:15:59.696 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:15:59.694+0000 7ff635c50700 1 -- 172.21.15.73:0/2657128976 wait complete. 2023-12-01T22:15:59.696 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:16:00.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:15:59 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:15:58.663696+0000 mgr.smithi029.loxrfu (mgr.14164) 144 : cluster [DBG] pgmap v137: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:16:00.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:15:59 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:15:59.693659+0000 mon.smithi029 (mon.0) 218 : audit [DBG] from='client.? 172.21.15.73:0/2657128976' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:16:01.065 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:16:01.065 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:16:01.776 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:01 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:16:00.663952+0000 mgr.smithi029.loxrfu (mgr.14164) 145 : cluster [DBG] pgmap v138: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:16:02.206 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:02.203+0000 7f206b59e700 1 -- 172.21.15.73:0/432729556 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f206c0f62a0 msgr2=0x7f206c0f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:16:02.206 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:02.203+0000 7f206b59e700 1 --2- 172.21.15.73:0/432729556 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f206c0f62a0 0x7f206c0f66c0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 rx=0x7f2060009fb0 tx=0x7f2060009b30).stop 2023-12-01T22:16:02.206 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:02.203+0000 7f206b59e700 1 -- 172.21.15.73:0/432729556 shutdown_connections 2023-12-01T22:16:02.206 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:02.203+0000 7f206b59e700 1 --2- 172.21.15.73:0/432729556 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f206c0f62a0 0x7f206c0f66c0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:16:02.207 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:02.203+0000 7f206b59e700 1 -- 172.21.15.73:0/432729556 >> 172.21.15.73:0/432729556 conn(0x7f206c0f1aa0 msgr2=0x7f206c0f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:16:02.207 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:02.203+0000 7f206b59e700 1 -- 172.21.15.73:0/432729556 shutdown_connections 2023-12-01T22:16:02.207 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:02.203+0000 7f206b59e700 1 -- 172.21.15.73:0/432729556 wait complete. 2023-12-01T22:16:02.207 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:02.203+0000 7f206b59e700 1 Processor -- start 2023-12-01T22:16:02.207 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:02.203+0000 7f206b59e700 1 -- start start 2023-12-01T22:16:02.207 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:02.203+0000 7f206b59e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f206c0f62a0 0x7f206c1064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:16:02.207 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:02.203+0000 7f206b59e700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f206c1069a0 con 0x7f206c0f62a0 2023-12-01T22:16:02.207 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:02.204+0000 7f206a59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f206c0f62a0 0x7f206c1064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:16:02.207 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:02.204+0000 7f206a59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f206c0f62a0 0x7f206c1064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:33972/0 (socket says 172.21.15.73:33972) 2023-12-01T22:16:02.207 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:02.204+0000 7f206a59c700 1 -- 172.21.15.73:0/2208988556 learned_addr learned my addr 172.21.15.73:0/2208988556 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:16:02.207 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:02.204+0000 7f206a59c700 1 -- 172.21.15.73:0/2208988556 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2060005040 con 0x7f206c0f62a0 2023-12-01T22:16:02.207 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:02.204+0000 7f206a59c700 1 --2- 172.21.15.73:0/2208988556 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f206c0f62a0 0x7f206c1064b0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7f2060013510 tx=0x7f206000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:16:02.207 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:02.204+0000 7f205affd700 1 -- 172.21.15.73:0/2208988556 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2060009320 con 0x7f206c0f62a0 2023-12-01T22:16:02.207 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:02.204+0000 7f206b59e700 1 -- 172.21.15.73:0/2208988556 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f206c106ba0 con 0x7f206c0f62a0 2023-12-01T22:16:02.208 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:02.204+0000 7f205affd700 1 -- 172.21.15.73:0/2208988556 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2060013f00 con 0x7f206c0f62a0 2023-12-01T22:16:02.208 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:02.204+0000 7f205affd700 1 -- 172.21.15.73:0/2208988556 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2060014dc0 con 0x7f206c0f62a0 2023-12-01T22:16:02.208 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:02.204+0000 7f206b59e700 1 -- 172.21.15.73:0/2208988556 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f206c107030 con 0x7f206c0f62a0 2023-12-01T22:16:02.208 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:02.205+0000 7f205affd700 1 -- 172.21.15.73:0/2208988556 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f20600095f0 con 0x7f206c0f62a0 2023-12-01T22:16:02.209 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:02.205+0000 7f206b59e700 1 -- 172.21.15.73:0/2208988556 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f206c045ad0 con 0x7f206c0f62a0 2023-12-01T22:16:02.209 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:02.205+0000 7f205affd700 1 --2- 172.21.15.73:0/2208988556 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f205402ff60 0x7f2054032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:16:02.209 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:02.205+0000 7f205affd700 1 -- 172.21.15.73:0/2208988556 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2060041040 con 0x7f206c0f62a0 2023-12-01T22:16:02.209 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:02.206+0000 7f2069d9b700 1 --2- 172.21.15.73:0/2208988556 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f205402ff60 0x7f2054032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:16:02.209 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:02.206+0000 7f2069d9b700 1 --2- 172.21.15.73:0/2208988556 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f205402ff60 0x7f2054032420 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f205c0096a0 tx=0x7f205c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:16:02.212 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:02.211+0000 7f205affd700 1 -- 172.21.15.73:0/2208988556 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2060014420 con 0x7f206c0f62a0 2023-12-01T22:16:02.422 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:02.420+0000 7f206b59e700 1 -- 172.21.15.73:0/2208988556 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f206c107ee0 con 0x7f206c0f62a0 2023-12-01T22:16:02.422 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:02.421+0000 7f205affd700 1 -- 172.21.15.73:0/2208988556 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2060010230 con 0x7f206c0f62a0 2023-12-01T22:16:02.422 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:16:02.422 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:16:02.424 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:02.422+0000 7f206b59e700 1 -- 172.21.15.73:0/2208988556 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f205402ff60 msgr2=0x7f2054032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:16:02.424 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:02.422+0000 7f206b59e700 1 --2- 172.21.15.73:0/2208988556 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f205402ff60 0x7f2054032420 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f205c0096a0 tx=0x7f205c006b40).stop 2023-12-01T22:16:02.424 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:02.422+0000 7f206b59e700 1 -- 172.21.15.73:0/2208988556 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f206c0f62a0 msgr2=0x7f206c1064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:16:02.424 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:02.422+0000 7f206b59e700 1 --2- 172.21.15.73:0/2208988556 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f206c0f62a0 0x7f206c1064b0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7f2060013510 tx=0x7f206000b040).stop 2023-12-01T22:16:02.424 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:02.422+0000 7f206b59e700 1 -- 172.21.15.73:0/2208988556 shutdown_connections 2023-12-01T22:16:02.424 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:02.422+0000 7f206b59e700 1 --2- 172.21.15.73:0/2208988556 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f205402ff60 0x7f2054032420 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:16:02.425 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:02.422+0000 7f206b59e700 1 --2- 172.21.15.73:0/2208988556 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f206c0f62a0 0x7f206c1064b0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:16:02.425 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:02.422+0000 7f206b59e700 1 -- 172.21.15.73:0/2208988556 >> 172.21.15.73:0/2208988556 conn(0x7f206c0f1aa0 msgr2=0x7f206c0f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:16:02.425 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:02.422+0000 7f206b59e700 1 -- 172.21.15.73:0/2208988556 shutdown_connections 2023-12-01T22:16:02.425 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:02.422+0000 7f206b59e700 1 -- 172.21.15.73:0/2208988556 wait complete. 2023-12-01T22:16:02.425 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:16:02.776 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:02 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:16:02.421933+0000 mon.smithi029 (mon.0) 219 : audit [DBG] from='client.? 172.21.15.73:0/2208988556' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:16:03.777 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:03 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:16:02.664273+0000 mgr.smithi029.loxrfu (mgr.14164) 146 : cluster [DBG] pgmap v139: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:16:03.796 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:16:03.796 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:16:04.953 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:04.951+0000 7f4d4cc7c700 1 -- 172.21.15.73:0/1079656130 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d480f51a0 msgr2=0x7f4d480f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:16:04.954 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:04.951+0000 7f4d4cc7c700 1 --2- 172.21.15.73:0/1079656130 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d480f51a0 0x7f4d480f55c0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 rx=0x7f4d38009fb0 tx=0x7f4d38009b30).stop 2023-12-01T22:16:04.954 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:04.951+0000 7f4d4cc7c700 1 -- 172.21.15.73:0/1079656130 shutdown_connections 2023-12-01T22:16:04.954 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:04.951+0000 7f4d4cc7c700 1 --2- 172.21.15.73:0/1079656130 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d480f51a0 0x7f4d480f55c0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:16:04.954 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:04.951+0000 7f4d4cc7c700 1 -- 172.21.15.73:0/1079656130 >> 172.21.15.73:0/1079656130 conn(0x7f4d480f09a0 msgr2=0x7f4d480f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:16:04.954 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:04.951+0000 7f4d4cc7c700 1 -- 172.21.15.73:0/1079656130 shutdown_connections 2023-12-01T22:16:04.954 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:04.951+0000 7f4d4cc7c700 1 -- 172.21.15.73:0/1079656130 wait complete. 2023-12-01T22:16:04.954 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:04.951+0000 7f4d4cc7c700 1 Processor -- start 2023-12-01T22:16:04.954 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:04.952+0000 7f4d4cc7c700 1 -- start start 2023-12-01T22:16:04.954 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:04.952+0000 7f4d4cc7c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d480f51a0 0x7f4d48109170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:16:04.954 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:04.952+0000 7f4d4cc7c700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d48060280 con 0x7f4d480f51a0 2023-12-01T22:16:04.956 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:04.952+0000 7f4d477fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d480f51a0 0x7f4d48109170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:16:04.956 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:04.952+0000 7f4d477fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d480f51a0 0x7f4d48109170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:39354/0 (socket says 172.21.15.73:39354) 2023-12-01T22:16:04.956 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:04.952+0000 7f4d477fe700 1 -- 172.21.15.73:0/3033190649 learned_addr learned my addr 172.21.15.73:0/3033190649 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:16:04.956 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:04.952+0000 7f4d477fe700 1 -- 172.21.15.73:0/3033190649 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d38005040 con 0x7f4d480f51a0 2023-12-01T22:16:04.956 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:04.953+0000 7f4d477fe700 1 --2- 172.21.15.73:0/3033190649 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d480f51a0 0x7f4d48109170 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7f4d38013510 tx=0x7f4d3800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:16:04.956 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:04.953+0000 7f4d2ffff700 1 -- 172.21.15.73:0/3033190649 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4d38009320 con 0x7f4d480f51a0 2023-12-01T22:16:04.956 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:04.953+0000 7f4d2ffff700 1 -- 172.21.15.73:0/3033190649 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4d38013f00 con 0x7f4d480f51a0 2023-12-01T22:16:04.956 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:04.953+0000 7f4d4cc7c700 1 -- 172.21.15.73:0/3033190649 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d48060480 con 0x7f4d480f51a0 2023-12-01T22:16:04.956 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:04.953+0000 7f4d2ffff700 1 -- 172.21.15.73:0/3033190649 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4d38014dc0 con 0x7f4d480f51a0 2023-12-01T22:16:04.957 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:04.953+0000 7f4d4cc7c700 1 -- 172.21.15.73:0/3033190649 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d481098a0 con 0x7f4d480f51a0 2023-12-01T22:16:04.957 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:04.954+0000 7f4d2ffff700 1 -- 172.21.15.73:0/3033190649 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4d38014420 con 0x7f4d480f51a0 2023-12-01T22:16:04.957 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:04.954+0000 7f4d4cc7c700 1 -- 172.21.15.73:0/3033190649 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d48045ad0 con 0x7f4d480f51a0 2023-12-01T22:16:04.957 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:04.954+0000 7f4d2ffff700 1 --2- 172.21.15.73:0/3033190649 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f4d3002ff60 0x7f4d30032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:16:04.957 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:04.954+0000 7f4d2ffff700 1 -- 172.21.15.73:0/3033190649 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4d380410f0 con 0x7f4d480f51a0 2023-12-01T22:16:04.957 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:04.954+0000 7f4d46ffd700 1 --2- 172.21.15.73:0/3033190649 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f4d3002ff60 0x7f4d30032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:16:04.957 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:04.954+0000 7f4d46ffd700 1 --2- 172.21.15.73:0/3033190649 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f4d3002ff60 0x7f4d30032420 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f4d3c0096a0 tx=0x7f4d3c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:16:04.960 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:04.958+0000 7f4d2ffff700 1 -- 172.21.15.73:0/3033190649 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4d380095f0 con 0x7f4d480f51a0 2023-12-01T22:16:05.166 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:05.164+0000 7f4d4cc7c700 1 -- 172.21.15.73:0/3033190649 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4d480fabe0 con 0x7f4d480f51a0 2023-12-01T22:16:05.167 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:05.165+0000 7f4d2ffff700 1 -- 172.21.15.73:0/3033190649 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4d3801cb10 con 0x7f4d480f51a0 2023-12-01T22:16:05.167 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:16:05.167 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:16:05.168 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:05.167+0000 7f4d4cc7c700 1 -- 172.21.15.73:0/3033190649 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f4d3002ff60 msgr2=0x7f4d30032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:16:05.168 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:05.167+0000 7f4d4cc7c700 1 --2- 172.21.15.73:0/3033190649 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f4d3002ff60 0x7f4d30032420 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f4d3c0096a0 tx=0x7f4d3c006b40).stop 2023-12-01T22:16:05.168 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:05.167+0000 7f4d4cc7c700 1 -- 172.21.15.73:0/3033190649 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d480f51a0 msgr2=0x7f4d48109170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:16:05.169 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:05.167+0000 7f4d4cc7c700 1 --2- 172.21.15.73:0/3033190649 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d480f51a0 0x7f4d48109170 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7f4d38013510 tx=0x7f4d3800b040).stop 2023-12-01T22:16:05.169 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:05.167+0000 7f4d4cc7c700 1 -- 172.21.15.73:0/3033190649 shutdown_connections 2023-12-01T22:16:05.169 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:05.167+0000 7f4d4cc7c700 1 --2- 172.21.15.73:0/3033190649 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f4d3002ff60 0x7f4d30032420 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:16:05.169 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:05.167+0000 7f4d4cc7c700 1 --2- 172.21.15.73:0/3033190649 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d480f51a0 0x7f4d48109170 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:16:05.169 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:05.167+0000 7f4d4cc7c700 1 -- 172.21.15.73:0/3033190649 >> 172.21.15.73:0/3033190649 conn(0x7f4d480f09a0 msgr2=0x7f4d480f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:16:05.169 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:05.167+0000 7f4d4cc7c700 1 -- 172.21.15.73:0/3033190649 shutdown_connections 2023-12-01T22:16:05.169 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:05.167+0000 7f4d4cc7c700 1 -- 172.21.15.73:0/3033190649 wait complete. 2023-12-01T22:16:05.169 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:16:06.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:05 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:16:04.664539+0000 mgr.smithi029.loxrfu (mgr.14164) 147 : cluster [DBG] pgmap v140: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:16:06.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:05 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:16:05.166300+0000 mon.smithi029 (mon.0) 220 : audit [DBG] from='client.? 172.21.15.73:0/3033190649' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:16:06.538 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:16:06.538 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:16:07.637 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:07.634+0000 7f81a1249700 1 -- 172.21.15.73:0/3272811024 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f819c0f51a0 msgr2=0x7f819c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:16:07.637 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:07.634+0000 7f81a1249700 1 --2- 172.21.15.73:0/3272811024 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f819c0f51a0 0x7f819c0f55c0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 rx=0x7f818c009fb0 tx=0x7f818c009b30).stop 2023-12-01T22:16:07.637 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:07.634+0000 7f81a1249700 1 -- 172.21.15.73:0/3272811024 shutdown_connections 2023-12-01T22:16:07.637 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:07.634+0000 7f81a1249700 1 --2- 172.21.15.73:0/3272811024 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f819c0f51a0 0x7f819c0f55c0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:16:07.637 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:07.634+0000 7f81a1249700 1 -- 172.21.15.73:0/3272811024 >> 172.21.15.73:0/3272811024 conn(0x7f819c0f09a0 msgr2=0x7f819c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:16:07.637 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:07.634+0000 7f81a1249700 1 -- 172.21.15.73:0/3272811024 shutdown_connections 2023-12-01T22:16:07.637 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:07.634+0000 7f81a1249700 1 -- 172.21.15.73:0/3272811024 wait complete. 2023-12-01T22:16:07.637 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:07.634+0000 7f81a1249700 1 Processor -- start 2023-12-01T22:16:07.637 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:07.634+0000 7f81a1249700 1 -- start start 2023-12-01T22:16:07.637 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:07.635+0000 7f81a1249700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f819c0f51a0 0x7f819c1091d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:16:07.637 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:07.635+0000 7f81a1249700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f819c060220 con 0x7f819c0f51a0 2023-12-01T22:16:07.638 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:07.635+0000 7f819bfff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f819c0f51a0 0x7f819c1091d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:16:07.638 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:07.635+0000 7f819bfff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f819c0f51a0 0x7f819c1091d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:39376/0 (socket says 172.21.15.73:39376) 2023-12-01T22:16:07.638 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:07.635+0000 7f819bfff700 1 -- 172.21.15.73:0/3363893037 learned_addr learned my addr 172.21.15.73:0/3363893037 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:16:07.638 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:07.635+0000 7f819bfff700 1 -- 172.21.15.73:0/3363893037 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f818c005040 con 0x7f819c0f51a0 2023-12-01T22:16:07.638 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:07.635+0000 7f819bfff700 1 --2- 172.21.15.73:0/3363893037 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f819c0f51a0 0x7f819c1091d0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7f818c013510 tx=0x7f818c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:16:07.639 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:07.635+0000 7f8198ff9700 1 -- 172.21.15.73:0/3363893037 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f818c009320 con 0x7f819c0f51a0 2023-12-01T22:16:07.639 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:07.636+0000 7f8198ff9700 1 -- 172.21.15.73:0/3363893037 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f818c013f00 con 0x7f819c0f51a0 2023-12-01T22:16:07.639 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:07.636+0000 7f81a1249700 1 -- 172.21.15.73:0/3363893037 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f819c060420 con 0x7f819c0f51a0 2023-12-01T22:16:07.639 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:07.636+0000 7f8198ff9700 1 -- 172.21.15.73:0/3363893037 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f818c014dc0 con 0x7f819c0f51a0 2023-12-01T22:16:07.639 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:07.636+0000 7f81a1249700 1 -- 172.21.15.73:0/3363893037 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f819c109880 con 0x7f819c0f51a0 2023-12-01T22:16:07.639 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:07.636+0000 7f8198ff9700 1 -- 172.21.15.73:0/3363893037 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f818c0095f0 con 0x7f819c0f51a0 2023-12-01T22:16:07.639 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:07.636+0000 7f81a1249700 1 -- 172.21.15.73:0/3363893037 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f819c059100 con 0x7f819c0f51a0 2023-12-01T22:16:07.639 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:07.637+0000 7f8198ff9700 1 --2- 172.21.15.73:0/3363893037 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f818402ff60 0x7f8184032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:16:07.640 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:07.637+0000 7f8198ff9700 1 -- 172.21.15.73:0/3363893037 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f818c041100 con 0x7f819c0f51a0 2023-12-01T22:16:07.640 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:07.637+0000 7f819b7fe700 1 --2- 172.21.15.73:0/3363893037 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f818402ff60 0x7f8184032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:16:07.640 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:07.637+0000 7f819b7fe700 1 --2- 172.21.15.73:0/3363893037 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f818402ff60 0x7f8184032420 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f81900096a0 tx=0x7f8190006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:16:07.643 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:07.641+0000 7f8198ff9700 1 -- 172.21.15.73:0/3363893037 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f818c014420 con 0x7f819c0f51a0 2023-12-01T22:16:07.854 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:07.853+0000 7f81a1249700 1 -- 172.21.15.73:0/3363893037 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f819c045ad0 con 0x7f819c0f51a0 2023-12-01T22:16:07.855 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:07.854+0000 7f8198ff9700 1 -- 172.21.15.73:0/3363893037 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f818c010230 con 0x7f819c0f51a0 2023-12-01T22:16:07.855 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:16:07.856 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:16:07.857 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:07.855+0000 7f81a1249700 1 -- 172.21.15.73:0/3363893037 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f818402ff60 msgr2=0x7f8184032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:16:07.857 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:07.855+0000 7f81a1249700 1 --2- 172.21.15.73:0/3363893037 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f818402ff60 0x7f8184032420 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f81900096a0 tx=0x7f8190006b40).stop 2023-12-01T22:16:07.857 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:07.855+0000 7f81a1249700 1 -- 172.21.15.73:0/3363893037 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f819c0f51a0 msgr2=0x7f819c1091d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:16:07.857 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:07.855+0000 7f81a1249700 1 --2- 172.21.15.73:0/3363893037 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f819c0f51a0 0x7f819c1091d0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7f818c013510 tx=0x7f818c00b040).stop 2023-12-01T22:16:07.858 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:07.855+0000 7f81a1249700 1 -- 172.21.15.73:0/3363893037 shutdown_connections 2023-12-01T22:16:07.858 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:07.855+0000 7f81a1249700 1 --2- 172.21.15.73:0/3363893037 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f818402ff60 0x7f8184032420 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:16:07.858 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:07.855+0000 7f81a1249700 1 --2- 172.21.15.73:0/3363893037 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f819c0f51a0 0x7f819c1091d0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:16:07.858 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:07.856+0000 7f81a1249700 1 -- 172.21.15.73:0/3363893037 >> 172.21.15.73:0/3363893037 conn(0x7f819c0f09a0 msgr2=0x7f819c0f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:16:07.858 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:07.856+0000 7f81a1249700 1 -- 172.21.15.73:0/3363893037 shutdown_connections 2023-12-01T22:16:07.858 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:07.856+0000 7f81a1249700 1 -- 172.21.15.73:0/3363893037 wait complete. 2023-12-01T22:16:07.858 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:16:08.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:07 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:16:06.664803+0000 mgr.smithi029.loxrfu (mgr.14164) 148 : cluster [DBG] pgmap v141: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:16:09.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:08 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:16:07.854964+0000 mon.smithi029 (mon.0) 221 : audit [DBG] from='client.? 172.21.15.73:0/3363893037' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:16:09.233 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:16:09.233 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:16:10.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:09 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:16:08.665074+0000 mgr.smithi029.loxrfu (mgr.14164) 149 : cluster [DBG] pgmap v142: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:16:10.373 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:10.370+0000 7f8724f17700 1 -- 172.21.15.73:0/1009392358 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87200f51a0 msgr2=0x7f87200f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:16:10.373 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:10.370+0000 7f8724f17700 1 --2- 172.21.15.73:0/1009392358 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87200f51a0 0x7f87200f55c0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 rx=0x7f8710009fb0 tx=0x7f8710009b30).stop 2023-12-01T22:16:10.373 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:10.371+0000 7f8724f17700 1 -- 172.21.15.73:0/1009392358 shutdown_connections 2023-12-01T22:16:10.373 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:10.371+0000 7f8724f17700 1 --2- 172.21.15.73:0/1009392358 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87200f51a0 0x7f87200f55c0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:16:10.373 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:10.371+0000 7f8724f17700 1 -- 172.21.15.73:0/1009392358 >> 172.21.15.73:0/1009392358 conn(0x7f87200f09a0 msgr2=0x7f87200f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:16:10.373 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:10.371+0000 7f8724f17700 1 -- 172.21.15.73:0/1009392358 shutdown_connections 2023-12-01T22:16:10.373 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:10.371+0000 7f8724f17700 1 -- 172.21.15.73:0/1009392358 wait complete. 2023-12-01T22:16:10.374 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:10.371+0000 7f8724f17700 1 Processor -- start 2023-12-01T22:16:10.374 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:10.371+0000 7f8724f17700 1 -- start start 2023-12-01T22:16:10.376 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:10.371+0000 7f8724f17700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87200fd160 0x7f87200fb620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:16:10.376 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:10.371+0000 7f8724f17700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87200fd580 con 0x7f87200fd160 2023-12-01T22:16:10.376 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:10.372+0000 7f871f7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87200fd160 0x7f87200fb620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:16:10.376 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:10.372+0000 7f871f7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87200fd160 0x7f87200fb620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:39392/0 (socket says 172.21.15.73:39392) 2023-12-01T22:16:10.376 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:10.372+0000 7f871f7fe700 1 -- 172.21.15.73:0/808707302 learned_addr learned my addr 172.21.15.73:0/808707302 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:16:10.376 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:10.372+0000 7f871f7fe700 1 -- 172.21.15.73:0/808707302 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8710005040 con 0x7f87200fd160 2023-12-01T22:16:10.377 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:10.372+0000 7f871f7fe700 1 --2- 172.21.15.73:0/808707302 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87200fd160 0x7f87200fb620 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7f8710009f80 tx=0x7f8710009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:16:10.377 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:10.372+0000 7f8707fff700 1 -- 172.21.15.73:0/808707302 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f87100136c0 con 0x7f87200fd160 2023-12-01T22:16:10.377 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:10.372+0000 7f8707fff700 1 -- 172.21.15.73:0/808707302 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f87100095f0 con 0x7f87200fd160 2023-12-01T22:16:10.377 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:10.372+0000 7f8724f17700 1 -- 172.21.15.73:0/808707302 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87200fbb10 con 0x7f87200fd160 2023-12-01T22:16:10.377 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:10.372+0000 7f8707fff700 1 -- 172.21.15.73:0/808707302 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8710014420 con 0x7f87200fd160 2023-12-01T22:16:10.377 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:10.372+0000 7f8724f17700 1 -- 172.21.15.73:0/808707302 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87200fc000 con 0x7f87200fd160 2023-12-01T22:16:10.377 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:10.373+0000 7f8707fff700 1 -- 172.21.15.73:0/808707302 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8710025060 con 0x7f87200fd160 2023-12-01T22:16:10.377 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:10.373+0000 7f8724f17700 1 -- 172.21.15.73:0/808707302 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8720059100 con 0x7f87200fd160 2023-12-01T22:16:10.377 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:10.373+0000 7f8707fff700 1 --2- 172.21.15.73:0/808707302 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f870802ff60 0x7f8708032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:16:10.378 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:10.373+0000 7f8707fff700 1 -- 172.21.15.73:0/808707302 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8710040e40 con 0x7f87200fd160 2023-12-01T22:16:10.378 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:10.373+0000 7f871effd700 1 --2- 172.21.15.73:0/808707302 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f870802ff60 0x7f8708032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:16:10.378 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:10.374+0000 7f871effd700 1 --2- 172.21.15.73:0/808707302 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f870802ff60 0x7f8708032420 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f87140096a0 tx=0x7f8714006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:16:10.379 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:10.378+0000 7f8707fff700 1 -- 172.21.15.73:0/808707302 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f871001c950 con 0x7f87200fd160 2023-12-01T22:16:10.586 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:10.584+0000 7f8724f17700 1 -- 172.21.15.73:0/808707302 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f87200fc3f0 con 0x7f87200fd160 2023-12-01T22:16:10.586 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:10.585+0000 7f8707fff700 1 -- 172.21.15.73:0/808707302 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f871001cb60 con 0x7f87200fd160 2023-12-01T22:16:10.587 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:16:10.587 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:16:10.588 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:10.586+0000 7f8724f17700 1 -- 172.21.15.73:0/808707302 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f870802ff60 msgr2=0x7f8708032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:16:10.588 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:10.586+0000 7f8724f17700 1 --2- 172.21.15.73:0/808707302 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f870802ff60 0x7f8708032420 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f87140096a0 tx=0x7f8714006b40).stop 2023-12-01T22:16:10.588 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:10.586+0000 7f8724f17700 1 -- 172.21.15.73:0/808707302 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87200fd160 msgr2=0x7f87200fb620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:16:10.589 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:10.586+0000 7f8724f17700 1 --2- 172.21.15.73:0/808707302 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87200fd160 0x7f87200fb620 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7f8710009f80 tx=0x7f8710009b30).stop 2023-12-01T22:16:10.589 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:10.586+0000 7f8724f17700 1 -- 172.21.15.73:0/808707302 shutdown_connections 2023-12-01T22:16:10.589 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:10.586+0000 7f8724f17700 1 --2- 172.21.15.73:0/808707302 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f870802ff60 0x7f8708032420 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:16:10.589 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:10.586+0000 7f8724f17700 1 --2- 172.21.15.73:0/808707302 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87200fd160 0x7f87200fb620 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:16:10.589 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:10.586+0000 7f8724f17700 1 -- 172.21.15.73:0/808707302 >> 172.21.15.73:0/808707302 conn(0x7f87200f09a0 msgr2=0x7f872005fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:16:10.589 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:10.587+0000 7f8724f17700 1 -- 172.21.15.73:0/808707302 shutdown_connections 2023-12-01T22:16:10.589 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:10.587+0000 7f8724f17700 1 -- 172.21.15.73:0/808707302 wait complete. 2023-12-01T22:16:10.589 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:16:11.777 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:11 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:16:10.586298+0000 mon.smithi029 (mon.0) 222 : audit [DBG] from='client.? 172.21.15.73:0/808707302' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:16:11.777 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:11 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:16:10.665373+0000 mgr.smithi029.loxrfu (mgr.14164) 150 : cluster [DBG] pgmap v143: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:16:11.956 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:16:11.957 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:16:13.130 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:13.128+0000 7f8da66f3700 1 -- 172.21.15.73:0/1496483817 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8da00f6510 msgr2=0x7f8da00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:16:13.131 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:13.128+0000 7f8da66f3700 1 --2- 172.21.15.73:0/1496483817 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8da00f6510 0x7f8da00f6930 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 rx=0x7f8d90009fb0 tx=0x7f8d90009b30).stop 2023-12-01T22:16:13.131 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:13.128+0000 7f8da66f3700 1 -- 172.21.15.73:0/1496483817 shutdown_connections 2023-12-01T22:16:13.131 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:13.128+0000 7f8da66f3700 1 --2- 172.21.15.73:0/1496483817 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8da00f6510 0x7f8da00f6930 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:16:13.131 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:13.128+0000 7f8da66f3700 1 -- 172.21.15.73:0/1496483817 >> 172.21.15.73:0/1496483817 conn(0x7f8da00f1c70 msgr2=0x7f8da00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:16:13.131 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:13.128+0000 7f8da66f3700 1 -- 172.21.15.73:0/1496483817 shutdown_connections 2023-12-01T22:16:13.131 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:13.128+0000 7f8da66f3700 1 -- 172.21.15.73:0/1496483817 wait complete. 2023-12-01T22:16:13.131 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:13.128+0000 7f8da66f3700 1 Processor -- start 2023-12-01T22:16:13.131 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:13.128+0000 7f8da66f3700 1 -- start start 2023-12-01T22:16:13.131 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:13.129+0000 7f8da66f3700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8da00f6510 0x7f8da0106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:16:13.131 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:13.129+0000 7f8da66f3700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8da0106c20 con 0x7f8da00f6510 2023-12-01T22:16:13.131 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:13.129+0000 7f8d9ffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8da00f6510 0x7f8da0106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:16:13.131 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:13.129+0000 7f8d9ffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8da00f6510 0x7f8da0106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:39418/0 (socket says 172.21.15.73:39418) 2023-12-01T22:16:13.131 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:13.129+0000 7f8d9ffff700 1 -- 172.21.15.73:0/842414028 learned_addr learned my addr 172.21.15.73:0/842414028 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:16:13.133 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:13.129+0000 7f8d9ffff700 1 -- 172.21.15.73:0/842414028 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d90005040 con 0x7f8da00f6510 2023-12-01T22:16:13.133 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:13.129+0000 7f8d9ffff700 1 --2- 172.21.15.73:0/842414028 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8da00f6510 0x7f8da0106730 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7f8d90013510 tx=0x7f8d9000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:16:13.133 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:13.129+0000 7f8d9cff9700 1 -- 172.21.15.73:0/842414028 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8d90009320 con 0x7f8da00f6510 2023-12-01T22:16:13.133 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:13.130+0000 7f8da66f3700 1 -- 172.21.15.73:0/842414028 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8da0106e20 con 0x7f8da00f6510 2023-12-01T22:16:13.133 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:13.130+0000 7f8d9cff9700 1 -- 172.21.15.73:0/842414028 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8d90013f00 con 0x7f8da00f6510 2023-12-01T22:16:13.133 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:13.130+0000 7f8d9cff9700 1 -- 172.21.15.73:0/842414028 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8d90014d90 con 0x7f8da00f6510 2023-12-01T22:16:13.133 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:13.130+0000 7f8da66f3700 1 -- 172.21.15.73:0/842414028 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8da01072b0 con 0x7f8da00f6510 2023-12-01T22:16:13.133 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:13.130+0000 7f8d9cff9700 1 -- 172.21.15.73:0/842414028 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8d900095f0 con 0x7f8da00f6510 2023-12-01T22:16:13.133 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:13.130+0000 7f8da66f3700 1 -- 172.21.15.73:0/842414028 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8da0045ad0 con 0x7f8da00f6510 2023-12-01T22:16:13.134 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:13.130+0000 7f8d9cff9700 1 --2- 172.21.15.73:0/842414028 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f8d8802ff60 0x7f8d88032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:16:13.134 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:13.131+0000 7f8d9cff9700 1 -- 172.21.15.73:0/842414028 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8d90040fe0 con 0x7f8da00f6510 2023-12-01T22:16:13.134 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:13.131+0000 7f8d9f7fe700 1 --2- 172.21.15.73:0/842414028 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f8d8802ff60 0x7f8d88032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:16:13.134 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:13.131+0000 7f8d9f7fe700 1 --2- 172.21.15.73:0/842414028 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f8d8802ff60 0x7f8d88032420 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f8d940096a0 tx=0x7f8d94006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:16:13.137 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:13.135+0000 7f8d9cff9700 1 -- 172.21.15.73:0/842414028 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8d9001d7f0 con 0x7f8da00f6510 2023-12-01T22:16:13.340 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:13.339+0000 7f8da66f3700 1 -- 172.21.15.73:0/842414028 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8da00fbf60 con 0x7f8da00f6510 2023-12-01T22:16:13.341 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:13.339+0000 7f8d9cff9700 1 -- 172.21.15.73:0/842414028 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8d90010230 con 0x7f8da00f6510 2023-12-01T22:16:13.341 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:16:13.341 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:16:13.343 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:13.341+0000 7f8da66f3700 1 -- 172.21.15.73:0/842414028 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f8d8802ff60 msgr2=0x7f8d88032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:16:13.343 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:13.341+0000 7f8da66f3700 1 --2- 172.21.15.73:0/842414028 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f8d8802ff60 0x7f8d88032420 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f8d940096a0 tx=0x7f8d94006b40).stop 2023-12-01T22:16:13.343 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:13.341+0000 7f8da66f3700 1 -- 172.21.15.73:0/842414028 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8da00f6510 msgr2=0x7f8da0106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:16:13.343 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:13.341+0000 7f8da66f3700 1 --2- 172.21.15.73:0/842414028 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8da00f6510 0x7f8da0106730 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7f8d90013510 tx=0x7f8d9000b040).stop 2023-12-01T22:16:13.344 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:13.341+0000 7f8da66f3700 1 -- 172.21.15.73:0/842414028 shutdown_connections 2023-12-01T22:16:13.344 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:13.341+0000 7f8da66f3700 1 --2- 172.21.15.73:0/842414028 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f8d8802ff60 0x7f8d88032420 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:16:13.344 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:13.341+0000 7f8da66f3700 1 --2- 172.21.15.73:0/842414028 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8da00f6510 0x7f8da0106730 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:16:13.344 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:13.341+0000 7f8da66f3700 1 -- 172.21.15.73:0/842414028 >> 172.21.15.73:0/842414028 conn(0x7f8da00f1c70 msgr2=0x7f8da00f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:16:13.344 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:13.341+0000 7f8da66f3700 1 -- 172.21.15.73:0/842414028 shutdown_connections 2023-12-01T22:16:13.344 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:13.341+0000 7f8da66f3700 1 -- 172.21.15.73:0/842414028 wait complete. 2023-12-01T22:16:13.344 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:16:14.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:13 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:16:12.665602+0000 mgr.smithi029.loxrfu (mgr.14164) 151 : cluster [DBG] pgmap v144: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:16:14.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:13 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:16:12.780670+0000 mon.smithi029 (mon.0) 223 : audit [INF] from='mgr.14164 172.21.15.29:0/3195927111' entity='mgr.smithi029.loxrfu' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi029.loxrfu/mirror_snapshot_schedule"}]: dispatch 2023-12-01T22:16:14.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:13 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:16:12.784373+0000 mon.smithi029 (mon.0) 224 : audit [INF] from='mgr.14164 172.21.15.29:0/3195927111' entity='mgr.smithi029.loxrfu' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi029.loxrfu/trash_purge_schedule"}]: dispatch 2023-12-01T22:16:14.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:13 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:16:13.340764+0000 mon.smithi029 (mon.0) 225 : audit [DBG] from='client.? 172.21.15.73:0/842414028' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:16:14.752 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:16:14.753 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:16:15.902 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:15.898+0000 7f8597331700 1 -- 172.21.15.73:0/605426196 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85900f5330 msgr2=0x7f85900f5750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:16:15.902 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:15.898+0000 7f8597331700 1 --2- 172.21.15.73:0/605426196 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85900f5330 0x7f85900f5750 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0x7f8580009fb0 tx=0x7f8580009b30).stop 2023-12-01T22:16:15.902 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:15.898+0000 7f8597331700 1 -- 172.21.15.73:0/605426196 shutdown_connections 2023-12-01T22:16:15.902 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:15.898+0000 7f8597331700 1 --2- 172.21.15.73:0/605426196 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85900f5330 0x7f85900f5750 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:16:15.902 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:15.898+0000 7f8597331700 1 -- 172.21.15.73:0/605426196 >> 172.21.15.73:0/605426196 conn(0x7f85900f0a90 msgr2=0x7f85900f2eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:16:15.902 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:15.898+0000 7f8597331700 1 -- 172.21.15.73:0/605426196 shutdown_connections 2023-12-01T22:16:15.902 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:15.898+0000 7f8597331700 1 -- 172.21.15.73:0/605426196 wait complete. 2023-12-01T22:16:15.902 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:15.899+0000 7f8597331700 1 Processor -- start 2023-12-01T22:16:15.902 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:15.899+0000 7f8597331700 1 -- start start 2023-12-01T22:16:15.903 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:15.899+0000 7f8597331700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85900f5330 0x7f85901069c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:16:15.903 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:15.899+0000 7f8597331700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8590106eb0 con 0x7f85900f5330 2023-12-01T22:16:15.903 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:15.899+0000 7f85950cd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85900f5330 0x7f85901069c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:16:15.903 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:15.899+0000 7f85950cd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85900f5330 0x7f85901069c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:44226/0 (socket says 172.21.15.73:44226) 2023-12-01T22:16:15.903 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:15.899+0000 7f85950cd700 1 -- 172.21.15.73:0/296971253 learned_addr learned my addr 172.21.15.73:0/296971253 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:16:15.903 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:15.899+0000 7f85950cd700 1 -- 172.21.15.73:0/296971253 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8580005040 con 0x7f85900f5330 2023-12-01T22:16:15.903 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:15.900+0000 7f85950cd700 1 --2- 172.21.15.73:0/296971253 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85900f5330 0x7f85901069c0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7f8580013510 tx=0x7f858000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:16:15.903 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:15.900+0000 7f8585ffb700 1 -- 172.21.15.73:0/296971253 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8580009320 con 0x7f85900f5330 2023-12-01T22:16:15.903 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:15.900+0000 7f8597331700 1 -- 172.21.15.73:0/296971253 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85901070b0 con 0x7f85900f5330 2023-12-01T22:16:15.903 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:15.900+0000 7f8585ffb700 1 -- 172.21.15.73:0/296971253 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8580013f00 con 0x7f85900f5330 2023-12-01T22:16:15.903 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:15.900+0000 7f8585ffb700 1 -- 172.21.15.73:0/296971253 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8580014d90 con 0x7f85900f5330 2023-12-01T22:16:15.903 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:15.900+0000 7f8597331700 1 -- 172.21.15.73:0/296971253 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8590107540 con 0x7f85900f5330 2023-12-01T22:16:15.904 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:15.901+0000 7f8597331700 1 -- 172.21.15.73:0/296971253 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8590045ad0 con 0x7f85900f5330 2023-12-01T22:16:15.904 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:15.901+0000 7f8585ffb700 1 -- 172.21.15.73:0/296971253 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f85800095f0 con 0x7f85900f5330 2023-12-01T22:16:15.904 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:15.901+0000 7f8585ffb700 1 --2- 172.21.15.73:0/296971253 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f857c02ff60 0x7f857c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:16:15.904 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:15.901+0000 7f8585ffb700 1 -- 172.21.15.73:0/296971253 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8580040fc0 con 0x7f85900f5330 2023-12-01T22:16:15.905 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:15.901+0000 7f85948cc700 1 --2- 172.21.15.73:0/296971253 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f857c02ff60 0x7f857c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:16:15.905 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:15.902+0000 7f85948cc700 1 --2- 172.21.15.73:0/296971253 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f857c02ff60 0x7f857c032420 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f858c0096a0 tx=0x7f858c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:16:15.908 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:15.906+0000 7f8585ffb700 1 -- 172.21.15.73:0/296971253 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8580014420 con 0x7f85900f5330 2023-12-01T22:16:16.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:16:14.665843+0000 mgr.smithi029.loxrfu (mgr.14164) 152 : cluster [DBG] pgmap v145: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:16:16.114 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:16.112+0000 7f8597331700 1 -- 172.21.15.73:0/296971253 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f859005ea50 con 0x7f85900f5330 2023-12-01T22:16:16.115 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:16.113+0000 7f8585ffb700 1 -- 172.21.15.73:0/296971253 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8580010230 con 0x7f85900f5330 2023-12-01T22:16:16.115 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:16:16.115 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:16:16.116 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:16.114+0000 7f8597331700 1 -- 172.21.15.73:0/296971253 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f857c02ff60 msgr2=0x7f857c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:16:16.116 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:16.114+0000 7f8597331700 1 --2- 172.21.15.73:0/296971253 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f857c02ff60 0x7f857c032420 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f858c0096a0 tx=0x7f858c006b40).stop 2023-12-01T22:16:16.117 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:16.114+0000 7f8597331700 1 -- 172.21.15.73:0/296971253 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85900f5330 msgr2=0x7f85901069c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:16:16.117 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:16.114+0000 7f8597331700 1 --2- 172.21.15.73:0/296971253 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85900f5330 0x7f85901069c0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7f8580013510 tx=0x7f858000b040).stop 2023-12-01T22:16:16.117 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:16.115+0000 7f8597331700 1 -- 172.21.15.73:0/296971253 shutdown_connections 2023-12-01T22:16:16.117 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:16.115+0000 7f8597331700 1 --2- 172.21.15.73:0/296971253 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f857c02ff60 0x7f857c032420 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:16:16.117 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:16.115+0000 7f8597331700 1 --2- 172.21.15.73:0/296971253 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85900f5330 0x7f85901069c0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:16:16.117 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:16.115+0000 7f8597331700 1 -- 172.21.15.73:0/296971253 >> 172.21.15.73:0/296971253 conn(0x7f85900f0a90 msgr2=0x7f85900f18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:16:16.117 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:16.115+0000 7f8597331700 1 -- 172.21.15.73:0/296971253 shutdown_connections 2023-12-01T22:16:16.117 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:16.115+0000 7f8597331700 1 -- 172.21.15.73:0/296971253 wait complete. 2023-12-01T22:16:16.117 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:16:17.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:16 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:16:16.114431+0000 mon.smithi029 (mon.0) 226 : audit [DBG] from='client.? 172.21.15.73:0/296971253' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:16:17.503 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:16:17.503 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:16:18.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:17 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:16:16.666127+0000 mgr.smithi029.loxrfu (mgr.14164) 153 : cluster [DBG] pgmap v146: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:16:18.695 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:18.693+0000 7fd9bd32a700 1 -- 172.21.15.73:0/3697340207 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9b8060280 msgr2=0x7fd9b80f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:16:18.695 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:18.693+0000 7fd9bd32a700 1 --2- 172.21.15.73:0/3697340207 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9b8060280 0x7fd9b80f90f0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 rx=0x7fd9ac009fb0 tx=0x7fd9ac009b30).stop 2023-12-01T22:16:18.695 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:18.693+0000 7fd9bd32a700 1 -- 172.21.15.73:0/3697340207 shutdown_connections 2023-12-01T22:16:18.695 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:18.693+0000 7fd9bd32a700 1 --2- 172.21.15.73:0/3697340207 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9b8060280 0x7fd9b80f90f0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:16:18.695 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:18.693+0000 7fd9bd32a700 1 -- 172.21.15.73:0/3697340207 >> 172.21.15.73:0/3697340207 conn(0x7fd9b80f0b70 msgr2=0x7fd9b80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:16:18.695 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:18.693+0000 7fd9bd32a700 1 -- 172.21.15.73:0/3697340207 shutdown_connections 2023-12-01T22:16:18.695 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:18.693+0000 7fd9bd32a700 1 -- 172.21.15.73:0/3697340207 wait complete. 2023-12-01T22:16:18.697 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:18.694+0000 7fd9bd32a700 1 Processor -- start 2023-12-01T22:16:18.697 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:18.694+0000 7fd9bd32a700 1 -- start start 2023-12-01T22:16:18.697 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:18.694+0000 7fd9bd32a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9b8060280 0x7fd9b81066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:16:18.697 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:18.694+0000 7fd9bd32a700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9b8106bd0 con 0x7fd9b8060280 2023-12-01T22:16:18.698 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:18.694+0000 7fd9b6ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9b8060280 0x7fd9b81066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:16:18.698 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:18.694+0000 7fd9b6ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9b8060280 0x7fd9b81066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:44240/0 (socket says 172.21.15.73:44240) 2023-12-01T22:16:18.698 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:18.694+0000 7fd9b6ffd700 1 -- 172.21.15.73:0/660613174 learned_addr learned my addr 172.21.15.73:0/660613174 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:16:18.698 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:18.694+0000 7fd9b6ffd700 1 -- 172.21.15.73:0/660613174 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9ac005040 con 0x7fd9b8060280 2023-12-01T22:16:18.698 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:18.694+0000 7fd9b6ffd700 1 --2- 172.21.15.73:0/660613174 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9b8060280 0x7fd9b81066e0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7fd9ac013510 tx=0x7fd9ac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:16:18.698 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:18.695+0000 7fd99f7fe700 1 -- 172.21.15.73:0/660613174 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd9ac009320 con 0x7fd9b8060280 2023-12-01T22:16:18.698 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:18.695+0000 7fd99f7fe700 1 -- 172.21.15.73:0/660613174 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd9ac013f00 con 0x7fd9b8060280 2023-12-01T22:16:18.698 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:18.695+0000 7fd9bd32a700 1 -- 172.21.15.73:0/660613174 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9b8106dd0 con 0x7fd9b8060280 2023-12-01T22:16:18.698 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:18.695+0000 7fd99f7fe700 1 -- 172.21.15.73:0/660613174 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd9ac014dc0 con 0x7fd9b8060280 2023-12-01T22:16:18.698 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:18.695+0000 7fd9bd32a700 1 -- 172.21.15.73:0/660613174 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9b8107260 con 0x7fd9b8060280 2023-12-01T22:16:18.698 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:18.695+0000 7fd99f7fe700 1 -- 172.21.15.73:0/660613174 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd9ac0095f0 con 0x7fd9b8060280 2023-12-01T22:16:18.698 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:18.695+0000 7fd9bd32a700 1 -- 172.21.15.73:0/660613174 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9b8045ad0 con 0x7fd9b8060280 2023-12-01T22:16:18.698 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:18.696+0000 7fd99f7fe700 1 --2- 172.21.15.73:0/660613174 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fd9a002ff60 0x7fd9a0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:16:18.698 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:18.696+0000 7fd99f7fe700 1 -- 172.21.15.73:0/660613174 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd9ac0410c0 con 0x7fd9b8060280 2023-12-01T22:16:18.698 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:18.696+0000 7fd9b67fc700 1 --2- 172.21.15.73:0/660613174 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fd9a002ff60 0x7fd9a0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:16:18.699 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:18.696+0000 7fd9b67fc700 1 --2- 172.21.15.73:0/660613174 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fd9a002ff60 0x7fd9a0032420 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7fd9a80096a0 tx=0x7fd9a8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:16:18.702 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:18.701+0000 7fd99f7fe700 1 -- 172.21.15.73:0/660613174 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd9ac014420 con 0x7fd9b8060280 2023-12-01T22:16:18.911 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:18.909+0000 7fd9bd32a700 1 -- 172.21.15.73:0/660613174 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd9b8108110 con 0x7fd9b8060280 2023-12-01T22:16:18.911 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:18.909+0000 7fd99f7fe700 1 -- 172.21.15.73:0/660613174 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd9ac010230 con 0x7fd9b8060280 2023-12-01T22:16:18.911 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:16:18.911 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:16:18.912 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:18.911+0000 7fd9bd32a700 1 -- 172.21.15.73:0/660613174 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fd9a002ff60 msgr2=0x7fd9a0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:16:18.913 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:18.911+0000 7fd9bd32a700 1 --2- 172.21.15.73:0/660613174 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fd9a002ff60 0x7fd9a0032420 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7fd9a80096a0 tx=0x7fd9a8006b40).stop 2023-12-01T22:16:18.913 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:18.911+0000 7fd9bd32a700 1 -- 172.21.15.73:0/660613174 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9b8060280 msgr2=0x7fd9b81066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:16:18.913 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:18.911+0000 7fd9bd32a700 1 --2- 172.21.15.73:0/660613174 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9b8060280 0x7fd9b81066e0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7fd9ac013510 tx=0x7fd9ac00b040).stop 2023-12-01T22:16:18.913 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:18.911+0000 7fd9bd32a700 1 -- 172.21.15.73:0/660613174 shutdown_connections 2023-12-01T22:16:18.913 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:18.911+0000 7fd9bd32a700 1 --2- 172.21.15.73:0/660613174 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fd9a002ff60 0x7fd9a0032420 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:16:18.913 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:18.911+0000 7fd9bd32a700 1 --2- 172.21.15.73:0/660613174 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9b8060280 0x7fd9b81066e0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:16:18.913 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:18.911+0000 7fd9bd32a700 1 -- 172.21.15.73:0/660613174 >> 172.21.15.73:0/660613174 conn(0x7fd9b80f0b70 msgr2=0x7fd9b80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:16:18.913 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:18.911+0000 7fd9bd32a700 1 -- 172.21.15.73:0/660613174 shutdown_connections 2023-12-01T22:16:18.913 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:18.911+0000 7fd9bd32a700 1 -- 172.21.15.73:0/660613174 wait complete. 2023-12-01T22:16:18.913 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:16:20.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:19 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:16:18.666448+0000 mgr.smithi029.loxrfu (mgr.14164) 154 : cluster [DBG] pgmap v147: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:16:20.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:19 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:16:18.910660+0000 mon.smithi029 (mon.0) 227 : audit [DBG] from='client.? 172.21.15.73:0/660613174' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:16:20.295 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:16:20.295 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:16:21.450 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:21.448+0000 7f475ebd9700 1 -- 172.21.15.73:0/1298633491 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47580f78e0 msgr2=0x7f47580f7d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:16:21.450 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:21.448+0000 7f475ebd9700 1 --2- 172.21.15.73:0/1298633491 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47580f78e0 0x7f47580f7d00 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 rx=0x7f4754009fb0 tx=0x7f4754009b30).stop 2023-12-01T22:16:21.450 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:21.448+0000 7f475ebd9700 1 -- 172.21.15.73:0/1298633491 shutdown_connections 2023-12-01T22:16:21.450 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:21.448+0000 7f475ebd9700 1 --2- 172.21.15.73:0/1298633491 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47580f78e0 0x7f47580f7d00 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:16:21.450 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:21.448+0000 7f475ebd9700 1 -- 172.21.15.73:0/1298633491 >> 172.21.15.73:0/1298633491 conn(0x7f47580f0120 msgr2=0x7f47580f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:16:21.450 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:21.448+0000 7f475ebd9700 1 -- 172.21.15.73:0/1298633491 shutdown_connections 2023-12-01T22:16:21.450 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:21.448+0000 7f475ebd9700 1 -- 172.21.15.73:0/1298633491 wait complete. 2023-12-01T22:16:21.453 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:21.449+0000 7f475ebd9700 1 Processor -- start 2023-12-01T22:16:21.453 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:21.449+0000 7f475ebd9700 1 -- start start 2023-12-01T22:16:21.453 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:21.449+0000 7f475ebd9700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47580f78e0 0x7f47581086e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:16:21.453 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:21.449+0000 7f475ebd9700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4758108bd0 con 0x7f47580f78e0 2023-12-01T22:16:21.453 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:21.449+0000 7f475dbd7700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47580f78e0 0x7f47581086e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:16:21.453 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:21.449+0000 7f475dbd7700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47580f78e0 0x7f47581086e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:44268/0 (socket says 172.21.15.73:44268) 2023-12-01T22:16:21.453 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:21.449+0000 7f475dbd7700 1 -- 172.21.15.73:0/1721941034 learned_addr learned my addr 172.21.15.73:0/1721941034 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:16:21.453 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:21.449+0000 7f475dbd7700 1 -- 172.21.15.73:0/1721941034 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4754005040 con 0x7f47580f78e0 2023-12-01T22:16:21.453 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:21.450+0000 7f475dbd7700 1 --2- 172.21.15.73:0/1721941034 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47580f78e0 0x7f47581086e0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7f4754013510 tx=0x7f475400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:16:21.453 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:21.450+0000 7f474e7fc700 1 -- 172.21.15.73:0/1721941034 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4754009320 con 0x7f47580f78e0 2023-12-01T22:16:21.453 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:21.450+0000 7f475ebd9700 1 -- 172.21.15.73:0/1721941034 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4758108dd0 con 0x7f47580f78e0 2023-12-01T22:16:21.454 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:21.450+0000 7f474e7fc700 1 -- 172.21.15.73:0/1721941034 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4754013f00 con 0x7f47580f78e0 2023-12-01T22:16:21.454 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:21.450+0000 7f474e7fc700 1 -- 172.21.15.73:0/1721941034 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4754014dc0 con 0x7f47580f78e0 2023-12-01T22:16:21.454 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:21.450+0000 7f475ebd9700 1 -- 172.21.15.73:0/1721941034 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4758109260 con 0x7f47580f78e0 2023-12-01T22:16:21.454 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:21.451+0000 7f474e7fc700 1 -- 172.21.15.73:0/1721941034 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f47540095f0 con 0x7f47580f78e0 2023-12-01T22:16:21.454 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:21.451+0000 7f475ebd9700 1 -- 172.21.15.73:0/1721941034 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4758045ad0 con 0x7f47580f78e0 2023-12-01T22:16:21.454 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:21.451+0000 7f474e7fc700 1 --2- 172.21.15.73:0/1721941034 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f474402ff60 0x7f4744032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:16:21.454 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:21.451+0000 7f474e7fc700 1 -- 172.21.15.73:0/1721941034 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4754041000 con 0x7f47580f78e0 2023-12-01T22:16:21.454 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:21.451+0000 7f475d3d6700 1 --2- 172.21.15.73:0/1721941034 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f474402ff60 0x7f4744032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:16:21.454 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:21.451+0000 7f475d3d6700 1 --2- 172.21.15.73:0/1721941034 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f474402ff60 0x7f4744032420 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f47480096a0 tx=0x7f4748006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:16:21.457 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:21.455+0000 7f474e7fc700 1 -- 172.21.15.73:0/1721941034 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4754014420 con 0x7f47580f78e0 2023-12-01T22:16:21.664 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:21.663+0000 7f475ebd9700 1 -- 172.21.15.73:0/1721941034 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f475810a110 con 0x7f47580f78e0 2023-12-01T22:16:21.665 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:21.663+0000 7f474e7fc700 1 -- 172.21.15.73:0/1721941034 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4754010230 con 0x7f47580f78e0 2023-12-01T22:16:21.665 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:16:21.665 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:16:21.667 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:21.665+0000 7f475ebd9700 1 -- 172.21.15.73:0/1721941034 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f474402ff60 msgr2=0x7f4744032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:16:21.667 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:21.665+0000 7f475ebd9700 1 --2- 172.21.15.73:0/1721941034 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f474402ff60 0x7f4744032420 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f47480096a0 tx=0x7f4748006b40).stop 2023-12-01T22:16:21.667 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:21.665+0000 7f475ebd9700 1 -- 172.21.15.73:0/1721941034 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47580f78e0 msgr2=0x7f47581086e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:16:21.667 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:21.665+0000 7f475ebd9700 1 --2- 172.21.15.73:0/1721941034 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47580f78e0 0x7f47581086e0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7f4754013510 tx=0x7f475400b040).stop 2023-12-01T22:16:21.667 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:21.665+0000 7f475ebd9700 1 -- 172.21.15.73:0/1721941034 shutdown_connections 2023-12-01T22:16:21.667 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:21.665+0000 7f475ebd9700 1 --2- 172.21.15.73:0/1721941034 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f474402ff60 0x7f4744032420 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:16:21.667 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:21.665+0000 7f475ebd9700 1 --2- 172.21.15.73:0/1721941034 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47580f78e0 0x7f47581086e0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:16:21.667 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:21.665+0000 7f475ebd9700 1 -- 172.21.15.73:0/1721941034 >> 172.21.15.73:0/1721941034 conn(0x7f47580f0120 msgr2=0x7f47580f0bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:16:21.667 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:21.665+0000 7f475ebd9700 1 -- 172.21.15.73:0/1721941034 shutdown_connections 2023-12-01T22:16:21.667 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:21.665+0000 7f475ebd9700 1 -- 172.21.15.73:0/1721941034 wait complete. 2023-12-01T22:16:21.668 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:16:21.776 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:21 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:16:20.666771+0000 mgr.smithi029.loxrfu (mgr.14164) 155 : cluster [DBG] pgmap v148: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:16:22.776 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:22 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:16:21.664569+0000 mon.smithi029 (mon.0) 228 : audit [DBG] from='client.? 172.21.15.73:0/1721941034' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:16:23.066 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:16:23.066 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:16:23.528 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:23 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:16:22.667047+0000 mgr.smithi029.loxrfu (mgr.14164) 156 : cluster [DBG] pgmap v149: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:16:23.528 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:23 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cephadm 2023-12-01T22:16:22.877145+0000 mgr.smithi029.loxrfu (mgr.14164) 157 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon alertmanager.smithi029 ... 2023-12-01T22:16:23.528 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:23 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi029 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/alertmanager 2023-12-01T22:16:23.528 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:23 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2023-12-01T22:16:23.528 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:23 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: stat: stderr time="2023-12-01T22:12:01Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-12-01T22:16:23.528 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:23 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: stat: stderr time="2023-12-01T22:12:38Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-12-01T22:16:23.528 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:23 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: stat: stderr time="2023-12-01T22:13:16Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-12-01T22:16:23.528 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:23 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2023-12-01T22:16:23.528 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:23 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi029 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/prometheus 2023-12-01T22:16:23.528 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:23 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2023-12-01T22:16:23.529 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:23 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: stat: stderr time="2023-12-01T22:14:30Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-12-01T22:16:23.529 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:23 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: stat: stderr time="2023-12-01T22:15:07Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-12-01T22:16:23.529 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:23 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: stat: stderr time="2023-12-01T22:15:45Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-12-01T22:16:23.529 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:23 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2023-12-01T22:16:23.529 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:23 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: Traceback (most recent call last): 2023-12-01T22:16:23.529 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:23 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: File "/var/lib/ceph/5558ab6c-9096-11ee-95a2-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2023-12-01T22:16:23.529 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:23 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: main() 2023-12-01T22:16:23.529 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:23 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: File "/var/lib/ceph/5558ab6c-9096-11ee-95a2-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2023-12-01T22:16:23.529 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:23 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: r = ctx.func(ctx) 2023-12-01T22:16:23.529 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:23 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: File "/var/lib/ceph/5558ab6c-9096-11ee-95a2-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2023-12-01T22:16:23.529 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:23 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: return func(ctx) 2023-12-01T22:16:23.529 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:23 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: File "/var/lib/ceph/5558ab6c-9096-11ee-95a2-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2023-12-01T22:16:23.529 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:23 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2023-12-01T22:16:23.529 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:23 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: File "/var/lib/ceph/5558ab6c-9096-11ee-95a2-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4191, in extract_uid_gid_monitoring 2023-12-01T22:16:23.529 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:23 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: uid, gid = extract_uid_gid(ctx, file_path=['/etc/alertmanager', '/etc/prometheus']) 2023-12-01T22:16:23.530 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:23 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: File "/var/lib/ceph/5558ab6c-9096-11ee-95a2-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2023-12-01T22:16:23.530 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:23 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: raise RuntimeError('uid/gid not found') 2023-12-01T22:16:23.530 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:23 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: RuntimeError: uid/gid not found 2023-12-01T22:16:23.530 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:23 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: Traceback (most recent call last): 2023-12-01T22:16:23.530 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:23 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2023-12-01T22:16:23.530 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:23 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: yield (conn, connr) 2023-12-01T22:16:23.530 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:23 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2023-12-01T22:16:23.530 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:23 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: code, '\n'.join(err))) 2023-12-01T22:16:23.530 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:23 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon alertmanager.smithi029 ... 2023-12-01T22:16:23.530 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:23 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi029 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/alertmanager 2023-12-01T22:16:23.530 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:23 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2023-12-01T22:16:23.530 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:23 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: stat: stderr time="2023-12-01T22:12:01Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-12-01T22:16:23.530 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:23 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: stat: stderr time="2023-12-01T22:12:38Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-12-01T22:16:23.530 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:23 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: stat: stderr time="2023-12-01T22:13:16Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-12-01T22:16:23.531 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:23 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2023-12-01T22:16:23.531 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:23 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi029 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/prometheus 2023-12-01T22:16:23.531 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:23 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2023-12-01T22:16:23.531 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:23 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: stat: stderr time="2023-12-01T22:14:30Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-12-01T22:16:23.531 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:23 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: stat: stderr time="2023-12-01T22:15:07Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-12-01T22:16:23.531 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:23 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: stat: stderr time="2023-12-01T22:15:45Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-12-01T22:16:23.531 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:23 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2023-12-01T22:16:23.531 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:23 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: Traceback (most recent call last): 2023-12-01T22:16:23.531 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:23 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: File "/var/lib/ceph/5558ab6c-9096-11ee-95a2-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2023-12-01T22:16:23.531 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:23 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: main() 2023-12-01T22:16:23.531 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:23 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: File "/var/lib/ceph/5558ab6c-9096-11ee-95a2-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2023-12-01T22:16:23.531 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:23 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: r = ctx.func(ctx) 2023-12-01T22:16:23.531 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:23 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: File "/var/lib/ceph/5558ab6c-9096-11ee-95a2-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2023-12-01T22:16:23.531 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:23 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: return func(ctx) 2023-12-01T22:16:23.531 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:23 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: File "/var/lib/ceph/5558ab6c-9096-11ee-95a2-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2023-12-01T22:16:23.532 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:23 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2023-12-01T22:16:23.532 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:23 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: File "/var/lib/ceph/5558ab6c-9096-11ee-95a2-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4191, in extract_uid_gid_monitoring 2023-12-01T22:16:23.532 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:23 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: uid, gid = extract_uid_gid(ctx, file_path=['/etc/alertmanager', '/etc/prometheus']) 2023-12-01T22:16:23.532 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:23 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: File "/var/lib/ceph/5558ab6c-9096-11ee-95a2-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2023-12-01T22:16:23.532 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:23 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: raise RuntimeError('uid/gid not found') 2023-12-01T22:16:23.532 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:23 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: RuntimeError: uid/gid not found 2023-12-01T22:16:23.532 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:23 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:16:22.878839+0000 mon.smithi029 (mon.0) 229 : audit [INF] from='mgr.14164 172.21.15.29:0/3195927111' entity='mgr.smithi029.loxrfu' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi029", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2023-12-01T22:16:23.532 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:23 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:16:22.881321+0000 mon.smithi029 (mon.0) 230 : audit [INF] from='mgr.14164 172.21.15.29:0/3195927111' entity='mgr.smithi029.loxrfu' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi029", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2023-12-01T22:16:23.532 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:23 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:16:22.882220+0000 mon.smithi029 (mon.0) 231 : audit [DBG] from='mgr.14164 172.21.15.29:0/3195927111' entity='mgr.smithi029.loxrfu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-01T22:16:23.532 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:23 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cephadm 2023-12-01T22:16:22.882835+0000 mgr.smithi029.loxrfu (mgr.14164) 158 : cephadm [INF] Deploying daemon crash.smithi029 on smithi029 2023-12-01T22:16:24.295 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:24.291+0000 7f7e2b9ca700 1 -- 172.21.15.73:0/3852734614 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e240f5410 msgr2=0x7f7e240f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:16:24.295 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:24.291+0000 7f7e2b9ca700 1 --2- 172.21.15.73:0/3852734614 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e240f5410 0x7f7e240f5830 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0x7f7e18009fb0 tx=0x7f7e18009b30).stop 2023-12-01T22:16:24.295 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:24.291+0000 7f7e2b9ca700 1 -- 172.21.15.73:0/3852734614 shutdown_connections 2023-12-01T22:16:24.295 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:24.291+0000 7f7e2b9ca700 1 --2- 172.21.15.73:0/3852734614 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e240f5410 0x7f7e240f5830 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:16:24.295 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:24.291+0000 7f7e2b9ca700 1 -- 172.21.15.73:0/3852734614 >> 172.21.15.73:0/3852734614 conn(0x7f7e240f0b70 msgr2=0x7f7e240f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:16:24.295 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:24.291+0000 7f7e2b9ca700 1 -- 172.21.15.73:0/3852734614 shutdown_connections 2023-12-01T22:16:24.295 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:24.291+0000 7f7e2b9ca700 1 -- 172.21.15.73:0/3852734614 wait complete. 2023-12-01T22:16:24.295 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:24.292+0000 7f7e2b9ca700 1 Processor -- start 2023-12-01T22:16:24.295 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:24.292+0000 7f7e2b9ca700 1 -- start start 2023-12-01T22:16:24.295 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:24.292+0000 7f7e2b9ca700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e240f5410 0x7f7e24109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:16:24.295 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:24.292+0000 7f7e2b9ca700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e24060240 con 0x7f7e240f5410 2023-12-01T22:16:24.296 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:24.292+0000 7f7e29766700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e240f5410 0x7f7e24109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:16:24.296 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:24.292+0000 7f7e29766700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e240f5410 0x7f7e24109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:44290/0 (socket says 172.21.15.73:44290) 2023-12-01T22:16:24.296 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:24.292+0000 7f7e29766700 1 -- 172.21.15.73:0/2788688762 learned_addr learned my addr 172.21.15.73:0/2788688762 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:16:24.296 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:24.293+0000 7f7e29766700 1 -- 172.21.15.73:0/2788688762 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e18005040 con 0x7f7e240f5410 2023-12-01T22:16:24.296 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:24.293+0000 7f7e29766700 1 --2- 172.21.15.73:0/2788688762 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e240f5410 0x7f7e24109430 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7f7e18013510 tx=0x7f7e1800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:16:24.296 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:24.293+0000 7f7e15ffb700 1 -- 172.21.15.73:0/2788688762 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7e18009320 con 0x7f7e240f5410 2023-12-01T22:16:24.296 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:24.293+0000 7f7e2b9ca700 1 -- 172.21.15.73:0/2788688762 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e24060440 con 0x7f7e240f5410 2023-12-01T22:16:24.296 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:24.293+0000 7f7e15ffb700 1 -- 172.21.15.73:0/2788688762 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7e18013f00 con 0x7f7e240f5410 2023-12-01T22:16:24.296 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:24.293+0000 7f7e15ffb700 1 -- 172.21.15.73:0/2788688762 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7e18014d90 con 0x7f7e240f5410 2023-12-01T22:16:24.296 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:24.293+0000 7f7e2b9ca700 1 -- 172.21.15.73:0/2788688762 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e24109ae0 con 0x7f7e240f5410 2023-12-01T22:16:24.297 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:24.294+0000 7f7e15ffb700 1 -- 172.21.15.73:0/2788688762 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7e180095f0 con 0x7f7e240f5410 2023-12-01T22:16:24.297 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:24.294+0000 7f7e2b9ca700 1 -- 172.21.15.73:0/2788688762 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e24045ad0 con 0x7f7e240f5410 2023-12-01T22:16:24.297 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:24.294+0000 7f7e15ffb700 1 --2- 172.21.15.73:0/2788688762 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f7e1002ff60 0x7f7e10032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:16:24.297 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:24.294+0000 7f7e15ffb700 1 -- 172.21.15.73:0/2788688762 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7e18040fc0 con 0x7f7e240f5410 2023-12-01T22:16:24.297 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:24.294+0000 7f7e28f65700 1 --2- 172.21.15.73:0/2788688762 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f7e1002ff60 0x7f7e10032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:16:24.297 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:24.295+0000 7f7e28f65700 1 --2- 172.21.15.73:0/2788688762 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f7e1002ff60 0x7f7e10032420 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f7e200096a0 tx=0x7f7e20006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:16:24.300 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:24.300+0000 7f7e15ffb700 1 -- 172.21.15.73:0/2788688762 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7e18014420 con 0x7f7e240f5410 2023-12-01T22:16:24.504 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:24.502+0000 7f7e2b9ca700 1 -- 172.21.15.73:0/2788688762 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7e240fae60 con 0x7f7e240f5410 2023-12-01T22:16:24.505 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:24.503+0000 7f7e15ffb700 1 -- 172.21.15.73:0/2788688762 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7e18010230 con 0x7f7e240f5410 2023-12-01T22:16:24.505 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:16:24.505 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:16:24.506 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:24.505+0000 7f7e2b9ca700 1 -- 172.21.15.73:0/2788688762 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f7e1002ff60 msgr2=0x7f7e10032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:16:24.507 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:24.505+0000 7f7e2b9ca700 1 --2- 172.21.15.73:0/2788688762 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f7e1002ff60 0x7f7e10032420 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f7e200096a0 tx=0x7f7e20006b40).stop 2023-12-01T22:16:24.507 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:24.505+0000 7f7e2b9ca700 1 -- 172.21.15.73:0/2788688762 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e240f5410 msgr2=0x7f7e24109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:16:24.507 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:24.505+0000 7f7e2b9ca700 1 --2- 172.21.15.73:0/2788688762 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e240f5410 0x7f7e24109430 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7f7e18013510 tx=0x7f7e1800b040).stop 2023-12-01T22:16:24.507 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:24.505+0000 7f7e2b9ca700 1 -- 172.21.15.73:0/2788688762 shutdown_connections 2023-12-01T22:16:24.507 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:24.505+0000 7f7e2b9ca700 1 --2- 172.21.15.73:0/2788688762 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f7e1002ff60 0x7f7e10032420 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:16:24.507 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:24.505+0000 7f7e2b9ca700 1 --2- 172.21.15.73:0/2788688762 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e240f5410 0x7f7e24109430 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:16:24.507 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:24.505+0000 7f7e2b9ca700 1 -- 172.21.15.73:0/2788688762 >> 172.21.15.73:0/2788688762 conn(0x7f7e240f0b70 msgr2=0x7f7e240f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:16:24.507 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:24.505+0000 7f7e2b9ca700 1 -- 172.21.15.73:0/2788688762 shutdown_connections 2023-12-01T22:16:24.507 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:24.505+0000 7f7e2b9ca700 1 -- 172.21.15.73:0/2788688762 wait complete. 2023-12-01T22:16:24.508 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:16:25.777 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:25 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:16:24.463103+0000 mon.smithi029 (mon.0) 232 : audit [INF] from='mgr.14164 172.21.15.29:0/3195927111' entity='mgr.smithi029.loxrfu' 2023-12-01T22:16:25.777 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:25 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:16:24.504368+0000 mon.smithi029 (mon.0) 233 : audit [DBG] from='client.? 172.21.15.73:0/2788688762' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:16:25.777 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:25 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:16:24.614352+0000 mon.smithi029 (mon.0) 234 : audit [INF] from='mgr.14164 172.21.15.29:0/3195927111' entity='mgr.smithi029.loxrfu' 2023-12-01T22:16:25.777 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:25 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:16:24.616725+0000 mon.smithi029 (mon.0) 235 : audit [INF] from='mgr.14164 172.21.15.29:0/3195927111' entity='mgr.smithi029.loxrfu' 2023-12-01T22:16:25.777 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:25 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:16:24.617191+0000 mon.smithi029 (mon.0) 236 : audit [INF] from='mgr.14164 172.21.15.29:0/3195927111' entity='mgr.smithi029.loxrfu' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2023-12-01T22:16:25.777 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:25 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:16:24.617659+0000 mgr.smithi029.loxrfu (mgr.14164) 159 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2023-12-01T22:16:25.777 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:25 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:16:24.620271+0000 mon.smithi029 (mon.0) 237 : audit [INF] from='mgr.14164 172.21.15.29:0/3195927111' entity='mgr.smithi029.loxrfu' 2023-12-01T22:16:25.777 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:25 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cephadm 2023-12-01T22:16:24.623430+0000 mgr.smithi029.loxrfu (mgr.14164) 160 : cephadm [INF] Deploying daemon grafana.smithi029 on smithi029 2023-12-01T22:16:25.777 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:25 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:16:24.667359+0000 mgr.smithi029.loxrfu (mgr.14164) 161 : cluster [DBG] pgmap v150: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:16:25.889 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:16:25.889 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:16:27.130 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:27.129+0000 7f81a09eb700 1 -- 172.21.15.73:0/2351644688 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f819c0f5410 msgr2=0x7f819c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:16:27.130 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:27.129+0000 7f81a09eb700 1 --2- 172.21.15.73:0/2351644688 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f819c0f5410 0x7f819c0f5830 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 rx=0x7f8184009fb0 tx=0x7f8184009b30).stop 2023-12-01T22:16:27.130 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:27.129+0000 7f81a09eb700 1 -- 172.21.15.73:0/2351644688 shutdown_connections 2023-12-01T22:16:27.130 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:27.129+0000 7f81a09eb700 1 --2- 172.21.15.73:0/2351644688 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f819c0f5410 0x7f819c0f5830 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:16:27.131 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:27.129+0000 7f81a09eb700 1 -- 172.21.15.73:0/2351644688 >> 172.21.15.73:0/2351644688 conn(0x7f819c0f0b70 msgr2=0x7f819c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:16:27.131 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:27.129+0000 7f81a09eb700 1 -- 172.21.15.73:0/2351644688 shutdown_connections 2023-12-01T22:16:27.131 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:27.129+0000 7f81a09eb700 1 -- 172.21.15.73:0/2351644688 wait complete. 2023-12-01T22:16:27.131 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:27.129+0000 7f81a09eb700 1 Processor -- start 2023-12-01T22:16:27.131 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:27.130+0000 7f81a09eb700 1 -- start start 2023-12-01T22:16:27.131 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:27.130+0000 7f81a09eb700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f819c0f5410 0x7f819c105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:16:27.131 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:27.130+0000 7f81a09eb700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f819c105b20 con 0x7f819c0f5410 2023-12-01T22:16:27.132 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:27.130+0000 7f819a59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f819c0f5410 0x7f819c105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:16:27.133 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:27.130+0000 7f819a59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f819c0f5410 0x7f819c105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:39240/0 (socket says 172.21.15.73:39240) 2023-12-01T22:16:27.133 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:27.130+0000 7f819a59c700 1 -- 172.21.15.73:0/1306583106 learned_addr learned my addr 172.21.15.73:0/1306583106 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:16:27.133 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:27.130+0000 7f819a59c700 1 -- 172.21.15.73:0/1306583106 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8184005040 con 0x7f819c0f5410 2023-12-01T22:16:27.133 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:27.130+0000 7f819a59c700 1 --2- 172.21.15.73:0/1306583106 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f819c0f5410 0x7f819c105630 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7f8184013510 tx=0x7f818400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:16:27.133 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:27.130+0000 7f8192ffd700 1 -- 172.21.15.73:0/1306583106 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8184009320 con 0x7f819c0f5410 2023-12-01T22:16:27.133 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:27.131+0000 7f81a09eb700 1 -- 172.21.15.73:0/1306583106 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f819c105d20 con 0x7f819c0f5410 2023-12-01T22:16:27.133 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:27.131+0000 7f8192ffd700 1 -- 172.21.15.73:0/1306583106 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8184013f00 con 0x7f819c0f5410 2023-12-01T22:16:27.133 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:27.131+0000 7f8192ffd700 1 -- 172.21.15.73:0/1306583106 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8184014d90 con 0x7f819c0f5410 2023-12-01T22:16:27.133 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:27.131+0000 7f81a09eb700 1 -- 172.21.15.73:0/1306583106 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f819c1061b0 con 0x7f819c0f5410 2023-12-01T22:16:27.133 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:27.131+0000 7f81a09eb700 1 -- 172.21.15.73:0/1306583106 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f819c045ad0 con 0x7f819c0f5410 2023-12-01T22:16:27.133 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:27.131+0000 7f8192ffd700 1 -- 172.21.15.73:0/1306583106 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f81840095f0 con 0x7f819c0f5410 2023-12-01T22:16:27.133 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:27.132+0000 7f8192ffd700 1 --2- 172.21.15.73:0/1306583106 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f818802ff60 0x7f8188032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:16:27.133 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:27.132+0000 7f8192ffd700 1 -- 172.21.15.73:0/1306583106 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f81840410c0 con 0x7f819c0f5410 2023-12-01T22:16:27.134 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:27.132+0000 7f8199d9b700 1 --2- 172.21.15.73:0/1306583106 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f818802ff60 0x7f8188032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:16:27.134 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:27.132+0000 7f8199d9b700 1 --2- 172.21.15.73:0/1306583106 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f818802ff60 0x7f8188032420 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f818c0096a0 tx=0x7f818c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:16:27.137 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:27.137+0000 7f8192ffd700 1 -- 172.21.15.73:0/1306583106 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f818401d7f0 con 0x7f819c0f5410 2023-12-01T22:16:27.342 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:27.342+0000 7f81a09eb700 1 -- 172.21.15.73:0/1306583106 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f819c0fae60 con 0x7f819c0f5410 2023-12-01T22:16:27.343 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:27.342+0000 7f8192ffd700 1 -- 172.21.15.73:0/1306583106 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8184010230 con 0x7f819c0f5410 2023-12-01T22:16:27.343 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:16:27.343 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:16:27.344 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:27.344+0000 7f81a09eb700 1 -- 172.21.15.73:0/1306583106 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f818802ff60 msgr2=0x7f8188032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:16:27.345 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:27.344+0000 7f81a09eb700 1 --2- 172.21.15.73:0/1306583106 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f818802ff60 0x7f8188032420 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f818c0096a0 tx=0x7f818c006b40).stop 2023-12-01T22:16:27.345 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:27.344+0000 7f81a09eb700 1 -- 172.21.15.73:0/1306583106 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f819c0f5410 msgr2=0x7f819c105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:16:27.345 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:27.344+0000 7f81a09eb700 1 --2- 172.21.15.73:0/1306583106 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f819c0f5410 0x7f819c105630 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7f8184013510 tx=0x7f818400b040).stop 2023-12-01T22:16:27.345 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:27.344+0000 7f81a09eb700 1 -- 172.21.15.73:0/1306583106 shutdown_connections 2023-12-01T22:16:27.345 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:27.344+0000 7f81a09eb700 1 --2- 172.21.15.73:0/1306583106 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f818802ff60 0x7f8188032420 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:16:27.345 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:27.344+0000 7f81a09eb700 1 --2- 172.21.15.73:0/1306583106 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f819c0f5410 0x7f819c105630 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:16:27.345 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:27.344+0000 7f81a09eb700 1 -- 172.21.15.73:0/1306583106 >> 172.21.15.73:0/1306583106 conn(0x7f819c0f0b70 msgr2=0x7f819c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:16:27.345 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:27.344+0000 7f81a09eb700 1 -- 172.21.15.73:0/1306583106 shutdown_connections 2023-12-01T22:16:27.345 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:27.344+0000 7f81a09eb700 1 -- 172.21.15.73:0/1306583106 wait complete. 2023-12-01T22:16:27.345 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:16:28.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:27 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:16:26.667712+0000 mgr.smithi029.loxrfu (mgr.14164) 162 : cluster [DBG] pgmap v151: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:16:28.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:27 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:16:27.342766+0000 mon.smithi029 (mon.0) 238 : audit [DBG] from='client.? 172.21.15.73:0/1306583106' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:16:28.785 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:16:28.785 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:16:30.004 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:30.002+0000 7fe1b5884700 1 -- 172.21.15.73:0/3369233864 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe1b00f51a0 msgr2=0x7fe1b00f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:16:30.004 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:30.002+0000 7fe1b5884700 1 --2- 172.21.15.73:0/3369233864 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe1b00f51a0 0x7fe1b00f55c0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0x7fe1a4009fb0 tx=0x7fe1a4009b30).stop 2023-12-01T22:16:30.004 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:30.002+0000 7fe1b5884700 1 -- 172.21.15.73:0/3369233864 shutdown_connections 2023-12-01T22:16:30.004 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:30.002+0000 7fe1b5884700 1 --2- 172.21.15.73:0/3369233864 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe1b00f51a0 0x7fe1b00f55c0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:16:30.004 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:30.003+0000 7fe1b5884700 1 -- 172.21.15.73:0/3369233864 >> 172.21.15.73:0/3369233864 conn(0x7fe1b00f09a0 msgr2=0x7fe1b00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:16:30.005 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:30.003+0000 7fe1b5884700 1 -- 172.21.15.73:0/3369233864 shutdown_connections 2023-12-01T22:16:30.005 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:30.003+0000 7fe1b5884700 1 -- 172.21.15.73:0/3369233864 wait complete. 2023-12-01T22:16:30.005 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:30.003+0000 7fe1b5884700 1 Processor -- start 2023-12-01T22:16:30.005 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:30.003+0000 7fe1b5884700 1 -- start start 2023-12-01T22:16:30.005 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:30.004+0000 7fe1b5884700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe1b00f51a0 0x7fe1b010b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:16:30.005 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:30.004+0000 7fe1b5884700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1b0106fc0 con 0x7fe1b00f51a0 2023-12-01T22:16:30.006 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:30.004+0000 7fe1b4882700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe1b00f51a0 0x7fe1b010b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:16:30.007 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:30.004+0000 7fe1b4882700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe1b00f51a0 0x7fe1b010b5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:39260/0 (socket says 172.21.15.73:39260) 2023-12-01T22:16:30.007 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:30.004+0000 7fe1b4882700 1 -- 172.21.15.73:0/147960568 learned_addr learned my addr 172.21.15.73:0/147960568 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:16:30.007 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:30.004+0000 7fe1b4882700 1 -- 172.21.15.73:0/147960568 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1a4005040 con 0x7fe1b00f51a0 2023-12-01T22:16:30.007 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:30.004+0000 7fe1b4882700 1 --2- 172.21.15.73:0/147960568 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe1b00f51a0 0x7fe1b010b5d0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7fe1a4013510 tx=0x7fe1a400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:16:30.007 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:30.005+0000 7fe1ad7fa700 1 -- 172.21.15.73:0/147960568 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe1a4009320 con 0x7fe1b00f51a0 2023-12-01T22:16:30.007 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:30.005+0000 7fe1b5884700 1 -- 172.21.15.73:0/147960568 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1b0060220 con 0x7fe1b00f51a0 2023-12-01T22:16:30.007 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:30.005+0000 7fe1ad7fa700 1 -- 172.21.15.73:0/147960568 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe1a4013f00 con 0x7fe1b00f51a0 2023-12-01T22:16:30.007 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:30.005+0000 7fe1ad7fa700 1 -- 172.21.15.73:0/147960568 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe1a4014dc0 con 0x7fe1b00f51a0 2023-12-01T22:16:30.007 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:30.005+0000 7fe1b5884700 1 -- 172.21.15.73:0/147960568 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1b010bac0 con 0x7fe1b00f51a0 2023-12-01T22:16:30.007 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:30.005+0000 7fe1ad7fa700 1 -- 172.21.15.73:0/147960568 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe1a4014420 con 0x7fe1b00f51a0 2023-12-01T22:16:30.007 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:30.005+0000 7fe1b5884700 1 -- 172.21.15.73:0/147960568 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1b0045ad0 con 0x7fe1b00f51a0 2023-12-01T22:16:30.007 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:30.006+0000 7fe1ad7fa700 1 --2- 172.21.15.73:0/147960568 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fe1a002ff60 0x7fe1a0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:16:30.007 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:30.006+0000 7fe1ad7fa700 1 -- 172.21.15.73:0/147960568 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe1a4041050 con 0x7fe1b00f51a0 2023-12-01T22:16:30.008 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:30.006+0000 7fe1affff700 1 --2- 172.21.15.73:0/147960568 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fe1a002ff60 0x7fe1a0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:16:30.008 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:30.006+0000 7fe1affff700 1 --2- 172.21.15.73:0/147960568 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fe1a002ff60 0x7fe1a0032420 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7fe19c0096a0 tx=0x7fe19c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:16:30.011 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:30.010+0000 7fe1ad7fa700 1 -- 172.21.15.73:0/147960568 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe1a40095f0 con 0x7fe1b00f51a0 2023-12-01T22:16:30.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:29 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:16:28.668082+0000 mgr.smithi029.loxrfu (mgr.14164) 163 : cluster [DBG] pgmap v152: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:16:30.216 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:30.215+0000 7fe1b5884700 1 -- 172.21.15.73:0/147960568 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe1b010c280 con 0x7fe1b00f51a0 2023-12-01T22:16:30.216 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:30.216+0000 7fe1ad7fa700 1 -- 172.21.15.73:0/147960568 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe1a401cb10 con 0x7fe1b00f51a0 2023-12-01T22:16:30.216 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:16:30.217 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:16:30.218 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:30.217+0000 7fe1b5884700 1 -- 172.21.15.73:0/147960568 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fe1a002ff60 msgr2=0x7fe1a0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:16:30.218 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:30.217+0000 7fe1b5884700 1 --2- 172.21.15.73:0/147960568 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fe1a002ff60 0x7fe1a0032420 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7fe19c0096a0 tx=0x7fe19c006b40).stop 2023-12-01T22:16:30.218 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:30.217+0000 7fe1b5884700 1 -- 172.21.15.73:0/147960568 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe1b00f51a0 msgr2=0x7fe1b010b5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:16:30.218 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:30.217+0000 7fe1b5884700 1 --2- 172.21.15.73:0/147960568 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe1b00f51a0 0x7fe1b010b5d0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7fe1a4013510 tx=0x7fe1a400b040).stop 2023-12-01T22:16:30.218 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:30.217+0000 7fe1b5884700 1 -- 172.21.15.73:0/147960568 shutdown_connections 2023-12-01T22:16:30.218 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:30.217+0000 7fe1b5884700 1 --2- 172.21.15.73:0/147960568 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fe1a002ff60 0x7fe1a0032420 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:16:30.219 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:30.217+0000 7fe1b5884700 1 --2- 172.21.15.73:0/147960568 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe1b00f51a0 0x7fe1b010b5d0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:16:30.219 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:30.217+0000 7fe1b5884700 1 -- 172.21.15.73:0/147960568 >> 172.21.15.73:0/147960568 conn(0x7fe1b00f09a0 msgr2=0x7fe1b00f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:16:30.219 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:30.217+0000 7fe1b5884700 1 -- 172.21.15.73:0/147960568 shutdown_connections 2023-12-01T22:16:30.219 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:30.217+0000 7fe1b5884700 1 -- 172.21.15.73:0/147960568 wait complete. 2023-12-01T22:16:30.219 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:16:31.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:30 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:16:30.216171+0000 mon.smithi029 (mon.0) 239 : audit [DBG] from='client.? 172.21.15.73:0/147960568' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:16:31.622 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:16:31.623 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:16:32.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:31 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:16:30.668383+0000 mgr.smithi029.loxrfu (mgr.14164) 164 : cluster [DBG] pgmap v153: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:16:32.773 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:32.772+0000 7f54ce4e4700 1 -- 172.21.15.73:0/2745164707 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54c8060280 msgr2=0x7f54c80f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:16:32.773 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:32.772+0000 7f54ce4e4700 1 --2- 172.21.15.73:0/2745164707 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54c8060280 0x7f54c80f90f0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 rx=0x7f54b0009fb0 tx=0x7f54b0009b30).stop 2023-12-01T22:16:32.773 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:32.772+0000 7f54ce4e4700 1 -- 172.21.15.73:0/2745164707 shutdown_connections 2023-12-01T22:16:32.773 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:32.772+0000 7f54ce4e4700 1 --2- 172.21.15.73:0/2745164707 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54c8060280 0x7f54c80f90f0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:16:32.773 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:32.772+0000 7f54ce4e4700 1 -- 172.21.15.73:0/2745164707 >> 172.21.15.73:0/2745164707 conn(0x7f54c80f0b70 msgr2=0x7f54c80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:16:32.773 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:32.772+0000 7f54ce4e4700 1 -- 172.21.15.73:0/2745164707 shutdown_connections 2023-12-01T22:16:32.774 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:32.772+0000 7f54ce4e4700 1 -- 172.21.15.73:0/2745164707 wait complete. 2023-12-01T22:16:32.775 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:32.773+0000 7f54ce4e4700 1 Processor -- start 2023-12-01T22:16:32.775 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:32.773+0000 7f54ce4e4700 1 -- start start 2023-12-01T22:16:32.775 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:32.773+0000 7f54ce4e4700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54c8060280 0x7f54c81066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:16:32.775 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:32.773+0000 7f54ce4e4700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54c8106bd0 con 0x7f54c8060280 2023-12-01T22:16:32.775 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:32.773+0000 7f54c7fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54c8060280 0x7f54c81066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:16:32.776 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:32.774+0000 7f54c7fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54c8060280 0x7f54c81066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:39276/0 (socket says 172.21.15.73:39276) 2023-12-01T22:16:32.776 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:32.774+0000 7f54c7fff700 1 -- 172.21.15.73:0/1860421798 learned_addr learned my addr 172.21.15.73:0/1860421798 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:16:32.776 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:32.774+0000 7f54c7fff700 1 -- 172.21.15.73:0/1860421798 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54b0005040 con 0x7f54c8060280 2023-12-01T22:16:32.776 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:32.774+0000 7f54c7fff700 1 --2- 172.21.15.73:0/1860421798 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54c8060280 0x7f54c81066e0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7f54b0013510 tx=0x7f54b000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:16:32.776 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:32.774+0000 7f54c4ff9700 1 -- 172.21.15.73:0/1860421798 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f54b0009320 con 0x7f54c8060280 2023-12-01T22:16:32.776 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:32.774+0000 7f54c4ff9700 1 -- 172.21.15.73:0/1860421798 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f54b0013f00 con 0x7f54c8060280 2023-12-01T22:16:32.776 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:32.774+0000 7f54ce4e4700 1 -- 172.21.15.73:0/1860421798 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54c8106dd0 con 0x7f54c8060280 2023-12-01T22:16:32.776 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:32.774+0000 7f54c4ff9700 1 -- 172.21.15.73:0/1860421798 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f54b0014dc0 con 0x7f54c8060280 2023-12-01T22:16:32.776 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:32.774+0000 7f54ce4e4700 1 -- 172.21.15.73:0/1860421798 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54c8107260 con 0x7f54c8060280 2023-12-01T22:16:32.777 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:32.775+0000 7f54ce4e4700 1 -- 172.21.15.73:0/1860421798 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54c8045ad0 con 0x7f54c8060280 2023-12-01T22:16:32.777 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:32.775+0000 7f54c4ff9700 1 -- 172.21.15.73:0/1860421798 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f54b00095f0 con 0x7f54c8060280 2023-12-01T22:16:32.777 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:32.775+0000 7f54c4ff9700 1 --2- 172.21.15.73:0/1860421798 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f54b402ff60 0x7f54b4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:16:32.777 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:32.775+0000 7f54c4ff9700 1 -- 172.21.15.73:0/1860421798 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f54b0041040 con 0x7f54c8060280 2023-12-01T22:16:32.777 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:32.776+0000 7f54c77fe700 1 --2- 172.21.15.73:0/1860421798 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f54b402ff60 0x7f54b4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:16:32.778 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:32.776+0000 7f54c77fe700 1 --2- 172.21.15.73:0/1860421798 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f54b402ff60 0x7f54b4032420 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f54b80096a0 tx=0x7f54b8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:16:32.783 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:32.783+0000 7f54c4ff9700 1 -- 172.21.15.73:0/1860421798 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f54b0014420 con 0x7f54c8060280 2023-12-01T22:16:32.993 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:32.992+0000 7f54ce4e4700 1 -- 172.21.15.73:0/1860421798 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f54c8108110 con 0x7f54c8060280 2023-12-01T22:16:32.993 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:32.993+0000 7f54c4ff9700 1 -- 172.21.15.73:0/1860421798 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f54b0010230 con 0x7f54c8060280 2023-12-01T22:16:32.994 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:16:32.994 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:16:32.995 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:32.994+0000 7f54ce4e4700 1 -- 172.21.15.73:0/1860421798 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f54b402ff60 msgr2=0x7f54b4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:16:32.995 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:32.994+0000 7f54ce4e4700 1 --2- 172.21.15.73:0/1860421798 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f54b402ff60 0x7f54b4032420 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f54b80096a0 tx=0x7f54b8006b40).stop 2023-12-01T22:16:32.995 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:32.994+0000 7f54ce4e4700 1 -- 172.21.15.73:0/1860421798 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54c8060280 msgr2=0x7f54c81066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:16:32.995 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:32.994+0000 7f54ce4e4700 1 --2- 172.21.15.73:0/1860421798 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54c8060280 0x7f54c81066e0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7f54b0013510 tx=0x7f54b000b040).stop 2023-12-01T22:16:32.995 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:32.994+0000 7f54ce4e4700 1 -- 172.21.15.73:0/1860421798 shutdown_connections 2023-12-01T22:16:32.995 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:32.994+0000 7f54ce4e4700 1 --2- 172.21.15.73:0/1860421798 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f54b402ff60 0x7f54b4032420 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:16:32.995 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:32.994+0000 7f54ce4e4700 1 --2- 172.21.15.73:0/1860421798 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54c8060280 0x7f54c81066e0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:16:32.995 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:32.994+0000 7f54ce4e4700 1 -- 172.21.15.73:0/1860421798 >> 172.21.15.73:0/1860421798 conn(0x7f54c80f0b70 msgr2=0x7f54c80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:16:32.995 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:32.994+0000 7f54ce4e4700 1 -- 172.21.15.73:0/1860421798 shutdown_connections 2023-12-01T22:16:32.995 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:32.994+0000 7f54ce4e4700 1 -- 172.21.15.73:0/1860421798 wait complete. 2023-12-01T22:16:32.996 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:16:34.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:33 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:16:32.668647+0000 mgr.smithi029.loxrfu (mgr.14164) 165 : cluster [DBG] pgmap v154: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:16:34.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:33 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:16:32.993079+0000 mon.smithi029 (mon.0) 240 : audit [DBG] from='client.? 172.21.15.73:0/1860421798' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:16:34.416 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:16:34.416 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:16:35.574 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:35.571+0000 7f1548f7a700 1 -- 172.21.15.73:0/3703480216 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f15440f51a0 msgr2=0x7f15440f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:16:35.574 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:35.571+0000 7f1548f7a700 1 --2- 172.21.15.73:0/3703480216 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f15440f51a0 0x7f15440f55c0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 rx=0x7f1534009fb0 tx=0x7f1534009b30).stop 2023-12-01T22:16:35.574 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:35.571+0000 7f1548f7a700 1 -- 172.21.15.73:0/3703480216 shutdown_connections 2023-12-01T22:16:35.574 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:35.571+0000 7f1548f7a700 1 --2- 172.21.15.73:0/3703480216 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f15440f51a0 0x7f15440f55c0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:16:35.574 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:35.571+0000 7f1548f7a700 1 -- 172.21.15.73:0/3703480216 >> 172.21.15.73:0/3703480216 conn(0x7f15440f09a0 msgr2=0x7f15440f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:16:35.574 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:35.572+0000 7f1548f7a700 1 -- 172.21.15.73:0/3703480216 shutdown_connections 2023-12-01T22:16:35.574 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:35.572+0000 7f1548f7a700 1 -- 172.21.15.73:0/3703480216 wait complete. 2023-12-01T22:16:35.575 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:35.572+0000 7f1548f7a700 1 Processor -- start 2023-12-01T22:16:35.575 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:35.572+0000 7f1548f7a700 1 -- start start 2023-12-01T22:16:35.575 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:35.572+0000 7f1548f7a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f15440f51a0 0x7f15441091d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:16:35.575 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:35.572+0000 7f1548f7a700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1544060220 con 0x7f15440f51a0 2023-12-01T22:16:35.575 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:35.573+0000 7f15437fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f15440f51a0 0x7f15441091d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:16:35.575 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:35.573+0000 7f15437fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f15440f51a0 0x7f15441091d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:34290/0 (socket says 172.21.15.73:34290) 2023-12-01T22:16:35.575 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:35.573+0000 7f15437fe700 1 -- 172.21.15.73:0/1163834584 learned_addr learned my addr 172.21.15.73:0/1163834584 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:16:35.575 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:35.573+0000 7f15437fe700 1 -- 172.21.15.73:0/1163834584 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1534005040 con 0x7f15440f51a0 2023-12-01T22:16:35.575 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:35.573+0000 7f15437fe700 1 --2- 172.21.15.73:0/1163834584 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f15440f51a0 0x7f15441091d0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7f1534013510 tx=0x7f153400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:16:35.575 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:35.573+0000 7f152bfff700 1 -- 172.21.15.73:0/1163834584 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1534009320 con 0x7f15440f51a0 2023-12-01T22:16:35.575 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:35.573+0000 7f1548f7a700 1 -- 172.21.15.73:0/1163834584 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1544060420 con 0x7f15440f51a0 2023-12-01T22:16:35.575 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:35.573+0000 7f152bfff700 1 -- 172.21.15.73:0/1163834584 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1534013f00 con 0x7f15440f51a0 2023-12-01T22:16:35.575 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:35.573+0000 7f152bfff700 1 -- 172.21.15.73:0/1163834584 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1534014dc0 con 0x7f15440f51a0 2023-12-01T22:16:35.575 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:35.573+0000 7f1548f7a700 1 -- 172.21.15.73:0/1163834584 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1544109880 con 0x7f15440f51a0 2023-12-01T22:16:35.576 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:35.574+0000 7f152bfff700 1 -- 172.21.15.73:0/1163834584 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1534014420 con 0x7f15440f51a0 2023-12-01T22:16:35.576 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:35.574+0000 7f1548f7a700 1 -- 172.21.15.73:0/1163834584 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1544059100 con 0x7f15440f51a0 2023-12-01T22:16:35.576 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:35.574+0000 7f152bfff700 1 --2- 172.21.15.73:0/1163834584 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f152c02ff60 0x7f152c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:16:35.576 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:35.574+0000 7f152bfff700 1 -- 172.21.15.73:0/1163834584 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1534041030 con 0x7f15440f51a0 2023-12-01T22:16:35.576 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:35.574+0000 7f1542ffd700 1 --2- 172.21.15.73:0/1163834584 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f152c02ff60 0x7f152c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:16:35.576 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:35.575+0000 7f1542ffd700 1 --2- 172.21.15.73:0/1163834584 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f152c02ff60 0x7f152c032420 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f15380096a0 tx=0x7f1538006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:16:35.579 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:35.579+0000 7f152bfff700 1 -- 172.21.15.73:0/1163834584 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f15340095f0 con 0x7f15440f51a0 2023-12-01T22:16:35.787 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:35.787+0000 7f1548f7a700 1 -- 172.21.15.73:0/1163834584 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1544045ad0 con 0x7f15440f51a0 2023-12-01T22:16:35.788 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:35.787+0000 7f152bfff700 1 -- 172.21.15.73:0/1163834584 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f153401cb10 con 0x7f15440f51a0 2023-12-01T22:16:35.788 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:16:35.788 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:16:35.790 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:35.789+0000 7f1548f7a700 1 -- 172.21.15.73:0/1163834584 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f152c02ff60 msgr2=0x7f152c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:16:35.790 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:35.789+0000 7f1548f7a700 1 --2- 172.21.15.73:0/1163834584 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f152c02ff60 0x7f152c032420 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f15380096a0 tx=0x7f1538006b40).stop 2023-12-01T22:16:35.790 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:35.789+0000 7f1548f7a700 1 -- 172.21.15.73:0/1163834584 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f15440f51a0 msgr2=0x7f15441091d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:16:35.790 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:35.789+0000 7f1548f7a700 1 --2- 172.21.15.73:0/1163834584 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f15440f51a0 0x7f15441091d0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7f1534013510 tx=0x7f153400b040).stop 2023-12-01T22:16:35.790 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:35.789+0000 7f1548f7a700 1 -- 172.21.15.73:0/1163834584 shutdown_connections 2023-12-01T22:16:35.790 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:35.789+0000 7f1548f7a700 1 --2- 172.21.15.73:0/1163834584 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f152c02ff60 0x7f152c032420 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:16:35.790 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:35.789+0000 7f1548f7a700 1 --2- 172.21.15.73:0/1163834584 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f15440f51a0 0x7f15441091d0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:16:35.790 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:35.789+0000 7f1548f7a700 1 -- 172.21.15.73:0/1163834584 >> 172.21.15.73:0/1163834584 conn(0x7f15440f09a0 msgr2=0x7f15440f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:16:35.790 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:35.789+0000 7f1548f7a700 1 -- 172.21.15.73:0/1163834584 shutdown_connections 2023-12-01T22:16:35.790 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:35.789+0000 7f1548f7a700 1 -- 172.21.15.73:0/1163834584 wait complete. 2023-12-01T22:16:35.791 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:16:36.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:35 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:16:34.668946+0000 mgr.smithi029.loxrfu (mgr.14164) 166 : cluster [DBG] pgmap v155: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:16:37.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:36 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:16:35.787734+0000 mon.smithi029 (mon.0) 241 : audit [DBG] from='client.? 172.21.15.73:0/1163834584' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:16:37.190 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:16:37.190 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:16:38.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:37 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:16:36.669215+0000 mgr.smithi029.loxrfu (mgr.14164) 167 : cluster [DBG] pgmap v156: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:16:38.373 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:38.370+0000 7feb7e6a5700 1 -- 172.21.15.73:0/4031669488 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb780f6510 msgr2=0x7feb780f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:16:38.373 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:38.370+0000 7feb7e6a5700 1 --2- 172.21.15.73:0/4031669488 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb780f6510 0x7feb780f6930 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 rx=0x7feb68009fb0 tx=0x7feb68009b30).stop 2023-12-01T22:16:38.373 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:38.370+0000 7feb7e6a5700 1 -- 172.21.15.73:0/4031669488 shutdown_connections 2023-12-01T22:16:38.373 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:38.370+0000 7feb7e6a5700 1 --2- 172.21.15.73:0/4031669488 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb780f6510 0x7feb780f6930 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:16:38.373 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:38.370+0000 7feb7e6a5700 1 -- 172.21.15.73:0/4031669488 >> 172.21.15.73:0/4031669488 conn(0x7feb780f1c70 msgr2=0x7feb780f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:16:38.373 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:38.370+0000 7feb7e6a5700 1 -- 172.21.15.73:0/4031669488 shutdown_connections 2023-12-01T22:16:38.373 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:38.370+0000 7feb7e6a5700 1 -- 172.21.15.73:0/4031669488 wait complete. 2023-12-01T22:16:38.373 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:38.371+0000 7feb7e6a5700 1 Processor -- start 2023-12-01T22:16:38.374 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:38.371+0000 7feb7e6a5700 1 -- start start 2023-12-01T22:16:38.374 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:38.371+0000 7feb7e6a5700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb780f6510 0x7feb78106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:16:38.374 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:38.371+0000 7feb7e6a5700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb78106c20 con 0x7feb780f6510 2023-12-01T22:16:38.374 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:38.371+0000 7feb77fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb780f6510 0x7feb78106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:16:38.374 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:38.371+0000 7feb77fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb780f6510 0x7feb78106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:34306/0 (socket says 172.21.15.73:34306) 2023-12-01T22:16:38.374 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:38.371+0000 7feb77fff700 1 -- 172.21.15.73:0/621022574 learned_addr learned my addr 172.21.15.73:0/621022574 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:16:38.376 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:38.372+0000 7feb77fff700 1 -- 172.21.15.73:0/621022574 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb68005040 con 0x7feb780f6510 2023-12-01T22:16:38.376 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:38.372+0000 7feb77fff700 1 --2- 172.21.15.73:0/621022574 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb780f6510 0x7feb78106730 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7feb68013510 tx=0x7feb6800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:16:38.377 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:38.372+0000 7feb74ff9700 1 -- 172.21.15.73:0/621022574 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feb68009320 con 0x7feb780f6510 2023-12-01T22:16:38.377 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:38.372+0000 7feb74ff9700 1 -- 172.21.15.73:0/621022574 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7feb68013f00 con 0x7feb780f6510 2023-12-01T22:16:38.377 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:38.372+0000 7feb7e6a5700 1 -- 172.21.15.73:0/621022574 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb78106e20 con 0x7feb780f6510 2023-12-01T22:16:38.377 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:38.372+0000 7feb74ff9700 1 -- 172.21.15.73:0/621022574 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feb68014d90 con 0x7feb780f6510 2023-12-01T22:16:38.377 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:38.372+0000 7feb7e6a5700 1 -- 172.21.15.73:0/621022574 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb781072b0 con 0x7feb780f6510 2023-12-01T22:16:38.377 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:38.373+0000 7feb74ff9700 1 -- 172.21.15.73:0/621022574 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7feb680095f0 con 0x7feb780f6510 2023-12-01T22:16:38.377 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:38.373+0000 7feb7e6a5700 1 -- 172.21.15.73:0/621022574 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb78045ad0 con 0x7feb780f6510 2023-12-01T22:16:38.378 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:38.373+0000 7feb74ff9700 1 --2- 172.21.15.73:0/621022574 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7feb6002ff60 0x7feb60032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:16:38.378 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:38.373+0000 7feb74ff9700 1 -- 172.21.15.73:0/621022574 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7feb68041000 con 0x7feb780f6510 2023-12-01T22:16:38.378 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:38.373+0000 7feb777fe700 1 --2- 172.21.15.73:0/621022574 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7feb6002ff60 0x7feb60032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:16:38.378 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:38.373+0000 7feb777fe700 1 --2- 172.21.15.73:0/621022574 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7feb6002ff60 0x7feb60032420 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7feb6c0096a0 tx=0x7feb6c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:16:38.379 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:38.378+0000 7feb74ff9700 1 -- 172.21.15.73:0/621022574 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7feb68014420 con 0x7feb780f6510 2023-12-01T22:16:38.590 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:38.589+0000 7feb7e6a5700 1 -- 172.21.15.73:0/621022574 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7feb780fbf60 con 0x7feb780f6510 2023-12-01T22:16:38.591 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:38.589+0000 7feb74ff9700 1 -- 172.21.15.73:0/621022574 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7feb6801d7f0 con 0x7feb780f6510 2023-12-01T22:16:38.591 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:16:38.591 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:16:38.593 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:38.591+0000 7feb7e6a5700 1 -- 172.21.15.73:0/621022574 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7feb6002ff60 msgr2=0x7feb60032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:16:38.593 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:38.591+0000 7feb7e6a5700 1 --2- 172.21.15.73:0/621022574 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7feb6002ff60 0x7feb60032420 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7feb6c0096a0 tx=0x7feb6c006b40).stop 2023-12-01T22:16:38.593 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:38.591+0000 7feb7e6a5700 1 -- 172.21.15.73:0/621022574 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb780f6510 msgr2=0x7feb78106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:16:38.593 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:38.591+0000 7feb7e6a5700 1 --2- 172.21.15.73:0/621022574 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb780f6510 0x7feb78106730 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7feb68013510 tx=0x7feb6800b040).stop 2023-12-01T22:16:38.593 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:38.591+0000 7feb7e6a5700 1 -- 172.21.15.73:0/621022574 shutdown_connections 2023-12-01T22:16:38.593 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:38.591+0000 7feb7e6a5700 1 --2- 172.21.15.73:0/621022574 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7feb6002ff60 0x7feb60032420 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:16:38.593 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:38.591+0000 7feb7e6a5700 1 --2- 172.21.15.73:0/621022574 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb780f6510 0x7feb78106730 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:16:38.594 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:38.591+0000 7feb7e6a5700 1 -- 172.21.15.73:0/621022574 >> 172.21.15.73:0/621022574 conn(0x7feb780f1c70 msgr2=0x7feb780f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:16:38.594 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:38.591+0000 7feb7e6a5700 1 -- 172.21.15.73:0/621022574 shutdown_connections 2023-12-01T22:16:38.594 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:38.591+0000 7feb7e6a5700 1 -- 172.21.15.73:0/621022574 wait complete. 2023-12-01T22:16:38.594 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:16:39.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:38 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:16:38.590751+0000 mon.smithi029 (mon.0) 242 : audit [DBG] from='client.? 172.21.15.73:0/621022574' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:16:39.974 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:16:39.974 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:16:40.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:39 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:16:38.669486+0000 mgr.smithi029.loxrfu (mgr.14164) 168 : cluster [DBG] pgmap v157: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:16:41.156 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:41.152+0000 7fe13a471700 1 -- 172.21.15.73:0/3560449772 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe1340f62a0 msgr2=0x7fe1340f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:16:41.156 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:41.153+0000 7fe13a471700 1 --2- 172.21.15.73:0/3560449772 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe1340f62a0 0x7fe1340f66c0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 rx=0x7fe130009fb0 tx=0x7fe130009b30).stop 2023-12-01T22:16:41.156 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:41.153+0000 7fe13a471700 1 -- 172.21.15.73:0/3560449772 shutdown_connections 2023-12-01T22:16:41.156 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:41.153+0000 7fe13a471700 1 --2- 172.21.15.73:0/3560449772 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe1340f62a0 0x7fe1340f66c0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:16:41.156 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:41.153+0000 7fe13a471700 1 -- 172.21.15.73:0/3560449772 >> 172.21.15.73:0/3560449772 conn(0x7fe1340f1aa0 msgr2=0x7fe1340f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:16:41.156 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:41.153+0000 7fe13a471700 1 -- 172.21.15.73:0/3560449772 shutdown_connections 2023-12-01T22:16:41.156 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:41.153+0000 7fe13a471700 1 -- 172.21.15.73:0/3560449772 wait complete. 2023-12-01T22:16:41.156 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:41.153+0000 7fe13a471700 1 Processor -- start 2023-12-01T22:16:41.156 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:41.153+0000 7fe13a471700 1 -- start start 2023-12-01T22:16:41.156 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:41.153+0000 7fe13a471700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe1340f62a0 0x7fe1341064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:16:41.157 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:41.153+0000 7fe13a471700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1341069a0 con 0x7fe1340f62a0 2023-12-01T22:16:41.157 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:41.154+0000 7fe13946f700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe1340f62a0 0x7fe1341064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:16:41.157 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:41.154+0000 7fe13946f700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe1340f62a0 0x7fe1341064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:34326/0 (socket says 172.21.15.73:34326) 2023-12-01T22:16:41.157 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:41.154+0000 7fe13946f700 1 -- 172.21.15.73:0/137500653 learned_addr learned my addr 172.21.15.73:0/137500653 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:16:41.157 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:41.154+0000 7fe13946f700 1 -- 172.21.15.73:0/137500653 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe130005040 con 0x7fe1340f62a0 2023-12-01T22:16:41.157 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:41.154+0000 7fe13946f700 1 --2- 172.21.15.73:0/137500653 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe1340f62a0 0x7fe1341064b0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7fe130013510 tx=0x7fe13000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:16:41.157 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:41.154+0000 7fe129ffb700 1 -- 172.21.15.73:0/137500653 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe130009320 con 0x7fe1340f62a0 2023-12-01T22:16:41.157 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:41.154+0000 7fe13a471700 1 -- 172.21.15.73:0/137500653 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe134106ba0 con 0x7fe1340f62a0 2023-12-01T22:16:41.157 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:41.154+0000 7fe129ffb700 1 -- 172.21.15.73:0/137500653 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe130013f00 con 0x7fe1340f62a0 2023-12-01T22:16:41.157 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:41.154+0000 7fe129ffb700 1 -- 172.21.15.73:0/137500653 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe130014dc0 con 0x7fe1340f62a0 2023-12-01T22:16:41.157 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:41.154+0000 7fe13a471700 1 -- 172.21.15.73:0/137500653 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe134107030 con 0x7fe1340f62a0 2023-12-01T22:16:41.158 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:41.155+0000 7fe129ffb700 1 -- 172.21.15.73:0/137500653 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe1300095f0 con 0x7fe1340f62a0 2023-12-01T22:16:41.158 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:41.155+0000 7fe13a471700 1 -- 172.21.15.73:0/137500653 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe134045ad0 con 0x7fe1340f62a0 2023-12-01T22:16:41.158 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:41.155+0000 7fe129ffb700 1 --2- 172.21.15.73:0/137500653 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fe12002ff60 0x7fe120032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:16:41.158 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:41.155+0000 7fe129ffb700 1 -- 172.21.15.73:0/137500653 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe130041100 con 0x7fe1340f62a0 2023-12-01T22:16:41.158 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:41.155+0000 7fe138c6e700 1 --2- 172.21.15.73:0/137500653 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fe12002ff60 0x7fe120032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:16:41.159 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:41.156+0000 7fe138c6e700 1 --2- 172.21.15.73:0/137500653 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fe12002ff60 0x7fe120032420 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7fe1240096a0 tx=0x7fe124006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:16:41.161 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:41.160+0000 7fe129ffb700 1 -- 172.21.15.73:0/137500653 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe130014420 con 0x7fe1340f62a0 2023-12-01T22:16:41.369 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:41.368+0000 7fe13a471700 1 -- 172.21.15.73:0/137500653 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe134107ee0 con 0x7fe1340f62a0 2023-12-01T22:16:41.370 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:41.368+0000 7fe129ffb700 1 -- 172.21.15.73:0/137500653 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe130010230 con 0x7fe1340f62a0 2023-12-01T22:16:41.370 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:16:41.370 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:16:41.372 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:41.370+0000 7fe13a471700 1 -- 172.21.15.73:0/137500653 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fe12002ff60 msgr2=0x7fe120032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:16:41.372 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:41.370+0000 7fe13a471700 1 --2- 172.21.15.73:0/137500653 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fe12002ff60 0x7fe120032420 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7fe1240096a0 tx=0x7fe124006b40).stop 2023-12-01T22:16:41.372 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:41.370+0000 7fe13a471700 1 -- 172.21.15.73:0/137500653 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe1340f62a0 msgr2=0x7fe1341064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:16:41.372 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:41.370+0000 7fe13a471700 1 --2- 172.21.15.73:0/137500653 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe1340f62a0 0x7fe1341064b0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7fe130013510 tx=0x7fe13000b040).stop 2023-12-01T22:16:41.372 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:41.370+0000 7fe13a471700 1 -- 172.21.15.73:0/137500653 shutdown_connections 2023-12-01T22:16:41.372 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:41.370+0000 7fe13a471700 1 --2- 172.21.15.73:0/137500653 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fe12002ff60 0x7fe120032420 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:16:41.372 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:41.370+0000 7fe13a471700 1 --2- 172.21.15.73:0/137500653 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe1340f62a0 0x7fe1341064b0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:16:41.372 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:41.370+0000 7fe13a471700 1 -- 172.21.15.73:0/137500653 >> 172.21.15.73:0/137500653 conn(0x7fe1340f1aa0 msgr2=0x7fe1340f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:16:41.372 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:41.370+0000 7fe13a471700 1 -- 172.21.15.73:0/137500653 shutdown_connections 2023-12-01T22:16:41.372 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:41.370+0000 7fe13a471700 1 -- 172.21.15.73:0/137500653 wait complete. 2023-12-01T22:16:41.373 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:16:41.777 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:41 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:16:40.669899+0000 mgr.smithi029.loxrfu (mgr.14164) 169 : cluster [DBG] pgmap v158: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:16:41.777 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:41 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:16:41.369686+0000 mon.smithi029 (mon.0) 243 : audit [DBG] from='client.? 172.21.15.73:0/137500653' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:16:42.756 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:16:42.757 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:16:43.777 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:43 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:16:42.292553+0000 mon.smithi029 (mon.0) 244 : audit [INF] from='mgr.14164 172.21.15.29:0/3195927111' entity='mgr.smithi029.loxrfu' 2023-12-01T22:16:43.777 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:43 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cephadm 2023-12-01T22:16:42.294138+0000 mgr.smithi029.loxrfu (mgr.14164) 170 : cephadm [INF] Deploying daemon node-exporter.smithi029 on smithi029 2023-12-01T22:16:43.777 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:43 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:16:42.670233+0000 mgr.smithi029.loxrfu (mgr.14164) 171 : cluster [DBG] pgmap v159: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:16:43.931 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:43.929+0000 7f96683e2700 1 -- 172.21.15.73:0/1577250771 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96600f6510 msgr2=0x7f96600f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:16:43.932 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:43.929+0000 7f96683e2700 1 --2- 172.21.15.73:0/1577250771 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96600f6510 0x7f96600f6930 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 rx=0x7f965c009fb0 tx=0x7f965c009b30).stop 2023-12-01T22:16:43.932 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:43.929+0000 7f96683e2700 1 -- 172.21.15.73:0/1577250771 shutdown_connections 2023-12-01T22:16:43.932 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:43.929+0000 7f96683e2700 1 --2- 172.21.15.73:0/1577250771 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96600f6510 0x7f96600f6930 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:16:43.932 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:43.929+0000 7f96683e2700 1 -- 172.21.15.73:0/1577250771 >> 172.21.15.73:0/1577250771 conn(0x7f96600f1c70 msgr2=0x7f96600f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:16:43.932 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:43.929+0000 7f96683e2700 1 -- 172.21.15.73:0/1577250771 shutdown_connections 2023-12-01T22:16:43.932 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:43.929+0000 7f96683e2700 1 -- 172.21.15.73:0/1577250771 wait complete. 2023-12-01T22:16:43.932 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:43.930+0000 7f96683e2700 1 Processor -- start 2023-12-01T22:16:43.932 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:43.930+0000 7f96683e2700 1 -- start start 2023-12-01T22:16:43.932 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:43.930+0000 7f96683e2700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96600f6510 0x7f9660106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:16:43.932 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:43.930+0000 7f96683e2700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9660106c20 con 0x7f96600f6510 2023-12-01T22:16:43.934 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:43.930+0000 7f966617e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96600f6510 0x7f9660106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:16:43.935 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:43.930+0000 7f966617e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96600f6510 0x7f9660106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:34350/0 (socket says 172.21.15.73:34350) 2023-12-01T22:16:43.935 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:43.930+0000 7f966617e700 1 -- 172.21.15.73:0/1057869627 learned_addr learned my addr 172.21.15.73:0/1057869627 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:16:43.935 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:43.931+0000 7f966617e700 1 -- 172.21.15.73:0/1057869627 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f965c005040 con 0x7f96600f6510 2023-12-01T22:16:43.935 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:43.931+0000 7f966617e700 1 --2- 172.21.15.73:0/1057869627 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96600f6510 0x7f9660106730 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7f965c013510 tx=0x7f965c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:16:43.935 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:43.931+0000 7f9656ffd700 1 -- 172.21.15.73:0/1057869627 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f965c009320 con 0x7f96600f6510 2023-12-01T22:16:43.935 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:43.931+0000 7f96683e2700 1 -- 172.21.15.73:0/1057869627 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9660106e20 con 0x7f96600f6510 2023-12-01T22:16:43.935 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:43.931+0000 7f9656ffd700 1 -- 172.21.15.73:0/1057869627 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f965c013f00 con 0x7f96600f6510 2023-12-01T22:16:43.935 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:43.931+0000 7f9656ffd700 1 -- 172.21.15.73:0/1057869627 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f965c014d90 con 0x7f96600f6510 2023-12-01T22:16:43.935 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:43.931+0000 7f96683e2700 1 -- 172.21.15.73:0/1057869627 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96601072b0 con 0x7f96600f6510 2023-12-01T22:16:43.935 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:43.932+0000 7f9656ffd700 1 -- 172.21.15.73:0/1057869627 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f965c0095f0 con 0x7f96600f6510 2023-12-01T22:16:43.935 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:43.932+0000 7f96683e2700 1 -- 172.21.15.73:0/1057869627 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9660045ad0 con 0x7f96600f6510 2023-12-01T22:16:43.935 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:43.932+0000 7f9656ffd700 1 --2- 172.21.15.73:0/1057869627 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f964c02ff60 0x7f964c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:16:43.936 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:43.932+0000 7f9656ffd700 1 -- 172.21.15.73:0/1057869627 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f965c041080 con 0x7f96600f6510 2023-12-01T22:16:43.936 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:43.932+0000 7f966597d700 1 --2- 172.21.15.73:0/1057869627 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f964c02ff60 0x7f964c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:16:43.936 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:43.933+0000 7f966597d700 1 --2- 172.21.15.73:0/1057869627 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f964c02ff60 0x7f964c032420 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f96500096a0 tx=0x7f9650006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:16:43.938 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:43.937+0000 7f9656ffd700 1 -- 172.21.15.73:0/1057869627 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f965c014420 con 0x7f96600f6510 2023-12-01T22:16:44.141 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:44.140+0000 7f96683e2700 1 -- 172.21.15.73:0/1057869627 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f96600fbf60 con 0x7f96600f6510 2023-12-01T22:16:44.142 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:44.140+0000 7f9656ffd700 1 -- 172.21.15.73:0/1057869627 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f965c010230 con 0x7f96600f6510 2023-12-01T22:16:44.142 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:16:44.143 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:16:44.144 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:44.142+0000 7f96683e2700 1 -- 172.21.15.73:0/1057869627 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f964c02ff60 msgr2=0x7f964c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:16:44.144 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:44.142+0000 7f96683e2700 1 --2- 172.21.15.73:0/1057869627 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f964c02ff60 0x7f964c032420 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f96500096a0 tx=0x7f9650006b40).stop 2023-12-01T22:16:44.144 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:44.142+0000 7f96683e2700 1 -- 172.21.15.73:0/1057869627 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96600f6510 msgr2=0x7f9660106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:16:44.144 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:44.142+0000 7f96683e2700 1 --2- 172.21.15.73:0/1057869627 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96600f6510 0x7f9660106730 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7f965c013510 tx=0x7f965c00b040).stop 2023-12-01T22:16:44.144 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:44.142+0000 7f96683e2700 1 -- 172.21.15.73:0/1057869627 shutdown_connections 2023-12-01T22:16:44.144 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:44.142+0000 7f96683e2700 1 --2- 172.21.15.73:0/1057869627 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f964c02ff60 0x7f964c032420 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:16:44.144 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:44.142+0000 7f96683e2700 1 --2- 172.21.15.73:0/1057869627 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96600f6510 0x7f9660106730 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:16:44.144 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:44.142+0000 7f96683e2700 1 -- 172.21.15.73:0/1057869627 >> 172.21.15.73:0/1057869627 conn(0x7f96600f1c70 msgr2=0x7f96600f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:16:44.144 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:44.142+0000 7f96683e2700 1 -- 172.21.15.73:0/1057869627 shutdown_connections 2023-12-01T22:16:44.144 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:44.142+0000 7f96683e2700 1 -- 172.21.15.73:0/1057869627 wait complete. 2023-12-01T22:16:44.145 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:16:44.776 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:44 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:16:44.141742+0000 mon.smithi029 (mon.0) 245 : audit [DBG] from='client.? 172.21.15.73:0/1057869627' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:16:45.535 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:16:45.536 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:16:45.776 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:45 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:16:44.670591+0000 mgr.smithi029.loxrfu (mgr.14164) 172 : cluster [DBG] pgmap v160: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:16:46.758 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:46.756+0000 7f5c9e991700 1 -- 172.21.15.73:0/2442808366 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5c980f5410 msgr2=0x7f5c980f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:16:46.758 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:46.756+0000 7f5c9e991700 1 --2- 172.21.15.73:0/2442808366 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5c980f5410 0x7f5c980f5830 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 rx=0x7f5c88009fb0 tx=0x7f5c88009b30).stop 2023-12-01T22:16:46.758 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:46.756+0000 7f5c9e991700 1 -- 172.21.15.73:0/2442808366 shutdown_connections 2023-12-01T22:16:46.758 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:46.756+0000 7f5c9e991700 1 --2- 172.21.15.73:0/2442808366 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5c980f5410 0x7f5c980f5830 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:16:46.759 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:46.756+0000 7f5c9e991700 1 -- 172.21.15.73:0/2442808366 >> 172.21.15.73:0/2442808366 conn(0x7f5c980f0b70 msgr2=0x7f5c980f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:16:46.759 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:46.756+0000 7f5c9e991700 1 -- 172.21.15.73:0/2442808366 shutdown_connections 2023-12-01T22:16:46.759 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:46.756+0000 7f5c9e991700 1 -- 172.21.15.73:0/2442808366 wait complete. 2023-12-01T22:16:46.760 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:46.757+0000 7f5c9e991700 1 Processor -- start 2023-12-01T22:16:46.760 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:46.757+0000 7f5c9e991700 1 -- start start 2023-12-01T22:16:46.760 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:46.757+0000 7f5c9e991700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5c980f5410 0x7f5c981093f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:16:46.760 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:46.757+0000 7f5c9e991700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c98060280 con 0x7f5c980f5410 2023-12-01T22:16:46.761 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:46.758+0000 7f5c97fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5c980f5410 0x7f5c981093f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:16:46.761 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:46.758+0000 7f5c97fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5c980f5410 0x7f5c981093f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:50360/0 (socket says 172.21.15.73:50360) 2023-12-01T22:16:46.761 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:46.758+0000 7f5c97fff700 1 -- 172.21.15.73:0/2020657579 learned_addr learned my addr 172.21.15.73:0/2020657579 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:16:46.761 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:46.758+0000 7f5c97fff700 1 -- 172.21.15.73:0/2020657579 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c88005040 con 0x7f5c980f5410 2023-12-01T22:16:46.761 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:46.758+0000 7f5c97fff700 1 --2- 172.21.15.73:0/2020657579 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5c980f5410 0x7f5c981093f0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7f5c88013510 tx=0x7f5c8800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:16:46.761 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:46.758+0000 7f5c94ff9700 1 -- 172.21.15.73:0/2020657579 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5c88009320 con 0x7f5c980f5410 2023-12-01T22:16:46.761 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:46.758+0000 7f5c9e991700 1 -- 172.21.15.73:0/2020657579 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c98060480 con 0x7f5c980f5410 2023-12-01T22:16:46.761 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:46.758+0000 7f5c94ff9700 1 -- 172.21.15.73:0/2020657579 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5c88013f00 con 0x7f5c980f5410 2023-12-01T22:16:46.761 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:46.758+0000 7f5c94ff9700 1 -- 172.21.15.73:0/2020657579 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5c88014d90 con 0x7f5c980f5410 2023-12-01T22:16:46.761 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:46.759+0000 7f5c9e991700 1 -- 172.21.15.73:0/2020657579 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c98109b20 con 0x7f5c980f5410 2023-12-01T22:16:46.762 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:46.759+0000 7f5c94ff9700 1 -- 172.21.15.73:0/2020657579 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5c88014420 con 0x7f5c980f5410 2023-12-01T22:16:46.762 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:46.759+0000 7f5c94ff9700 1 --2- 172.21.15.73:0/2020657579 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f5c8002ff60 0x7f5c80032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:16:46.762 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:46.759+0000 7f5c9e991700 1 -- 172.21.15.73:0/2020657579 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c98045ad0 con 0x7f5c980f5410 2023-12-01T22:16:46.762 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:46.759+0000 7f5c94ff9700 1 -- 172.21.15.73:0/2020657579 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5c880410c0 con 0x7f5c980f5410 2023-12-01T22:16:46.762 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:46.760+0000 7f5c977fe700 1 --2- 172.21.15.73:0/2020657579 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f5c8002ff60 0x7f5c80032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:16:46.762 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:46.760+0000 7f5c977fe700 1 --2- 172.21.15.73:0/2020657579 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f5c8002ff60 0x7f5c80032420 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f5c8c0096a0 tx=0x7f5c8c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:16:46.766 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:46.764+0000 7f5c94ff9700 1 -- 172.21.15.73:0/2020657579 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5c8806d2b0 con 0x7f5c980f5410 2023-12-01T22:16:46.969 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:46.968+0000 7f5c9e991700 1 -- 172.21.15.73:0/2020657579 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5c9810a340 con 0x7f5c980f5410 2023-12-01T22:16:46.970 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:46.968+0000 7f5c94ff9700 1 -- 172.21.15.73:0/2020657579 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5c880095f0 con 0x7f5c980f5410 2023-12-01T22:16:46.970 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:16:46.970 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:16:46.972 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:46.970+0000 7f5c9e991700 1 -- 172.21.15.73:0/2020657579 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f5c8002ff60 msgr2=0x7f5c80032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:16:46.972 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:46.970+0000 7f5c9e991700 1 --2- 172.21.15.73:0/2020657579 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f5c8002ff60 0x7f5c80032420 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f5c8c0096a0 tx=0x7f5c8c006b40).stop 2023-12-01T22:16:46.972 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:46.970+0000 7f5c9e991700 1 -- 172.21.15.73:0/2020657579 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5c980f5410 msgr2=0x7f5c981093f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:16:46.972 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:46.970+0000 7f5c9e991700 1 --2- 172.21.15.73:0/2020657579 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5c980f5410 0x7f5c981093f0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7f5c88013510 tx=0x7f5c8800b040).stop 2023-12-01T22:16:46.972 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:46.970+0000 7f5c9e991700 1 -- 172.21.15.73:0/2020657579 shutdown_connections 2023-12-01T22:16:46.972 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:46.970+0000 7f5c9e991700 1 --2- 172.21.15.73:0/2020657579 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f5c8002ff60 0x7f5c80032420 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:16:46.972 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:46.970+0000 7f5c9e991700 1 --2- 172.21.15.73:0/2020657579 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5c980f5410 0x7f5c981093f0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:16:46.972 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:46.970+0000 7f5c9e991700 1 -- 172.21.15.73:0/2020657579 >> 172.21.15.73:0/2020657579 conn(0x7f5c980f0b70 msgr2=0x7f5c980f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:16:46.972 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:46.970+0000 7f5c9e991700 1 -- 172.21.15.73:0/2020657579 shutdown_connections 2023-12-01T22:16:46.972 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:46.970+0000 7f5c9e991700 1 -- 172.21.15.73:0/2020657579 wait complete. 2023-12-01T22:16:46.973 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:16:48.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:47 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:16:46.670894+0000 mgr.smithi029.loxrfu (mgr.14164) 173 : cluster [DBG] pgmap v161: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:16:48.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:47 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:16:46.969616+0000 mon.smithi029 (mon.0) 246 : audit [DBG] from='client.? 172.21.15.73:0/2020657579' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:16:48.364 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:16:48.365 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:16:49.510 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:49.507+0000 7f75d0808700 1 -- 172.21.15.73:0/239661841 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f75cc0f5410 msgr2=0x7f75cc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:16:49.510 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:49.507+0000 7f75d0808700 1 --2- 172.21.15.73:0/239661841 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f75cc0f5410 0x7f75cc0f5830 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 rx=0x7f75c0009fb0 tx=0x7f75c0009b30).stop 2023-12-01T22:16:49.510 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:49.507+0000 7f75d0808700 1 -- 172.21.15.73:0/239661841 shutdown_connections 2023-12-01T22:16:49.511 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:49.507+0000 7f75d0808700 1 --2- 172.21.15.73:0/239661841 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f75cc0f5410 0x7f75cc0f5830 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:16:49.511 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:49.507+0000 7f75d0808700 1 -- 172.21.15.73:0/239661841 >> 172.21.15.73:0/239661841 conn(0x7f75cc0f0b70 msgr2=0x7f75cc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:16:49.511 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:49.507+0000 7f75d0808700 1 -- 172.21.15.73:0/239661841 shutdown_connections 2023-12-01T22:16:49.511 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:49.507+0000 7f75d0808700 1 -- 172.21.15.73:0/239661841 wait complete. 2023-12-01T22:16:49.511 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:49.508+0000 7f75d0808700 1 Processor -- start 2023-12-01T22:16:49.511 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:49.508+0000 7f75d0808700 1 -- start start 2023-12-01T22:16:49.511 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:49.508+0000 7f75d0808700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f75cc0f5410 0x7f75cc109400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:16:49.511 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:49.508+0000 7f75d0808700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75cc060240 con 0x7f75cc0f5410 2023-12-01T22:16:49.511 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:49.508+0000 7f75ca59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f75cc0f5410 0x7f75cc109400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:16:49.511 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:49.508+0000 7f75ca59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f75cc0f5410 0x7f75cc109400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:50382/0 (socket says 172.21.15.73:50382) 2023-12-01T22:16:49.511 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:49.508+0000 7f75ca59c700 1 -- 172.21.15.73:0/2165061480 learned_addr learned my addr 172.21.15.73:0/2165061480 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:16:49.511 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:49.508+0000 7f75ca59c700 1 -- 172.21.15.73:0/2165061480 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75c0005040 con 0x7f75cc0f5410 2023-12-01T22:16:49.511 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:49.509+0000 7f75ca59c700 1 --2- 172.21.15.73:0/2165061480 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f75cc0f5410 0x7f75cc109400 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7f75c0013510 tx=0x7f75c000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:16:49.511 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:49.509+0000 7f75baffd700 1 -- 172.21.15.73:0/2165061480 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f75c0009320 con 0x7f75cc0f5410 2023-12-01T22:16:49.511 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:49.509+0000 7f75d0808700 1 -- 172.21.15.73:0/2165061480 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75cc060440 con 0x7f75cc0f5410 2023-12-01T22:16:49.511 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:49.509+0000 7f75baffd700 1 -- 172.21.15.73:0/2165061480 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f75c0013f00 con 0x7f75cc0f5410 2023-12-01T22:16:49.512 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:49.509+0000 7f75baffd700 1 -- 172.21.15.73:0/2165061480 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f75c0014d90 con 0x7f75cc0f5410 2023-12-01T22:16:49.512 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:49.509+0000 7f75d0808700 1 -- 172.21.15.73:0/2165061480 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75cc109ab0 con 0x7f75cc0f5410 2023-12-01T22:16:49.512 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:49.510+0000 7f75baffd700 1 -- 172.21.15.73:0/2165061480 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f75c00095f0 con 0x7f75cc0f5410 2023-12-01T22:16:49.512 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:49.510+0000 7f75d0808700 1 -- 172.21.15.73:0/2165061480 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75cc045ad0 con 0x7f75cc0f5410 2023-12-01T22:16:49.512 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:49.510+0000 7f75baffd700 1 --2- 172.21.15.73:0/2165061480 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f75b402ff60 0x7f75b4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:16:49.513 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:49.510+0000 7f75baffd700 1 -- 172.21.15.73:0/2165061480 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f75c0041080 con 0x7f75cc0f5410 2023-12-01T22:16:49.513 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:49.510+0000 7f75c9d9b700 1 --2- 172.21.15.73:0/2165061480 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f75b402ff60 0x7f75b4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:16:49.513 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:49.510+0000 7f75c9d9b700 1 --2- 172.21.15.73:0/2165061480 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f75b402ff60 0x7f75b4032420 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f75bc0096a0 tx=0x7f75bc006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:16:49.516 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:49.514+0000 7f75baffd700 1 -- 172.21.15.73:0/2165061480 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f75c0014420 con 0x7f75cc0f5410 2023-12-01T22:16:49.723 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:49.721+0000 7f75d0808700 1 -- 172.21.15.73:0/2165061480 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f75cc0fae60 con 0x7f75cc0f5410 2023-12-01T22:16:49.724 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:49.723+0000 7f75baffd700 1 -- 172.21.15.73:0/2165061480 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f75c0010230 con 0x7f75cc0f5410 2023-12-01T22:16:49.724 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:16:49.724 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:16:49.726 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:49.724+0000 7f75d0808700 1 -- 172.21.15.73:0/2165061480 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f75b402ff60 msgr2=0x7f75b4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:16:49.726 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:49.724+0000 7f75d0808700 1 --2- 172.21.15.73:0/2165061480 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f75b402ff60 0x7f75b4032420 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f75bc0096a0 tx=0x7f75bc006b40).stop 2023-12-01T22:16:49.726 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:49.724+0000 7f75d0808700 1 -- 172.21.15.73:0/2165061480 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f75cc0f5410 msgr2=0x7f75cc109400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:16:49.726 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:49.724+0000 7f75d0808700 1 --2- 172.21.15.73:0/2165061480 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f75cc0f5410 0x7f75cc109400 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7f75c0013510 tx=0x7f75c000b040).stop 2023-12-01T22:16:49.726 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:49.724+0000 7f75d0808700 1 -- 172.21.15.73:0/2165061480 shutdown_connections 2023-12-01T22:16:49.726 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:49.724+0000 7f75d0808700 1 --2- 172.21.15.73:0/2165061480 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f75b402ff60 0x7f75b4032420 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:16:49.726 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:49.724+0000 7f75d0808700 1 --2- 172.21.15.73:0/2165061480 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f75cc0f5410 0x7f75cc109400 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:16:49.726 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:49.724+0000 7f75d0808700 1 -- 172.21.15.73:0/2165061480 >> 172.21.15.73:0/2165061480 conn(0x7f75cc0f0b70 msgr2=0x7f75cc0f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:16:49.726 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:49.724+0000 7f75d0808700 1 -- 172.21.15.73:0/2165061480 shutdown_connections 2023-12-01T22:16:49.726 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:49.724+0000 7f75d0808700 1 -- 172.21.15.73:0/2165061480 wait complete. 2023-12-01T22:16:49.727 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:16:50.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:49 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:16:48.671326+0000 mgr.smithi029.loxrfu (mgr.14164) 174 : cluster [DBG] pgmap v162: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:16:51.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:50 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:16:49.723849+0000 mon.smithi029 (mon.0) 247 : audit [DBG] from='client.? 172.21.15.73:0/2165061480' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:16:51.103 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:16:51.103 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:16:52.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:51 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:16:50.671631+0000 mgr.smithi029.loxrfu (mgr.14164) 175 : cluster [DBG] pgmap v163: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:16:52.256 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:52.253+0000 7f284ba1f700 1 -- 172.21.15.73:0/1538294360 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28440f5410 msgr2=0x7f28440f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:16:52.257 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:52.253+0000 7f284ba1f700 1 --2- 172.21.15.73:0/1538294360 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28440f5410 0x7f28440f5830 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 rx=0x7f2838009fb0 tx=0x7f2838009b30).stop 2023-12-01T22:16:52.257 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:52.253+0000 7f284ba1f700 1 -- 172.21.15.73:0/1538294360 shutdown_connections 2023-12-01T22:16:52.257 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:52.253+0000 7f284ba1f700 1 --2- 172.21.15.73:0/1538294360 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28440f5410 0x7f28440f5830 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:16:52.257 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:52.253+0000 7f284ba1f700 1 -- 172.21.15.73:0/1538294360 >> 172.21.15.73:0/1538294360 conn(0x7f28440f0b70 msgr2=0x7f28440f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:16:52.257 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:52.253+0000 7f284ba1f700 1 -- 172.21.15.73:0/1538294360 shutdown_connections 2023-12-01T22:16:52.257 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:52.253+0000 7f284ba1f700 1 -- 172.21.15.73:0/1538294360 wait complete. 2023-12-01T22:16:52.257 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:52.254+0000 7f284ba1f700 1 Processor -- start 2023-12-01T22:16:52.257 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:52.254+0000 7f284ba1f700 1 -- start start 2023-12-01T22:16:52.257 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:52.254+0000 7f284ba1f700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28440f5410 0x7f28440fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:16:52.257 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:52.254+0000 7f284ba1f700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28440fb820 con 0x7f28440f5410 2023-12-01T22:16:52.257 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:52.254+0000 7f28497bb700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28440f5410 0x7f28440fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:16:52.257 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:52.254+0000 7f28497bb700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28440f5410 0x7f28440fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:50408/0 (socket says 172.21.15.73:50408) 2023-12-01T22:16:52.257 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:52.254+0000 7f28497bb700 1 -- 172.21.15.73:0/2163357399 learned_addr learned my addr 172.21.15.73:0/2163357399 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:16:52.257 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:52.254+0000 7f28497bb700 1 -- 172.21.15.73:0/2163357399 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2838005040 con 0x7f28440f5410 2023-12-01T22:16:52.258 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:52.255+0000 7f28497bb700 1 --2- 172.21.15.73:0/2163357399 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28440f5410 0x7f28440fd3a0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7f2838013510 tx=0x7f283800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:16:52.258 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:52.255+0000 7f2835ffb700 1 -- 172.21.15.73:0/2163357399 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2838009320 con 0x7f28440f5410 2023-12-01T22:16:52.258 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:52.255+0000 7f284ba1f700 1 -- 172.21.15.73:0/2163357399 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28440fb9c0 con 0x7f28440f5410 2023-12-01T22:16:52.258 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:52.255+0000 7f2835ffb700 1 -- 172.21.15.73:0/2163357399 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2838013f00 con 0x7f28440f5410 2023-12-01T22:16:52.258 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:52.255+0000 7f2835ffb700 1 -- 172.21.15.73:0/2163357399 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2838014d90 con 0x7f28440f5410 2023-12-01T22:16:52.258 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:52.255+0000 7f284ba1f700 1 -- 172.21.15.73:0/2163357399 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28440fbe50 con 0x7f28440f5410 2023-12-01T22:16:52.259 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:52.256+0000 7f2835ffb700 1 -- 172.21.15.73:0/2163357399 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f28380095f0 con 0x7f28440f5410 2023-12-01T22:16:52.259 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:52.256+0000 7f284ba1f700 1 -- 172.21.15.73:0/2163357399 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2844059100 con 0x7f28440f5410 2023-12-01T22:16:52.259 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:52.256+0000 7f2835ffb700 1 --2- 172.21.15.73:0/2163357399 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f283002ffb0 0x7f2830032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:16:52.259 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:52.256+0000 7f2835ffb700 1 -- 172.21.15.73:0/2163357399 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f28380410c0 con 0x7f28440f5410 2023-12-01T22:16:52.259 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:52.256+0000 7f2848fba700 1 --2- 172.21.15.73:0/2163357399 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f283002ffb0 0x7f2830032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:16:52.259 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:52.257+0000 7f2848fba700 1 --2- 172.21.15.73:0/2163357399 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f283002ffb0 0x7f2830032470 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f28400096a0 tx=0x7f2840006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:16:52.262 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:52.261+0000 7f2835ffb700 1 -- 172.21.15.73:0/2163357399 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f283801d7f0 con 0x7f28440f5410 2023-12-01T22:16:52.469 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:52.467+0000 7f284ba1f700 1 -- 172.21.15.73:0/2163357399 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f28440fae60 con 0x7f28440f5410 2023-12-01T22:16:52.469 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:52.468+0000 7f2835ffb700 1 -- 172.21.15.73:0/2163357399 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2838010230 con 0x7f28440f5410 2023-12-01T22:16:52.470 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:16:52.470 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:16:52.471 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:52.470+0000 7f284ba1f700 1 -- 172.21.15.73:0/2163357399 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f283002ffb0 msgr2=0x7f2830032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:16:52.472 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:52.470+0000 7f284ba1f700 1 --2- 172.21.15.73:0/2163357399 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f283002ffb0 0x7f2830032470 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f28400096a0 tx=0x7f2840006b40).stop 2023-12-01T22:16:52.472 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:52.470+0000 7f284ba1f700 1 -- 172.21.15.73:0/2163357399 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28440f5410 msgr2=0x7f28440fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:16:52.472 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:52.470+0000 7f284ba1f700 1 --2- 172.21.15.73:0/2163357399 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28440f5410 0x7f28440fd3a0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7f2838013510 tx=0x7f283800b040).stop 2023-12-01T22:16:52.472 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:52.470+0000 7f284ba1f700 1 -- 172.21.15.73:0/2163357399 shutdown_connections 2023-12-01T22:16:52.472 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:52.470+0000 7f284ba1f700 1 --2- 172.21.15.73:0/2163357399 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f283002ffb0 0x7f2830032470 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:16:52.472 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:52.470+0000 7f284ba1f700 1 --2- 172.21.15.73:0/2163357399 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28440f5410 0x7f28440fd3a0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:16:52.472 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:52.470+0000 7f284ba1f700 1 -- 172.21.15.73:0/2163357399 >> 172.21.15.73:0/2163357399 conn(0x7f28440f0b70 msgr2=0x7f284405fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:16:52.472 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:52.470+0000 7f284ba1f700 1 -- 172.21.15.73:0/2163357399 shutdown_connections 2023-12-01T22:16:52.472 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:52.470+0000 7f284ba1f700 1 -- 172.21.15.73:0/2163357399 wait complete. 2023-12-01T22:16:52.472 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:16:53.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:52 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:16:52.469306+0000 mon.smithi029 (mon.0) 248 : audit [DBG] from='client.? 172.21.15.73:0/2163357399' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:16:53.857 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:16:53.857 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:16:54.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:53 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:16:52.671989+0000 mgr.smithi029.loxrfu (mgr.14164) 176 : cluster [DBG] pgmap v164: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:16:55.087 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:55.084+0000 7f0cb528d700 1 -- 172.21.15.73:0/1219584981 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0cb00f5410 msgr2=0x7f0cb00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:16:55.087 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:55.084+0000 7f0cb528d700 1 --2- 172.21.15.73:0/1219584981 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0cb00f5410 0x7f0cb00f5830 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 rx=0x7f0ca4009fb0 tx=0x7f0ca4009b30).stop 2023-12-01T22:16:55.087 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:55.084+0000 7f0cb528d700 1 -- 172.21.15.73:0/1219584981 shutdown_connections 2023-12-01T22:16:55.087 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:55.084+0000 7f0cb528d700 1 --2- 172.21.15.73:0/1219584981 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0cb00f5410 0x7f0cb00f5830 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:16:55.087 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:55.084+0000 7f0cb528d700 1 -- 172.21.15.73:0/1219584981 >> 172.21.15.73:0/1219584981 conn(0x7f0cb00f0b70 msgr2=0x7f0cb00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:16:55.087 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:55.084+0000 7f0cb528d700 1 -- 172.21.15.73:0/1219584981 shutdown_connections 2023-12-01T22:16:55.088 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:55.084+0000 7f0cb528d700 1 -- 172.21.15.73:0/1219584981 wait complete. 2023-12-01T22:16:55.088 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:55.085+0000 7f0cb528d700 1 Processor -- start 2023-12-01T22:16:55.088 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:55.085+0000 7f0cb528d700 1 -- start start 2023-12-01T22:16:55.088 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:55.085+0000 7f0cb528d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0cb00f5410 0x7f0cb0106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:16:55.088 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:55.085+0000 7f0cb528d700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cb0106c10 con 0x7f0cb00f5410 2023-12-01T22:16:55.088 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:55.085+0000 7f0caeffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0cb00f5410 0x7f0cb0106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:16:55.088 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:55.085+0000 7f0caeffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0cb00f5410 0x7f0cb0106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:59348/0 (socket says 172.21.15.73:59348) 2023-12-01T22:16:55.088 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:55.085+0000 7f0caeffd700 1 -- 172.21.15.73:0/1577760244 learned_addr learned my addr 172.21.15.73:0/1577760244 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:16:55.088 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:55.086+0000 7f0caeffd700 1 -- 172.21.15.73:0/1577760244 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ca4005040 con 0x7f0cb00f5410 2023-12-01T22:16:55.088 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:55.086+0000 7f0caeffd700 1 --2- 172.21.15.73:0/1577760244 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0cb00f5410 0x7f0cb0106720 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7f0ca4013510 tx=0x7f0ca400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:16:55.089 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:55.086+0000 7f0c977fe700 1 -- 172.21.15.73:0/1577760244 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0ca4009320 con 0x7f0cb00f5410 2023-12-01T22:16:55.089 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:55.086+0000 7f0cb528d700 1 -- 172.21.15.73:0/1577760244 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cb0106e10 con 0x7f0cb00f5410 2023-12-01T22:16:55.089 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:55.086+0000 7f0cb528d700 1 -- 172.21.15.73:0/1577760244 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cb01072a0 con 0x7f0cb00f5410 2023-12-01T22:16:55.090 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:55.086+0000 7f0c977fe700 1 -- 172.21.15.73:0/1577760244 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0ca4013f00 con 0x7f0cb00f5410 2023-12-01T22:16:55.090 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:55.086+0000 7f0c977fe700 1 -- 172.21.15.73:0/1577760244 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0ca4014cc0 con 0x7f0cb00f5410 2023-12-01T22:16:55.090 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:55.087+0000 7f0c977fe700 1 -- 172.21.15.73:0/1577760244 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0ca4014420 con 0x7f0cb00f5410 2023-12-01T22:16:55.090 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:55.087+0000 7f0cb528d700 1 -- 172.21.15.73:0/1577760244 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c90004fa0 con 0x7f0cb00f5410 2023-12-01T22:16:55.090 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:55.087+0000 7f0c977fe700 1 --2- 172.21.15.73:0/1577760244 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f0c9802ff60 0x7f0c98032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:16:55.090 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:55.087+0000 7f0c977fe700 1 -- 172.21.15.73:0/1577760244 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0ca4041160 con 0x7f0cb00f5410 2023-12-01T22:16:55.090 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:55.087+0000 7f0cae7fc700 1 --2- 172.21.15.73:0/1577760244 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f0c9802ff60 0x7f0c98032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:16:55.090 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:55.088+0000 7f0cae7fc700 1 --2- 172.21.15.73:0/1577760244 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f0c9802ff60 0x7f0c98032420 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f0ca00096a0 tx=0x7f0ca0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:16:55.093 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:55.092+0000 7f0c977fe700 1 -- 172.21.15.73:0/1577760244 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0ca40095f0 con 0x7f0cb00f5410 2023-12-01T22:16:55.296 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:55.295+0000 7f0cb528d700 1 -- 172.21.15.73:0/1577760244 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0c90005670 con 0x7f0cb00f5410 2023-12-01T22:16:55.297 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:55.295+0000 7f0c977fe700 1 -- 172.21.15.73:0/1577760244 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0ca4012980 con 0x7f0cb00f5410 2023-12-01T22:16:55.297 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:16:55.297 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:16:55.299 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:55.297+0000 7f0cb528d700 1 -- 172.21.15.73:0/1577760244 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f0c9802ff60 msgr2=0x7f0c98032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:16:55.299 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:55.297+0000 7f0cb528d700 1 --2- 172.21.15.73:0/1577760244 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f0c9802ff60 0x7f0c98032420 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f0ca00096a0 tx=0x7f0ca0006b40).stop 2023-12-01T22:16:55.299 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:55.297+0000 7f0cb528d700 1 -- 172.21.15.73:0/1577760244 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0cb00f5410 msgr2=0x7f0cb0106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:16:55.299 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:55.297+0000 7f0cb528d700 1 --2- 172.21.15.73:0/1577760244 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0cb00f5410 0x7f0cb0106720 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7f0ca4013510 tx=0x7f0ca400b040).stop 2023-12-01T22:16:55.299 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:55.297+0000 7f0cb528d700 1 -- 172.21.15.73:0/1577760244 shutdown_connections 2023-12-01T22:16:55.299 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:55.297+0000 7f0cb528d700 1 --2- 172.21.15.73:0/1577760244 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f0c9802ff60 0x7f0c98032420 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:16:55.299 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:55.297+0000 7f0cb528d700 1 --2- 172.21.15.73:0/1577760244 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0cb00f5410 0x7f0cb0106720 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:16:55.299 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:55.297+0000 7f0cb528d700 1 -- 172.21.15.73:0/1577760244 >> 172.21.15.73:0/1577760244 conn(0x7f0cb00f0b70 msgr2=0x7f0cb005fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:16:55.300 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:55.297+0000 7f0cb528d700 1 -- 172.21.15.73:0/1577760244 shutdown_connections 2023-12-01T22:16:55.300 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:55.297+0000 7f0cb528d700 1 -- 172.21.15.73:0/1577760244 wait complete. 2023-12-01T22:16:55.300 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:16:56.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:55 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:16:54.672417+0000 mgr.smithi029.loxrfu (mgr.14164) 177 : cluster [DBG] pgmap v165: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:16:56.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:55 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:16:55.296828+0000 mon.smithi029 (mon.0) 249 : audit [DBG] from='client.? 172.21.15.73:0/1577760244' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:16:56.681 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:16:56.681 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:16:57.864 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:57.862+0000 7f8aef59e700 1 -- 172.21.15.73:0/2380598588 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8af00f62a0 msgr2=0x7f8af00f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:16:57.864 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:57.862+0000 7f8aef59e700 1 --2- 172.21.15.73:0/2380598588 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8af00f62a0 0x7f8af00f66c0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 rx=0x7f8ae4009fb0 tx=0x7f8ae4009b30).stop 2023-12-01T22:16:57.864 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:57.862+0000 7f8aef59e700 1 -- 172.21.15.73:0/2380598588 shutdown_connections 2023-12-01T22:16:57.865 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:57.862+0000 7f8aef59e700 1 --2- 172.21.15.73:0/2380598588 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8af00f62a0 0x7f8af00f66c0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:16:57.865 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:57.862+0000 7f8aef59e700 1 -- 172.21.15.73:0/2380598588 >> 172.21.15.73:0/2380598588 conn(0x7f8af00f1ac0 msgr2=0x7f8af00f3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:16:57.865 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:57.862+0000 7f8aef59e700 1 -- 172.21.15.73:0/2380598588 shutdown_connections 2023-12-01T22:16:57.865 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:57.862+0000 7f8aef59e700 1 -- 172.21.15.73:0/2380598588 wait complete. 2023-12-01T22:16:57.865 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:57.862+0000 7f8aef59e700 1 Processor -- start 2023-12-01T22:16:57.865 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:57.862+0000 7f8aef59e700 1 -- start start 2023-12-01T22:16:57.865 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:57.863+0000 7f8aef59e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8af00f62a0 0x7f8af01064a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:16:57.865 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:57.863+0000 7f8aef59e700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8af0106990 con 0x7f8af00f62a0 2023-12-01T22:16:57.867 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:57.863+0000 7f8aee59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8af00f62a0 0x7f8af01064a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:16:57.867 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:57.863+0000 7f8aee59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8af00f62a0 0x7f8af01064a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:59366/0 (socket says 172.21.15.73:59366) 2023-12-01T22:16:57.867 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:57.863+0000 7f8aee59c700 1 -- 172.21.15.73:0/1270388013 learned_addr learned my addr 172.21.15.73:0/1270388013 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:16:57.867 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:57.863+0000 7f8aee59c700 1 -- 172.21.15.73:0/1270388013 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ae4005040 con 0x7f8af00f62a0 2023-12-01T22:16:57.867 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:57.863+0000 7f8aee59c700 1 --2- 172.21.15.73:0/1270388013 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8af00f62a0 0x7f8af01064a0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7f8ae4013510 tx=0x7f8ae400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:16:57.867 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:57.864+0000 7f8adeffd700 1 -- 172.21.15.73:0/1270388013 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8ae4009320 con 0x7f8af00f62a0 2023-12-01T22:16:57.867 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:57.864+0000 7f8aef59e700 1 -- 172.21.15.73:0/1270388013 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8af0106b90 con 0x7f8af00f62a0 2023-12-01T22:16:57.867 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:57.864+0000 7f8adeffd700 1 -- 172.21.15.73:0/1270388013 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8ae4013f00 con 0x7f8af00f62a0 2023-12-01T22:16:57.867 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:57.864+0000 7f8adeffd700 1 -- 172.21.15.73:0/1270388013 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8ae4014dc0 con 0x7f8af00f62a0 2023-12-01T22:16:57.867 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:57.864+0000 7f8aef59e700 1 -- 172.21.15.73:0/1270388013 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8af0107020 con 0x7f8af00f62a0 2023-12-01T22:16:57.868 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:57.864+0000 7f8adeffd700 1 -- 172.21.15.73:0/1270388013 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8ae40095f0 con 0x7f8af00f62a0 2023-12-01T22:16:57.868 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:57.864+0000 7f8aef59e700 1 -- 172.21.15.73:0/1270388013 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8af0045ad0 con 0x7f8af00f62a0 2023-12-01T22:16:57.868 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:57.865+0000 7f8adeffd700 1 --2- 172.21.15.73:0/1270388013 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f8ad802ff60 0x7f8ad8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:16:57.868 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:57.865+0000 7f8adeffd700 1 -- 172.21.15.73:0/1270388013 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8ae4041040 con 0x7f8af00f62a0 2023-12-01T22:16:57.868 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:57.865+0000 7f8aedd9b700 1 --2- 172.21.15.73:0/1270388013 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f8ad802ff60 0x7f8ad8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:16:57.868 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:57.865+0000 7f8aedd9b700 1 --2- 172.21.15.73:0/1270388013 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f8ad802ff60 0x7f8ad8032420 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f8ae00096a0 tx=0x7f8ae0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:16:57.871 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:57.869+0000 7f8adeffd700 1 -- 172.21.15.73:0/1270388013 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8ae4014420 con 0x7f8af00f62a0 2023-12-01T22:16:58.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:57 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:16:56.672805+0000 mgr.smithi029.loxrfu (mgr.14164) 178 : cluster [DBG] pgmap v166: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:16:58.076 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:58.075+0000 7f8aef59e700 1 -- 172.21.15.73:0/1270388013 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8af0107ed0 con 0x7f8af00f62a0 2023-12-01T22:16:58.077 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:58.076+0000 7f8adeffd700 1 -- 172.21.15.73:0/1270388013 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8ae4010230 con 0x7f8af00f62a0 2023-12-01T22:16:58.077 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:16:58.077 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:16:58.079 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:58.077+0000 7f8aef59e700 1 -- 172.21.15.73:0/1270388013 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f8ad802ff60 msgr2=0x7f8ad8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:16:58.079 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:58.077+0000 7f8aef59e700 1 --2- 172.21.15.73:0/1270388013 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f8ad802ff60 0x7f8ad8032420 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f8ae00096a0 tx=0x7f8ae0006b40).stop 2023-12-01T22:16:58.079 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:58.077+0000 7f8aef59e700 1 -- 172.21.15.73:0/1270388013 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8af00f62a0 msgr2=0x7f8af01064a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:16:58.079 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:58.077+0000 7f8aef59e700 1 --2- 172.21.15.73:0/1270388013 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8af00f62a0 0x7f8af01064a0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7f8ae4013510 tx=0x7f8ae400b040).stop 2023-12-01T22:16:58.080 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:58.077+0000 7f8aef59e700 1 -- 172.21.15.73:0/1270388013 shutdown_connections 2023-12-01T22:16:58.080 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:58.077+0000 7f8aef59e700 1 --2- 172.21.15.73:0/1270388013 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f8ad802ff60 0x7f8ad8032420 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:16:58.080 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:58.077+0000 7f8aef59e700 1 --2- 172.21.15.73:0/1270388013 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8af00f62a0 0x7f8af01064a0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:16:58.080 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:58.077+0000 7f8aef59e700 1 -- 172.21.15.73:0/1270388013 >> 172.21.15.73:0/1270388013 conn(0x7f8af00f1ac0 msgr2=0x7f8af00f2560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:16:58.080 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:58.077+0000 7f8aef59e700 1 -- 172.21.15.73:0/1270388013 shutdown_connections 2023-12-01T22:16:58.080 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:16:58.077+0000 7f8aef59e700 1 -- 172.21.15.73:0/1270388013 wait complete. 2023-12-01T22:16:58.080 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:16:59.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:58 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:16:58.076892+0000 mon.smithi029 (mon.0) 250 : audit [DBG] from='client.? 172.21.15.73:0/1270388013' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:16:59.472 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:16:59.472 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:17:00.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:16:59 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:16:58.673235+0000 mgr.smithi029.loxrfu (mgr.14164) 179 : cluster [DBG] pgmap v167: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:17:00.686 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:00.684+0000 7fcfc2b25700 1 -- 172.21.15.73:0/490983344 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcfbc0f5410 msgr2=0x7fcfbc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:17:00.687 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:00.684+0000 7fcfc2b25700 1 --2- 172.21.15.73:0/490983344 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcfbc0f5410 0x7fcfbc0f5830 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 rx=0x7fcfb0009fb0 tx=0x7fcfb0009b30).stop 2023-12-01T22:17:00.687 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:00.684+0000 7fcfc2b25700 1 -- 172.21.15.73:0/490983344 shutdown_connections 2023-12-01T22:17:00.687 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:00.684+0000 7fcfc2b25700 1 --2- 172.21.15.73:0/490983344 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcfbc0f5410 0x7fcfbc0f5830 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:17:00.687 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:00.684+0000 7fcfc2b25700 1 -- 172.21.15.73:0/490983344 >> 172.21.15.73:0/490983344 conn(0x7fcfbc0f0b70 msgr2=0x7fcfbc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:17:00.687 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:00.684+0000 7fcfc2b25700 1 -- 172.21.15.73:0/490983344 shutdown_connections 2023-12-01T22:17:00.687 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:00.684+0000 7fcfc2b25700 1 -- 172.21.15.73:0/490983344 wait complete. 2023-12-01T22:17:00.687 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:00.685+0000 7fcfc2b25700 1 Processor -- start 2023-12-01T22:17:00.687 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:00.685+0000 7fcfc2b25700 1 -- start start 2023-12-01T22:17:00.687 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:00.685+0000 7fcfc2b25700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcfbc0f5410 0x7fcfbc107170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:17:00.687 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:00.685+0000 7fcfc2b25700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfbc0602f0 con 0x7fcfbc0f5410 2023-12-01T22:17:00.689 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:00.685+0000 7fcfc08c1700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcfbc0f5410 0x7fcfbc107170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:17:00.689 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:00.685+0000 7fcfc08c1700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcfbc0f5410 0x7fcfbc107170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:59392/0 (socket says 172.21.15.73:59392) 2023-12-01T22:17:00.689 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:00.685+0000 7fcfc08c1700 1 -- 172.21.15.73:0/2795772255 learned_addr learned my addr 172.21.15.73:0/2795772255 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:17:00.689 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:00.686+0000 7fcfc08c1700 1 -- 172.21.15.73:0/2795772255 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfb0005040 con 0x7fcfbc0f5410 2023-12-01T22:17:00.689 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:00.686+0000 7fcfc08c1700 1 --2- 172.21.15.73:0/2795772255 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcfbc0f5410 0x7fcfbc107170 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7fcfb0013510 tx=0x7fcfb000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:17:00.689 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:00.686+0000 7fcfb97fa700 1 -- 172.21.15.73:0/2795772255 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcfb0009320 con 0x7fcfbc0f5410 2023-12-01T22:17:00.689 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:00.686+0000 7fcfb97fa700 1 -- 172.21.15.73:0/2795772255 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcfb0013f00 con 0x7fcfbc0f5410 2023-12-01T22:17:00.689 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:00.686+0000 7fcfc2b25700 1 -- 172.21.15.73:0/2795772255 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfbc0604f0 con 0x7fcfbc0f5410 2023-12-01T22:17:00.689 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:00.686+0000 7fcfb97fa700 1 -- 172.21.15.73:0/2795772255 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcfb0014d90 con 0x7fcfbc0f5410 2023-12-01T22:17:00.690 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:00.686+0000 7fcfc2b25700 1 -- 172.21.15.73:0/2795772255 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfbc1078a0 con 0x7fcfbc0f5410 2023-12-01T22:17:00.690 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:00.687+0000 7fcfb97fa700 1 -- 172.21.15.73:0/2795772255 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcfb0014420 con 0x7fcfbc0f5410 2023-12-01T22:17:00.690 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:00.687+0000 7fcfc2b25700 1 -- 172.21.15.73:0/2795772255 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfbc045ad0 con 0x7fcfbc0f5410 2023-12-01T22:17:00.690 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:00.687+0000 7fcfb97fa700 1 --2- 172.21.15.73:0/2795772255 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fcfac02ff60 0x7fcfac032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:17:00.690 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:00.687+0000 7fcfb97fa700 1 -- 172.21.15.73:0/2795772255 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcfb00410c0 con 0x7fcfbc0f5410 2023-12-01T22:17:00.690 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:00.687+0000 7fcfbbfff700 1 --2- 172.21.15.73:0/2795772255 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fcfac02ff60 0x7fcfac032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:17:00.690 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:00.688+0000 7fcfbbfff700 1 --2- 172.21.15.73:0/2795772255 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fcfac02ff60 0x7fcfac032420 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7fcfa80096a0 tx=0x7fcfa8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:17:00.693 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:00.692+0000 7fcfb97fa700 1 -- 172.21.15.73:0/2795772255 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcfb00095f0 con 0x7fcfbc0f5410 2023-12-01T22:17:00.897 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:00.895+0000 7fcfc2b25700 1 -- 172.21.15.73:0/2795772255 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcfbc108170 con 0x7fcfbc0f5410 2023-12-01T22:17:00.897 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:00.896+0000 7fcfb97fa700 1 -- 172.21.15.73:0/2795772255 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fcfb001db70 con 0x7fcfbc0f5410 2023-12-01T22:17:00.898 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:17:00.898 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:17:00.899 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:00.898+0000 7fcfc2b25700 1 -- 172.21.15.73:0/2795772255 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fcfac02ff60 msgr2=0x7fcfac032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:17:00.899 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:00.898+0000 7fcfc2b25700 1 --2- 172.21.15.73:0/2795772255 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fcfac02ff60 0x7fcfac032420 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7fcfa80096a0 tx=0x7fcfa8006b40).stop 2023-12-01T22:17:00.899 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:00.898+0000 7fcfc2b25700 1 -- 172.21.15.73:0/2795772255 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcfbc0f5410 msgr2=0x7fcfbc107170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:17:00.900 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:00.898+0000 7fcfc2b25700 1 --2- 172.21.15.73:0/2795772255 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcfbc0f5410 0x7fcfbc107170 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7fcfb0013510 tx=0x7fcfb000b040).stop 2023-12-01T22:17:00.900 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:00.898+0000 7fcfc2b25700 1 -- 172.21.15.73:0/2795772255 shutdown_connections 2023-12-01T22:17:00.900 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:00.898+0000 7fcfc2b25700 1 --2- 172.21.15.73:0/2795772255 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fcfac02ff60 0x7fcfac032420 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:17:00.900 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:00.898+0000 7fcfc2b25700 1 --2- 172.21.15.73:0/2795772255 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcfbc0f5410 0x7fcfbc107170 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:17:00.900 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:00.898+0000 7fcfc2b25700 1 -- 172.21.15.73:0/2795772255 >> 172.21.15.73:0/2795772255 conn(0x7fcfbc0f0b70 msgr2=0x7fcfbc0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:17:00.900 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:00.898+0000 7fcfc2b25700 1 -- 172.21.15.73:0/2795772255 shutdown_connections 2023-12-01T22:17:00.900 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:00.898+0000 7fcfc2b25700 1 -- 172.21.15.73:0/2795772255 wait complete. 2023-12-01T22:17:00.900 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:17:01.776 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:17:01 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:17:00.673488+0000 mgr.smithi029.loxrfu (mgr.14164) 180 : cluster [DBG] pgmap v168: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:17:01.777 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:17:01 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:17:00.897299+0000 mon.smithi029 (mon.0) 251 : audit [DBG] from='client.? 172.21.15.73:0/2795772255' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:17:02.261 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:17:02.261 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:17:03.437 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:03.434+0000 7f204c90a700 1 -- 172.21.15.73:0/3908014611 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f20480f5410 msgr2=0x7f20480f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:17:03.437 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:03.434+0000 7f204c90a700 1 --2- 172.21.15.73:0/3908014611 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f20480f5410 0x7f20480f5830 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 rx=0x7f2030009fb0 tx=0x7f2030009b30).stop 2023-12-01T22:17:03.437 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:03.434+0000 7f204c90a700 1 -- 172.21.15.73:0/3908014611 shutdown_connections 2023-12-01T22:17:03.437 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:03.434+0000 7f204c90a700 1 --2- 172.21.15.73:0/3908014611 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f20480f5410 0x7f20480f5830 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:17:03.438 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:03.434+0000 7f204c90a700 1 -- 172.21.15.73:0/3908014611 >> 172.21.15.73:0/3908014611 conn(0x7f20480f0b70 msgr2=0x7f20480f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:17:03.438 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:03.434+0000 7f204c90a700 1 -- 172.21.15.73:0/3908014611 shutdown_connections 2023-12-01T22:17:03.438 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:03.434+0000 7f204c90a700 1 -- 172.21.15.73:0/3908014611 wait complete. 2023-12-01T22:17:03.438 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:03.435+0000 7f204c90a700 1 Processor -- start 2023-12-01T22:17:03.438 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:03.435+0000 7f204c90a700 1 -- start start 2023-12-01T22:17:03.438 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:03.435+0000 7f204c90a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f20480f5410 0x7f2048106710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:17:03.438 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:03.435+0000 7f204c90a700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2048106c00 con 0x7f20480f5410 2023-12-01T22:17:03.438 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:03.435+0000 7f204659c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f20480f5410 0x7f2048106710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:17:03.438 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:03.435+0000 7f204659c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f20480f5410 0x7f2048106710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:59404/0 (socket says 172.21.15.73:59404) 2023-12-01T22:17:03.438 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:03.435+0000 7f204659c700 1 -- 172.21.15.73:0/166117713 learned_addr learned my addr 172.21.15.73:0/166117713 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:17:03.438 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:03.435+0000 7f204659c700 1 -- 172.21.15.73:0/166117713 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2030005040 con 0x7f20480f5410 2023-12-01T22:17:03.438 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:03.435+0000 7f204659c700 1 --2- 172.21.15.73:0/166117713 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f20480f5410 0x7f2048106710 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7f2030013510 tx=0x7f203000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:17:03.438 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:03.436+0000 7f203effd700 1 -- 172.21.15.73:0/166117713 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2030009320 con 0x7f20480f5410 2023-12-01T22:17:03.438 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:03.436+0000 7f204c90a700 1 -- 172.21.15.73:0/166117713 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2048106e00 con 0x7f20480f5410 2023-12-01T22:17:03.439 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:03.436+0000 7f203effd700 1 -- 172.21.15.73:0/166117713 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2030013f00 con 0x7f20480f5410 2023-12-01T22:17:03.439 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:03.436+0000 7f203effd700 1 -- 172.21.15.73:0/166117713 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2030014d90 con 0x7f20480f5410 2023-12-01T22:17:03.439 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:03.436+0000 7f204c90a700 1 -- 172.21.15.73:0/166117713 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2048107290 con 0x7f20480f5410 2023-12-01T22:17:03.439 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:03.436+0000 7f203effd700 1 -- 172.21.15.73:0/166117713 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f20300095f0 con 0x7f20480f5410 2023-12-01T22:17:03.440 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:03.436+0000 7f204c90a700 1 -- 172.21.15.73:0/166117713 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2048045ad0 con 0x7f20480f5410 2023-12-01T22:17:03.440 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:03.437+0000 7f203effd700 1 --2- 172.21.15.73:0/166117713 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f203402ff60 0x7f2034032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:17:03.440 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:03.437+0000 7f203effd700 1 -- 172.21.15.73:0/166117713 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2030041000 con 0x7f20480f5410 2023-12-01T22:17:03.440 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:03.437+0000 7f2045d9b700 1 --2- 172.21.15.73:0/166117713 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f203402ff60 0x7f2034032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:17:03.440 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:03.437+0000 7f2045d9b700 1 --2- 172.21.15.73:0/166117713 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f203402ff60 0x7f2034032420 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f20380096a0 tx=0x7f2038006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:17:03.443 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:03.442+0000 7f203effd700 1 -- 172.21.15.73:0/166117713 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2030014420 con 0x7f20480f5410 2023-12-01T22:17:03.653 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:03.650+0000 7f204c90a700 1 -- 172.21.15.73:0/166117713 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f20480fae60 con 0x7f20480f5410 2023-12-01T22:17:03.653 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:03.650+0000 7f203effd700 1 -- 172.21.15.73:0/166117713 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2030010230 con 0x7f20480f5410 2023-12-01T22:17:03.654 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:17:03.654 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:17:03.654 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:03.652+0000 7f204c90a700 1 -- 172.21.15.73:0/166117713 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f203402ff60 msgr2=0x7f2034032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:17:03.654 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:03.652+0000 7f204c90a700 1 --2- 172.21.15.73:0/166117713 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f203402ff60 0x7f2034032420 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f20380096a0 tx=0x7f2038006b40).stop 2023-12-01T22:17:03.655 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:03.652+0000 7f204c90a700 1 -- 172.21.15.73:0/166117713 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f20480f5410 msgr2=0x7f2048106710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:17:03.655 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:03.652+0000 7f204c90a700 1 --2- 172.21.15.73:0/166117713 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f20480f5410 0x7f2048106710 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7f2030013510 tx=0x7f203000b040).stop 2023-12-01T22:17:03.655 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:03.652+0000 7f204c90a700 1 -- 172.21.15.73:0/166117713 shutdown_connections 2023-12-01T22:17:03.655 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:03.652+0000 7f204c90a700 1 --2- 172.21.15.73:0/166117713 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f203402ff60 0x7f2034032420 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:17:03.655 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:03.652+0000 7f204c90a700 1 --2- 172.21.15.73:0/166117713 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f20480f5410 0x7f2048106710 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:17:03.655 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:03.652+0000 7f204c90a700 1 -- 172.21.15.73:0/166117713 >> 172.21.15.73:0/166117713 conn(0x7f20480f0b70 msgr2=0x7f204805fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:17:03.655 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:03.652+0000 7f204c90a700 1 -- 172.21.15.73:0/166117713 shutdown_connections 2023-12-01T22:17:03.655 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:03.652+0000 7f204c90a700 1 -- 172.21.15.73:0/166117713 wait complete. 2023-12-01T22:17:03.655 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:17:04.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:17:03 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:17:02.673872+0000 mgr.smithi029.loxrfu (mgr.14164) 181 : cluster [DBG] pgmap v169: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:17:04.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:17:03 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:17:03.651802+0000 mon.smithi029 (mon.0) 252 : audit [DBG] from='client.? 172.21.15.73:0/166117713' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:17:05.047 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:17:05.048 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:17:06.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:17:05 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:17:04.674264+0000 mgr.smithi029.loxrfu (mgr.14164) 182 : cluster [DBG] pgmap v170: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:17:06.293 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:06.289+0000 7feaf4a5f700 1 -- 172.21.15.73:0/988076698 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feaf00601e0 msgr2=0x7feaf00f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:17:06.293 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:06.289+0000 7feaf4a5f700 1 --2- 172.21.15.73:0/988076698 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feaf00601e0 0x7feaf00f8f20 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 rx=0x7feae0009fb0 tx=0x7feae0009b30).stop 2023-12-01T22:17:06.293 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:06.289+0000 7feaf4a5f700 1 -- 172.21.15.73:0/988076698 shutdown_connections 2023-12-01T22:17:06.293 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:06.289+0000 7feaf4a5f700 1 --2- 172.21.15.73:0/988076698 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feaf00601e0 0x7feaf00f8f20 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:17:06.293 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:06.289+0000 7feaf4a5f700 1 -- 172.21.15.73:0/988076698 >> 172.21.15.73:0/988076698 conn(0x7feaf00f09a0 msgr2=0x7feaf00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:17:06.293 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:06.289+0000 7feaf4a5f700 1 -- 172.21.15.73:0/988076698 shutdown_connections 2023-12-01T22:17:06.293 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:06.289+0000 7feaf4a5f700 1 -- 172.21.15.73:0/988076698 wait complete. 2023-12-01T22:17:06.293 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:06.290+0000 7feaf4a5f700 1 Processor -- start 2023-12-01T22:17:06.294 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:06.290+0000 7feaf4a5f700 1 -- start start 2023-12-01T22:17:06.294 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:06.290+0000 7feaf4a5f700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feaf00601e0 0x7feaf0106500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:17:06.294 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:06.290+0000 7feaf4a5f700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feaf01069f0 con 0x7feaf00601e0 2023-12-01T22:17:06.294 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:06.290+0000 7feaef7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feaf00601e0 0x7feaf0106500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:17:06.294 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:06.290+0000 7feaef7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feaf00601e0 0x7feaf0106500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:39314/0 (socket says 172.21.15.73:39314) 2023-12-01T22:17:06.294 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:06.290+0000 7feaef7fe700 1 -- 172.21.15.73:0/4273684066 learned_addr learned my addr 172.21.15.73:0/4273684066 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:17:06.294 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:06.291+0000 7feaef7fe700 1 -- 172.21.15.73:0/4273684066 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feae0005040 con 0x7feaf00601e0 2023-12-01T22:17:06.294 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:06.291+0000 7feaef7fe700 1 --2- 172.21.15.73:0/4273684066 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feaf00601e0 0x7feaf0106500 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7feae0013510 tx=0x7feae000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:17:06.294 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:06.291+0000 7fead7fff700 1 -- 172.21.15.73:0/4273684066 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feae0009320 con 0x7feaf00601e0 2023-12-01T22:17:06.294 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:06.291+0000 7fead7fff700 1 -- 172.21.15.73:0/4273684066 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7feae0013f00 con 0x7feaf00601e0 2023-12-01T22:17:06.294 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:06.291+0000 7feaf4a5f700 1 -- 172.21.15.73:0/4273684066 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feaf0106bf0 con 0x7feaf00601e0 2023-12-01T22:17:06.294 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:06.291+0000 7fead7fff700 1 -- 172.21.15.73:0/4273684066 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feae0014d90 con 0x7feaf00601e0 2023-12-01T22:17:06.294 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:06.291+0000 7feaf4a5f700 1 -- 172.21.15.73:0/4273684066 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feaf0107080 con 0x7feaf00601e0 2023-12-01T22:17:06.295 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:06.292+0000 7fead7fff700 1 -- 172.21.15.73:0/4273684066 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7feae00263a0 con 0x7feaf00601e0 2023-12-01T22:17:06.295 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:06.292+0000 7fead7fff700 1 --2- 172.21.15.73:0/4273684066 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fead802ff60 0x7fead8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:17:06.295 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:06.292+0000 7fead7fff700 1 -- 172.21.15.73:0/4273684066 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7feae0040ff0 con 0x7feaf00601e0 2023-12-01T22:17:06.295 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:06.292+0000 7feaf4a5f700 1 -- 172.21.15.73:0/4273684066 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feaf0045ad0 con 0x7feaf00601e0 2023-12-01T22:17:06.295 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:06.292+0000 7feaeeffd700 1 --2- 172.21.15.73:0/4273684066 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fead802ff60 0x7fead8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:17:06.296 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:06.292+0000 7feaeeffd700 1 --2- 172.21.15.73:0/4273684066 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fead802ff60 0x7fead8032420 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7feae40096a0 tx=0x7feae4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:17:06.298 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:06.297+0000 7fead7fff700 1 -- 172.21.15.73:0/4273684066 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7feae0012920 con 0x7feaf00601e0 2023-12-01T22:17:06.503 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:06.501+0000 7feaf4a5f700 1 -- 172.21.15.73:0/4273684066 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7feaf0107da0 con 0x7feaf00601e0 2023-12-01T22:17:06.504 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:06.502+0000 7fead7fff700 1 -- 172.21.15.73:0/4273684066 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7feae002b020 con 0x7feaf00601e0 2023-12-01T22:17:06.504 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:17:06.504 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:17:06.505 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:06.504+0000 7feaf4a5f700 1 -- 172.21.15.73:0/4273684066 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fead802ff60 msgr2=0x7fead8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:17:06.506 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:06.504+0000 7feaf4a5f700 1 --2- 172.21.15.73:0/4273684066 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fead802ff60 0x7fead8032420 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7feae40096a0 tx=0x7feae4006b40).stop 2023-12-01T22:17:06.506 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:06.504+0000 7feaf4a5f700 1 -- 172.21.15.73:0/4273684066 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feaf00601e0 msgr2=0x7feaf0106500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:17:06.506 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:06.504+0000 7feaf4a5f700 1 --2- 172.21.15.73:0/4273684066 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feaf00601e0 0x7feaf0106500 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7feae0013510 tx=0x7feae000b040).stop 2023-12-01T22:17:06.506 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:06.504+0000 7feaf4a5f700 1 -- 172.21.15.73:0/4273684066 shutdown_connections 2023-12-01T22:17:06.506 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:06.504+0000 7feaf4a5f700 1 --2- 172.21.15.73:0/4273684066 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fead802ff60 0x7fead8032420 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:17:06.506 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:06.504+0000 7feaf4a5f700 1 --2- 172.21.15.73:0/4273684066 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feaf00601e0 0x7feaf0106500 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:17:06.506 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:06.504+0000 7feaf4a5f700 1 -- 172.21.15.73:0/4273684066 >> 172.21.15.73:0/4273684066 conn(0x7feaf00f09a0 msgr2=0x7feaf00f1420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:17:06.506 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:06.504+0000 7feaf4a5f700 1 -- 172.21.15.73:0/4273684066 shutdown_connections 2023-12-01T22:17:06.506 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:06.504+0000 7feaf4a5f700 1 -- 172.21.15.73:0/4273684066 wait complete. 2023-12-01T22:17:06.506 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:17:07.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:17:06 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:17:06.503464+0000 mon.smithi029 (mon.0) 253 : audit [DBG] from='client.? 172.21.15.73:0/4273684066' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:17:07.898 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:17:07.899 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:17:08.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:17:07 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:17:06.674550+0000 mgr.smithi029.loxrfu (mgr.14164) 183 : cluster [DBG] pgmap v171: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:17:09.114 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:09.112+0000 7fecc1aa9700 1 -- 172.21.15.73:0/1182087709 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fecbc0f62a0 msgr2=0x7fecbc0f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:17:09.114 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:09.112+0000 7fecc1aa9700 1 --2- 172.21.15.73:0/1182087709 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fecbc0f62a0 0x7fecbc0f66c0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 rx=0x7fecb0009fb0 tx=0x7fecb0009b30).stop 2023-12-01T22:17:09.114 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:09.112+0000 7fecc1aa9700 1 -- 172.21.15.73:0/1182087709 shutdown_connections 2023-12-01T22:17:09.114 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:09.112+0000 7fecc1aa9700 1 --2- 172.21.15.73:0/1182087709 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fecbc0f62a0 0x7fecbc0f66c0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:17:09.114 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:09.112+0000 7fecc1aa9700 1 -- 172.21.15.73:0/1182087709 >> 172.21.15.73:0/1182087709 conn(0x7fecbc0f1aa0 msgr2=0x7fecbc0f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:17:09.114 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:09.112+0000 7fecc1aa9700 1 -- 172.21.15.73:0/1182087709 shutdown_connections 2023-12-01T22:17:09.114 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:09.112+0000 7fecc1aa9700 1 -- 172.21.15.73:0/1182087709 wait complete. 2023-12-01T22:17:09.114 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:09.113+0000 7fecc1aa9700 1 Processor -- start 2023-12-01T22:17:09.114 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:09.113+0000 7fecc1aa9700 1 -- start start 2023-12-01T22:17:09.114 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:09.113+0000 7fecc1aa9700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fecbc0f62a0 0x7fecbc1064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:17:09.115 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:09.113+0000 7fecc1aa9700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecbc1069a0 con 0x7fecbc0f62a0 2023-12-01T22:17:09.115 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:09.113+0000 7fecc0aa7700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fecbc0f62a0 0x7fecbc1064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:17:09.115 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:09.113+0000 7fecc0aa7700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fecbc0f62a0 0x7fecbc1064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:39336/0 (socket says 172.21.15.73:39336) 2023-12-01T22:17:09.115 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:09.113+0000 7fecc0aa7700 1 -- 172.21.15.73:0/1906222133 learned_addr learned my addr 172.21.15.73:0/1906222133 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:17:09.116 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:09.114+0000 7fecc0aa7700 1 -- 172.21.15.73:0/1906222133 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecb0005040 con 0x7fecbc0f62a0 2023-12-01T22:17:09.116 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:09.114+0000 7fecc0aa7700 1 --2- 172.21.15.73:0/1906222133 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fecbc0f62a0 0x7fecbc1064b0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7fecb0009830 tx=0x7fecb000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:17:09.116 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:09.114+0000 7fecb97fa700 1 -- 172.21.15.73:0/1906222133 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fecb0009320 con 0x7fecbc0f62a0 2023-12-01T22:17:09.116 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:09.114+0000 7fecc1aa9700 1 -- 172.21.15.73:0/1906222133 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecbc106ba0 con 0x7fecbc0f62a0 2023-12-01T22:17:09.117 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:09.114+0000 7fecb97fa700 1 -- 172.21.15.73:0/1906222133 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fecb00136c0 con 0x7fecbc0f62a0 2023-12-01T22:17:09.117 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:09.114+0000 7fecb97fa700 1 -- 172.21.15.73:0/1906222133 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fecb001ee90 con 0x7fecbc0f62a0 2023-12-01T22:17:09.117 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:09.114+0000 7fecc1aa9700 1 -- 172.21.15.73:0/1906222133 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecbc107030 con 0x7fecbc0f62a0 2023-12-01T22:17:09.117 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:09.115+0000 7fecb97fa700 1 -- 172.21.15.73:0/1906222133 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fecb001e420 con 0x7fecbc0f62a0 2023-12-01T22:17:09.117 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:09.115+0000 7fecc1aa9700 1 -- 172.21.15.73:0/1906222133 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fecbc045ad0 con 0x7fecbc0f62a0 2023-12-01T22:17:09.117 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:09.115+0000 7fecb97fa700 1 --2- 172.21.15.73:0/1906222133 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fecac02ff10 0x7fecac0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:17:09.117 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:09.115+0000 7fecb97fa700 1 -- 172.21.15.73:0/1906222133 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fecb0040fe0 con 0x7fecbc0f62a0 2023-12-01T22:17:09.117 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:09.115+0000 7fecbbfff700 1 --2- 172.21.15.73:0/1906222133 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fecac02ff10 0x7fecac0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:17:09.117 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:09.115+0000 7fecbbfff700 1 --2- 172.21.15.73:0/1906222133 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fecac02ff10 0x7fecac0323d0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7feca80096a0 tx=0x7feca8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:17:09.120 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:09.120+0000 7fecb97fa700 1 -- 172.21.15.73:0/1906222133 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fecb00095f0 con 0x7fecbc0f62a0 2023-12-01T22:17:09.324 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:09.323+0000 7fecc1aa9700 1 -- 172.21.15.73:0/1906222133 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fecbc107ee0 con 0x7fecbc0f62a0 2023-12-01T22:17:09.325 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:09.324+0000 7fecb97fa700 1 -- 172.21.15.73:0/1906222133 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fecb002c2d0 con 0x7fecbc0f62a0 2023-12-01T22:17:09.325 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:17:09.325 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:17:09.326 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:09.325+0000 7fecc1aa9700 1 -- 172.21.15.73:0/1906222133 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fecac02ff10 msgr2=0x7fecac0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:17:09.326 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:09.325+0000 7fecc1aa9700 1 --2- 172.21.15.73:0/1906222133 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fecac02ff10 0x7fecac0323d0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7feca80096a0 tx=0x7feca8006b40).stop 2023-12-01T22:17:09.326 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:09.325+0000 7fecc1aa9700 1 -- 172.21.15.73:0/1906222133 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fecbc0f62a0 msgr2=0x7fecbc1064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:17:09.326 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:09.325+0000 7fecc1aa9700 1 --2- 172.21.15.73:0/1906222133 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fecbc0f62a0 0x7fecbc1064b0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7fecb0009830 tx=0x7fecb000b040).stop 2023-12-01T22:17:09.327 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:09.326+0000 7fecc1aa9700 1 -- 172.21.15.73:0/1906222133 shutdown_connections 2023-12-01T22:17:09.327 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:09.326+0000 7fecc1aa9700 1 --2- 172.21.15.73:0/1906222133 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fecac02ff10 0x7fecac0323d0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:17:09.327 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:09.326+0000 7fecc1aa9700 1 --2- 172.21.15.73:0/1906222133 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fecbc0f62a0 0x7fecbc1064b0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:17:09.327 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:09.326+0000 7fecc1aa9700 1 -- 172.21.15.73:0/1906222133 >> 172.21.15.73:0/1906222133 conn(0x7fecbc0f1aa0 msgr2=0x7fecbc0f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:17:09.327 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:09.326+0000 7fecc1aa9700 1 -- 172.21.15.73:0/1906222133 shutdown_connections 2023-12-01T22:17:09.327 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:09.326+0000 7fecc1aa9700 1 -- 172.21.15.73:0/1906222133 wait complete. 2023-12-01T22:17:09.327 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:17:10.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:17:09 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:17:08.674892+0000 mgr.smithi029.loxrfu (mgr.14164) 184 : cluster [DBG] pgmap v172: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:17:10.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:17:09 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:17:09.324468+0000 mon.smithi029 (mon.0) 254 : audit [DBG] from='client.? 172.21.15.73:0/1906222133' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:17:10.718 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:17:10.719 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:17:11.776 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:17:11 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:17:10.675243+0000 mgr.smithi029.loxrfu (mgr.14164) 185 : cluster [DBG] pgmap v173: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:17:11.881 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:11.879+0000 7fa5a8272700 1 -- 172.21.15.73:0/3631041816 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5a00601e0 msgr2=0x7fa5a00f9190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:17:11.882 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:11.879+0000 7fa5a8272700 1 --2- 172.21.15.73:0/3631041816 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5a00601e0 0x7fa5a00f9190 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 rx=0x7fa590009fb0 tx=0x7fa590009b30).stop 2023-12-01T22:17:11.882 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:11.879+0000 7fa5a8272700 1 -- 172.21.15.73:0/3631041816 shutdown_connections 2023-12-01T22:17:11.882 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:11.879+0000 7fa5a8272700 1 --2- 172.21.15.73:0/3631041816 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5a00601e0 0x7fa5a00f9190 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:17:11.882 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:11.880+0000 7fa5a8272700 1 -- 172.21.15.73:0/3631041816 >> 172.21.15.73:0/3631041816 conn(0x7fa5a00f0b70 msgr2=0x7fa5a00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:17:11.882 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:11.880+0000 7fa5a8272700 1 -- 172.21.15.73:0/3631041816 shutdown_connections 2023-12-01T22:17:11.882 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:11.880+0000 7fa5a8272700 1 -- 172.21.15.73:0/3631041816 wait complete. 2023-12-01T22:17:11.882 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:11.880+0000 7fa5a8272700 1 Processor -- start 2023-12-01T22:17:11.882 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:11.880+0000 7fa5a8272700 1 -- start start 2023-12-01T22:17:11.882 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:11.880+0000 7fa5a8272700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5a00601e0 0x7fa5a01066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:17:11.882 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:11.880+0000 7fa5a8272700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5a0106bd0 con 0x7fa5a00601e0 2023-12-01T22:17:11.882 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:11.880+0000 7fa5a600e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5a00601e0 0x7fa5a01066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:17:11.882 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:11.880+0000 7fa5a600e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5a00601e0 0x7fa5a01066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:39352/0 (socket says 172.21.15.73:39352) 2023-12-01T22:17:11.882 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:11.880+0000 7fa5a600e700 1 -- 172.21.15.73:0/2180775209 learned_addr learned my addr 172.21.15.73:0/2180775209 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:17:11.883 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:11.881+0000 7fa5a600e700 1 -- 172.21.15.73:0/2180775209 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa590005040 con 0x7fa5a00601e0 2023-12-01T22:17:11.883 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:11.881+0000 7fa5a600e700 1 --2- 172.21.15.73:0/2180775209 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5a00601e0 0x7fa5a01066e0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7fa590004dd0 tx=0x7fa590009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:17:11.883 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:11.881+0000 7fa596ffd700 1 -- 172.21.15.73:0/2180775209 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa590018070 con 0x7fa5a00601e0 2023-12-01T22:17:11.883 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:11.881+0000 7fa5a8272700 1 -- 172.21.15.73:0/2180775209 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5a0106dd0 con 0x7fa5a00601e0 2023-12-01T22:17:11.884 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:11.881+0000 7fa596ffd700 1 -- 172.21.15.73:0/2180775209 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa590013410 con 0x7fa5a00601e0 2023-12-01T22:17:11.884 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:11.881+0000 7fa596ffd700 1 -- 172.21.15.73:0/2180775209 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa59001c3c0 con 0x7fa5a00601e0 2023-12-01T22:17:11.884 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:11.881+0000 7fa5a8272700 1 -- 172.21.15.73:0/2180775209 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5a0107260 con 0x7fa5a00601e0 2023-12-01T22:17:11.884 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:11.882+0000 7fa596ffd700 1 -- 172.21.15.73:0/2180775209 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa59001c520 con 0x7fa5a00601e0 2023-12-01T22:17:11.884 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:11.882+0000 7fa596ffd700 1 --2- 172.21.15.73:0/2180775209 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fa58c02ffb0 0x7fa58c032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:17:11.884 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:11.882+0000 7fa5a8272700 1 -- 172.21.15.73:0/2180775209 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5a0045ad0 con 0x7fa5a00601e0 2023-12-01T22:17:11.884 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:11.882+0000 7fa596ffd700 1 -- 172.21.15.73:0/2180775209 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa590041710 con 0x7fa5a00601e0 2023-12-01T22:17:11.884 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:11.882+0000 7fa5a580d700 1 --2- 172.21.15.73:0/2180775209 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fa58c02ffb0 0x7fa58c032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:17:11.884 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:11.883+0000 7fa5a580d700 1 --2- 172.21.15.73:0/2180775209 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fa58c02ffb0 0x7fa58c032470 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7fa59c0096a0 tx=0x7fa59c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:17:11.887 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:11.887+0000 7fa596ffd700 1 -- 172.21.15.73:0/2180775209 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa59001c920 con 0x7fa5a00601e0 2023-12-01T22:17:12.094 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:12.093+0000 7fa5a8272700 1 -- 172.21.15.73:0/2180775209 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa5a0108110 con 0x7fa5a00601e0 2023-12-01T22:17:12.104 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:12.094+0000 7fa596ffd700 1 -- 172.21.15.73:0/2180775209 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa590004490 con 0x7fa5a00601e0 2023-12-01T22:17:12.104 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:17:12.105 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:17:12.105 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:12.095+0000 7fa5a8272700 1 -- 172.21.15.73:0/2180775209 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fa58c02ffb0 msgr2=0x7fa58c032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:17:12.105 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:12.095+0000 7fa5a8272700 1 --2- 172.21.15.73:0/2180775209 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fa58c02ffb0 0x7fa58c032470 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7fa59c0096a0 tx=0x7fa59c006b40).stop 2023-12-01T22:17:12.105 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:12.095+0000 7fa5a8272700 1 -- 172.21.15.73:0/2180775209 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5a00601e0 msgr2=0x7fa5a01066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:17:12.105 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:12.095+0000 7fa5a8272700 1 --2- 172.21.15.73:0/2180775209 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5a00601e0 0x7fa5a01066e0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7fa590004dd0 tx=0x7fa590009b30).stop 2023-12-01T22:17:12.106 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:12.095+0000 7fa5a8272700 1 -- 172.21.15.73:0/2180775209 shutdown_connections 2023-12-01T22:17:12.106 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:12.095+0000 7fa5a8272700 1 --2- 172.21.15.73:0/2180775209 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fa58c02ffb0 0x7fa58c032470 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:17:12.106 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:12.095+0000 7fa5a8272700 1 --2- 172.21.15.73:0/2180775209 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5a00601e0 0x7fa5a01066e0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:17:12.106 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:12.095+0000 7fa5a8272700 1 -- 172.21.15.73:0/2180775209 >> 172.21.15.73:0/2180775209 conn(0x7fa5a00f0b70 msgr2=0x7fa5a00f1670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:17:12.106 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:12.095+0000 7fa5a8272700 1 -- 172.21.15.73:0/2180775209 shutdown_connections 2023-12-01T22:17:12.106 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:12.095+0000 7fa5a8272700 1 -- 172.21.15.73:0/2180775209 wait complete. 2023-12-01T22:17:12.106 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:17:12.777 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:17:12 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:17:12.094142+0000 mon.smithi029 (mon.0) 255 : audit [DBG] from='client.? 172.21.15.73:0/2180775209' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:17:13.489 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:17:13.489 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:17:13.776 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:17:13 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:17:12.675538+0000 mgr.smithi029.loxrfu (mgr.14164) 186 : cluster [DBG] pgmap v174: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:17:13.777 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:17:13 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:17:12.781902+0000 mon.smithi029 (mon.0) 256 : audit [INF] from='mgr.14164 172.21.15.29:0/3195927111' entity='mgr.smithi029.loxrfu' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi029.loxrfu/mirror_snapshot_schedule"}]: dispatch 2023-12-01T22:17:13.777 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:17:13 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:17:12.785281+0000 mon.smithi029 (mon.0) 257 : audit [INF] from='mgr.14164 172.21.15.29:0/3195927111' entity='mgr.smithi029.loxrfu' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi029.loxrfu/trash_purge_schedule"}]: dispatch 2023-12-01T22:17:14.667 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:14.666+0000 7fcf400dd700 1 -- 172.21.15.73:0/1523658135 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf380f5410 msgr2=0x7fcf380f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:17:14.667 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:14.666+0000 7fcf400dd700 1 --2- 172.21.15.73:0/1523658135 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf380f5410 0x7fcf380f5830 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 rx=0x7fcf28009fb0 tx=0x7fcf28009b30).stop 2023-12-01T22:17:14.667 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:14.666+0000 7fcf400dd700 1 -- 172.21.15.73:0/1523658135 shutdown_connections 2023-12-01T22:17:14.668 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:14.666+0000 7fcf400dd700 1 --2- 172.21.15.73:0/1523658135 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf380f5410 0x7fcf380f5830 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:17:14.668 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:14.666+0000 7fcf400dd700 1 -- 172.21.15.73:0/1523658135 >> 172.21.15.73:0/1523658135 conn(0x7fcf380f0b70 msgr2=0x7fcf380f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:17:14.668 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:14.666+0000 7fcf400dd700 1 -- 172.21.15.73:0/1523658135 shutdown_connections 2023-12-01T22:17:14.668 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:14.666+0000 7fcf400dd700 1 -- 172.21.15.73:0/1523658135 wait complete. 2023-12-01T22:17:14.669 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:14.667+0000 7fcf400dd700 1 Processor -- start 2023-12-01T22:17:14.669 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:14.667+0000 7fcf400dd700 1 -- start start 2023-12-01T22:17:14.669 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:14.667+0000 7fcf400dd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf380f5410 0x7fcf38105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:17:14.670 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:14.667+0000 7fcf400dd700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf38105b20 con 0x7fcf380f5410 2023-12-01T22:17:14.670 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:14.667+0000 7fcf3de79700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf380f5410 0x7fcf38105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:17:14.670 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:14.667+0000 7fcf3de79700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf380f5410 0x7fcf38105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:39330/0 (socket says 172.21.15.73:39330) 2023-12-01T22:17:14.670 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:14.667+0000 7fcf3de79700 1 -- 172.21.15.73:0/3077953800 learned_addr learned my addr 172.21.15.73:0/3077953800 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:17:14.670 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:14.668+0000 7fcf3de79700 1 -- 172.21.15.73:0/3077953800 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf28005040 con 0x7fcf380f5410 2023-12-01T22:17:14.670 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:14.668+0000 7fcf3de79700 1 --2- 172.21.15.73:0/3077953800 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf380f5410 0x7fcf38105630 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7fcf28013510 tx=0x7fcf2800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:17:14.670 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:14.668+0000 7fcf2e7fc700 1 -- 172.21.15.73:0/3077953800 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcf28009320 con 0x7fcf380f5410 2023-12-01T22:17:14.670 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:14.668+0000 7fcf400dd700 1 -- 172.21.15.73:0/3077953800 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf38105d20 con 0x7fcf380f5410 2023-12-01T22:17:14.670 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:14.668+0000 7fcf2e7fc700 1 -- 172.21.15.73:0/3077953800 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcf28013f00 con 0x7fcf380f5410 2023-12-01T22:17:14.670 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:14.668+0000 7fcf400dd700 1 -- 172.21.15.73:0/3077953800 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf381061b0 con 0x7fcf380f5410 2023-12-01T22:17:14.670 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:14.668+0000 7fcf2e7fc700 1 -- 172.21.15.73:0/3077953800 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcf28014db0 con 0x7fcf380f5410 2023-12-01T22:17:14.670 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:14.669+0000 7fcf2e7fc700 1 -- 172.21.15.73:0/3077953800 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcf2801e420 con 0x7fcf380f5410 2023-12-01T22:17:14.670 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:14.669+0000 7fcf2e7fc700 1 --2- 172.21.15.73:0/3077953800 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fcf2402ff60 0x7fcf24032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:17:14.670 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:14.669+0000 7fcf400dd700 1 -- 172.21.15.73:0/3077953800 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf38045ad0 con 0x7fcf380f5410 2023-12-01T22:17:14.671 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:14.669+0000 7fcf2e7fc700 1 -- 172.21.15.73:0/3077953800 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcf28040ff0 con 0x7fcf380f5410 2023-12-01T22:17:14.671 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:14.669+0000 7fcf3d678700 1 --2- 172.21.15.73:0/3077953800 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fcf2402ff60 0x7fcf24032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:17:14.671 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:14.670+0000 7fcf3d678700 1 --2- 172.21.15.73:0/3077953800 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fcf2402ff60 0x7fcf24032420 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7fcf340096a0 tx=0x7fcf34006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:17:14.674 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:14.674+0000 7fcf2e7fc700 1 -- 172.21.15.73:0/3077953800 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcf280273f0 con 0x7fcf380f5410 2023-12-01T22:17:14.881 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:14.881+0000 7fcf400dd700 1 -- 172.21.15.73:0/3077953800 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcf380faf50 con 0x7fcf380f5410 2023-12-01T22:17:14.882 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:14.881+0000 7fcf2e7fc700 1 -- 172.21.15.73:0/3077953800 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fcf2801c970 con 0x7fcf380f5410 2023-12-01T22:17:14.882 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:17:14.882 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:17:14.883 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:14.883+0000 7fcf400dd700 1 -- 172.21.15.73:0/3077953800 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fcf2402ff60 msgr2=0x7fcf24032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:17:14.884 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:14.883+0000 7fcf400dd700 1 --2- 172.21.15.73:0/3077953800 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fcf2402ff60 0x7fcf24032420 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7fcf340096a0 tx=0x7fcf34006b40).stop 2023-12-01T22:17:14.884 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:14.883+0000 7fcf400dd700 1 -- 172.21.15.73:0/3077953800 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf380f5410 msgr2=0x7fcf38105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:17:14.884 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:14.883+0000 7fcf400dd700 1 --2- 172.21.15.73:0/3077953800 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf380f5410 0x7fcf38105630 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7fcf28013510 tx=0x7fcf2800b040).stop 2023-12-01T22:17:14.884 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:14.883+0000 7fcf400dd700 1 -- 172.21.15.73:0/3077953800 shutdown_connections 2023-12-01T22:17:14.884 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:14.883+0000 7fcf400dd700 1 --2- 172.21.15.73:0/3077953800 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fcf2402ff60 0x7fcf24032420 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:17:14.884 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:14.883+0000 7fcf400dd700 1 --2- 172.21.15.73:0/3077953800 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf380f5410 0x7fcf38105630 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:17:14.884 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:14.883+0000 7fcf400dd700 1 -- 172.21.15.73:0/3077953800 >> 172.21.15.73:0/3077953800 conn(0x7fcf380f0b70 msgr2=0x7fcf380f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:17:14.884 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:14.883+0000 7fcf400dd700 1 -- 172.21.15.73:0/3077953800 shutdown_connections 2023-12-01T22:17:14.884 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:14.883+0000 7fcf400dd700 1 -- 172.21.15.73:0/3077953800 wait complete. 2023-12-01T22:17:14.884 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:17:16.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:17:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:17:14.675816+0000 mgr.smithi029.loxrfu (mgr.14164) 187 : cluster [DBG] pgmap v175: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:17:16.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:17:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:17:14.881643+0000 mon.smithi029 (mon.0) 258 : audit [DBG] from='client.? 172.21.15.73:0/3077953800' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:17:16.249 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:17:16.249 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:17:17.355 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:17.350+0000 7f600a6e3700 1 -- 172.21.15.73:0/3873510568 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60040f5410 msgr2=0x7f60040f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:17:17.355 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:17.350+0000 7f600a6e3700 1 --2- 172.21.15.73:0/3873510568 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60040f5410 0x7f60040f5830 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 rx=0x7f5ff4009fb0 tx=0x7f5ff4009b30).stop 2023-12-01T22:17:17.355 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:17.350+0000 7f600a6e3700 1 -- 172.21.15.73:0/3873510568 shutdown_connections 2023-12-01T22:17:17.355 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:17.350+0000 7f600a6e3700 1 --2- 172.21.15.73:0/3873510568 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60040f5410 0x7f60040f5830 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:17:17.355 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:17.350+0000 7f600a6e3700 1 -- 172.21.15.73:0/3873510568 >> 172.21.15.73:0/3873510568 conn(0x7f60040f0b70 msgr2=0x7f60040f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:17:17.355 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:17.350+0000 7f600a6e3700 1 -- 172.21.15.73:0/3873510568 shutdown_connections 2023-12-01T22:17:17.355 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:17.350+0000 7f600a6e3700 1 -- 172.21.15.73:0/3873510568 wait complete. 2023-12-01T22:17:17.355 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:17.351+0000 7f600a6e3700 1 Processor -- start 2023-12-01T22:17:17.356 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:17.351+0000 7f600a6e3700 1 -- start start 2023-12-01T22:17:17.356 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:17.351+0000 7f600a6e3700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60040f5410 0x7f6004106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:17:17.356 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:17.351+0000 7f600a6e3700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6004106c10 con 0x7f60040f5410 2023-12-01T22:17:17.356 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:17.351+0000 7f6003fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60040f5410 0x7f6004106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:17:17.356 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:17.351+0000 7f6003fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60040f5410 0x7f6004106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:39354/0 (socket says 172.21.15.73:39354) 2023-12-01T22:17:17.356 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:17.351+0000 7f6003fff700 1 -- 172.21.15.73:0/2584395517 learned_addr learned my addr 172.21.15.73:0/2584395517 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:17:17.356 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:17.352+0000 7f6003fff700 1 -- 172.21.15.73:0/2584395517 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ff4005040 con 0x7f60040f5410 2023-12-01T22:17:17.356 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:17.352+0000 7f6003fff700 1 --2- 172.21.15.73:0/2584395517 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60040f5410 0x7f6004106720 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7f5ff4013510 tx=0x7f5ff400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:17:17.356 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:17.352+0000 7f6000ff9700 1 -- 172.21.15.73:0/2584395517 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5ff4009320 con 0x7f60040f5410 2023-12-01T22:17:17.356 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:17.352+0000 7f600a6e3700 1 -- 172.21.15.73:0/2584395517 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6004106e10 con 0x7f60040f5410 2023-12-01T22:17:17.356 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:17.352+0000 7f6000ff9700 1 -- 172.21.15.73:0/2584395517 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5ff4013f00 con 0x7f60040f5410 2023-12-01T22:17:17.356 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:17.352+0000 7f6000ff9700 1 -- 172.21.15.73:0/2584395517 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5ff4014d90 con 0x7f60040f5410 2023-12-01T22:17:17.356 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:17.352+0000 7f600a6e3700 1 -- 172.21.15.73:0/2584395517 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60041072a0 con 0x7f60040f5410 2023-12-01T22:17:17.356 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:17.353+0000 7f6000ff9700 1 -- 172.21.15.73:0/2584395517 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5ff4014420 con 0x7f60040f5410 2023-12-01T22:17:17.357 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:17.353+0000 7f600a6e3700 1 -- 172.21.15.73:0/2584395517 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6004045ad0 con 0x7f60040f5410 2023-12-01T22:17:17.357 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:17.353+0000 7f6000ff9700 1 --2- 172.21.15.73:0/2584395517 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f5fec02ff60 0x7f5fec032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:17:17.357 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:17.353+0000 7f6000ff9700 1 -- 172.21.15.73:0/2584395517 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5ff40410b0 con 0x7f60040f5410 2023-12-01T22:17:17.357 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:17.353+0000 7f60037fe700 1 --2- 172.21.15.73:0/2584395517 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f5fec02ff60 0x7f5fec032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:17:17.357 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:17.354+0000 7f60037fe700 1 --2- 172.21.15.73:0/2584395517 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f5fec02ff60 0x7f5fec032420 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f5ff80096a0 tx=0x7f5ff8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:17:17.358 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:17.358+0000 7f6000ff9700 1 -- 172.21.15.73:0/2584395517 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5ff40095f0 con 0x7f60040f5410 2023-12-01T22:17:17.565 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:17.564+0000 7f600a6e3700 1 -- 172.21.15.73:0/2584395517 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f60041080f0 con 0x7f60040f5410 2023-12-01T22:17:17.565 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:17.565+0000 7f6000ff9700 1 -- 172.21.15.73:0/2584395517 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5ff401db70 con 0x7f60040f5410 2023-12-01T22:17:17.565 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:17:17.566 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:17:17.567 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:17.567+0000 7f600a6e3700 1 -- 172.21.15.73:0/2584395517 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f5fec02ff60 msgr2=0x7f5fec032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:17:17.567 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:17.567+0000 7f600a6e3700 1 --2- 172.21.15.73:0/2584395517 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f5fec02ff60 0x7f5fec032420 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f5ff80096a0 tx=0x7f5ff8006b40).stop 2023-12-01T22:17:17.567 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:17.567+0000 7f600a6e3700 1 -- 172.21.15.73:0/2584395517 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60040f5410 msgr2=0x7f6004106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:17:17.567 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:17.567+0000 7f600a6e3700 1 --2- 172.21.15.73:0/2584395517 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60040f5410 0x7f6004106720 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7f5ff4013510 tx=0x7f5ff400b040).stop 2023-12-01T22:17:17.568 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:17.567+0000 7f600a6e3700 1 -- 172.21.15.73:0/2584395517 shutdown_connections 2023-12-01T22:17:17.568 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:17.567+0000 7f600a6e3700 1 --2- 172.21.15.73:0/2584395517 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f5fec02ff60 0x7f5fec032420 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:17:17.568 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:17.567+0000 7f600a6e3700 1 --2- 172.21.15.73:0/2584395517 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60040f5410 0x7f6004106720 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:17:17.568 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:17.567+0000 7f600a6e3700 1 -- 172.21.15.73:0/2584395517 >> 172.21.15.73:0/2584395517 conn(0x7f60040f0b70 msgr2=0x7f600405fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:17:17.568 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:17.567+0000 7f600a6e3700 1 -- 172.21.15.73:0/2584395517 shutdown_connections 2023-12-01T22:17:17.568 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:17.567+0000 7f600a6e3700 1 -- 172.21.15.73:0/2584395517 wait complete. 2023-12-01T22:17:17.568 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:17:18.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:17:17 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:17:16.676222+0000 mgr.smithi029.loxrfu (mgr.14164) 188 : cluster [DBG] pgmap v176: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:17:18.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:17:17 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:17:17.565175+0000 mon.smithi029 (mon.0) 259 : audit [DBG] from='client.? 172.21.15.73:0/2584395517' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:17:18.952 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:17:18.952 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:17:20.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:17:19 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:17:18.676605+0000 mgr.smithi029.loxrfu (mgr.14164) 189 : cluster [DBG] pgmap v177: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:17:20.108 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:20.106+0000 7f3f28d8d700 1 -- 172.21.15.73:0/65765494 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3f240f5410 msgr2=0x7f3f240f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:17:20.108 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:20.106+0000 7f3f28d8d700 1 --2- 172.21.15.73:0/65765494 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3f240f5410 0x7f3f240f5830 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 rx=0x7f3f0c009fb0 tx=0x7f3f0c009b30).stop 2023-12-01T22:17:20.108 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:20.106+0000 7f3f28d8d700 1 -- 172.21.15.73:0/65765494 shutdown_connections 2023-12-01T22:17:20.108 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:20.106+0000 7f3f28d8d700 1 --2- 172.21.15.73:0/65765494 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3f240f5410 0x7f3f240f5830 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:17:20.108 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:20.106+0000 7f3f28d8d700 1 -- 172.21.15.73:0/65765494 >> 172.21.15.73:0/65765494 conn(0x7f3f240f0b70 msgr2=0x7f3f240f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:17:20.108 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:20.106+0000 7f3f28d8d700 1 -- 172.21.15.73:0/65765494 shutdown_connections 2023-12-01T22:17:20.108 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:20.106+0000 7f3f28d8d700 1 -- 172.21.15.73:0/65765494 wait complete. 2023-12-01T22:17:20.109 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:20.107+0000 7f3f28d8d700 1 Processor -- start 2023-12-01T22:17:20.110 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:20.107+0000 7f3f28d8d700 1 -- start start 2023-12-01T22:17:20.110 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:20.107+0000 7f3f28d8d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3f240f5410 0x7f3f24105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:17:20.110 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:20.107+0000 7f3f28d8d700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f24105b20 con 0x7f3f240f5410 2023-12-01T22:17:20.110 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:20.107+0000 7f3f2259c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3f240f5410 0x7f3f24105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:17:20.110 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:20.107+0000 7f3f2259c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3f240f5410 0x7f3f24105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:39374/0 (socket says 172.21.15.73:39374) 2023-12-01T22:17:20.110 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:20.107+0000 7f3f2259c700 1 -- 172.21.15.73:0/1036671165 learned_addr learned my addr 172.21.15.73:0/1036671165 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:17:20.110 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:20.107+0000 7f3f2259c700 1 -- 172.21.15.73:0/1036671165 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f0c005040 con 0x7f3f240f5410 2023-12-01T22:17:20.110 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:20.108+0000 7f3f2259c700 1 --2- 172.21.15.73:0/1036671165 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3f240f5410 0x7f3f24105630 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7f3f0c014000 tx=0x7f3f0c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:17:20.110 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:20.108+0000 7f3f1affd700 1 -- 172.21.15.73:0/1036671165 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3f0c0095f0 con 0x7f3f240f5410 2023-12-01T22:17:20.110 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:20.108+0000 7f3f28d8d700 1 -- 172.21.15.73:0/1036671165 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f24105d20 con 0x7f3f240f5410 2023-12-01T22:17:20.110 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:20.108+0000 7f3f1affd700 1 -- 172.21.15.73:0/1036671165 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3f0c007e70 con 0x7f3f240f5410 2023-12-01T22:17:20.110 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:20.108+0000 7f3f1affd700 1 -- 172.21.15.73:0/1036671165 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3f0c014ea0 con 0x7f3f240f5410 2023-12-01T22:17:20.110 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:20.108+0000 7f3f28d8d700 1 -- 172.21.15.73:0/1036671165 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f241061b0 con 0x7f3f240f5410 2023-12-01T22:17:20.111 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:20.109+0000 7f3f28d8d700 1 -- 172.21.15.73:0/1036671165 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f24045ad0 con 0x7f3f240f5410 2023-12-01T22:17:20.111 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:20.109+0000 7f3f1affd700 1 -- 172.21.15.73:0/1036671165 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3f0c025070 con 0x7f3f240f5410 2023-12-01T22:17:20.111 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:20.109+0000 7f3f1affd700 1 --2- 172.21.15.73:0/1036671165 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f3f1002ff60 0x7f3f10032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:17:20.111 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:20.109+0000 7f3f1affd700 1 -- 172.21.15.73:0/1036671165 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3f0c0409f0 con 0x7f3f240f5410 2023-12-01T22:17:20.112 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:20.109+0000 7f3f21d9b700 1 --2- 172.21.15.73:0/1036671165 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f3f1002ff60 0x7f3f10032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:17:20.112 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:20.109+0000 7f3f21d9b700 1 --2- 172.21.15.73:0/1036671165 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f3f1002ff60 0x7f3f10032420 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f3f140096a0 tx=0x7f3f14006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:17:20.115 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:20.114+0000 7f3f1affd700 1 -- 172.21.15.73:0/1036671165 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3f0c0079c0 con 0x7f3f240f5410 2023-12-01T22:17:20.321 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:20.319+0000 7f3f28d8d700 1 -- 172.21.15.73:0/1036671165 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3f240fae60 con 0x7f3f240f5410 2023-12-01T22:17:20.322 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:20.320+0000 7f3f1affd700 1 -- 172.21.15.73:0/1036671165 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3f0c00f300 con 0x7f3f240f5410 2023-12-01T22:17:20.322 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:17:20.322 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:17:20.324 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:20.322+0000 7f3f28d8d700 1 -- 172.21.15.73:0/1036671165 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f3f1002ff60 msgr2=0x7f3f10032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:17:20.324 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:20.322+0000 7f3f28d8d700 1 --2- 172.21.15.73:0/1036671165 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f3f1002ff60 0x7f3f10032420 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f3f140096a0 tx=0x7f3f14006b40).stop 2023-12-01T22:17:20.324 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:20.322+0000 7f3f28d8d700 1 -- 172.21.15.73:0/1036671165 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3f240f5410 msgr2=0x7f3f24105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:17:20.324 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:20.322+0000 7f3f28d8d700 1 --2- 172.21.15.73:0/1036671165 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3f240f5410 0x7f3f24105630 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7f3f0c014000 tx=0x7f3f0c00b040).stop 2023-12-01T22:17:20.324 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:20.322+0000 7f3f28d8d700 1 -- 172.21.15.73:0/1036671165 shutdown_connections 2023-12-01T22:17:20.324 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:20.322+0000 7f3f28d8d700 1 --2- 172.21.15.73:0/1036671165 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f3f1002ff60 0x7f3f10032420 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:17:20.324 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:20.322+0000 7f3f28d8d700 1 --2- 172.21.15.73:0/1036671165 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3f240f5410 0x7f3f24105630 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:17:20.324 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:20.322+0000 7f3f28d8d700 1 -- 172.21.15.73:0/1036671165 >> 172.21.15.73:0/1036671165 conn(0x7f3f240f0b70 msgr2=0x7f3f240f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:17:20.324 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:20.322+0000 7f3f28d8d700 1 -- 172.21.15.73:0/1036671165 shutdown_connections 2023-12-01T22:17:20.324 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:20.322+0000 7f3f28d8d700 1 -- 172.21.15.73:0/1036671165 wait complete. 2023-12-01T22:17:20.325 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:17:21.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:17:20 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:17:20.321433+0000 mon.smithi029 (mon.0) 260 : audit [DBG] from='client.? 172.21.15.73:0/1036671165' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:17:21.755 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:17:21.755 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:17:22.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:17:21 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:17:20.676987+0000 mgr.smithi029.loxrfu (mgr.14164) 190 : cluster [DBG] pgmap v178: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:17:22.962 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:22.959+0000 7fc77c0b4700 1 -- 172.21.15.73:0/932536714 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7740f6a00 msgr2=0x7fc7740f6e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:17:22.963 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:22.959+0000 7fc77c0b4700 1 --2- 172.21.15.73:0/932536714 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7740f6a00 0x7fc7740f6e20 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 rx=0x7fc76c009960 tx=0x7fc76c004d10).stop 2023-12-01T22:17:22.963 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:22.959+0000 7fc77c0b4700 1 -- 172.21.15.73:0/932536714 shutdown_connections 2023-12-01T22:17:22.963 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:22.959+0000 7fc77c0b4700 1 --2- 172.21.15.73:0/932536714 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7740f6a00 0x7fc7740f6e20 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:17:22.963 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:22.959+0000 7fc77c0b4700 1 -- 172.21.15.73:0/932536714 >> 172.21.15.73:0/932536714 conn(0x7fc7740f0260 msgr2=0x7fc7740f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:17:22.963 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:22.959+0000 7fc77c0b4700 1 -- 172.21.15.73:0/932536714 shutdown_connections 2023-12-01T22:17:22.963 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:22.959+0000 7fc77c0b4700 1 -- 172.21.15.73:0/932536714 wait complete. 2023-12-01T22:17:22.963 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:22.960+0000 7fc77c0b4700 1 Processor -- start 2023-12-01T22:17:22.963 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:22.960+0000 7fc77c0b4700 1 -- start start 2023-12-01T22:17:22.963 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:22.960+0000 7fc77c0b4700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7740f6a00 0x7fc7741093b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:17:22.963 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:22.960+0000 7fc77c0b4700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7740f52b0 con 0x7fc7740f6a00 2023-12-01T22:17:22.964 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:22.960+0000 7fc779e50700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7740f6a00 0x7fc7741093b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:17:22.964 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:22.960+0000 7fc779e50700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7740f6a00 0x7fc7741093b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:39384/0 (socket says 172.21.15.73:39384) 2023-12-01T22:17:22.964 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:22.960+0000 7fc779e50700 1 -- 172.21.15.73:0/1470198087 learned_addr learned my addr 172.21.15.73:0/1470198087 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:17:22.964 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:22.960+0000 7fc779e50700 1 -- 172.21.15.73:0/1470198087 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc76c009640 con 0x7fc7740f6a00 2023-12-01T22:17:22.964 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:22.960+0000 7fc779e50700 1 --2- 172.21.15.73:0/1470198087 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7740f6a00 0x7fc7741093b0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7fc76c007bd0 tx=0x7fc76c004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:17:22.964 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:22.961+0000 7fc76a7fc700 1 -- 172.21.15.73:0/1470198087 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc76c018070 con 0x7fc7740f6a00 2023-12-01T22:17:22.964 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:22.961+0000 7fc76a7fc700 1 -- 172.21.15.73:0/1470198087 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc76c00b870 con 0x7fc7740f6a00 2023-12-01T22:17:22.964 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:22.961+0000 7fc77c0b4700 1 -- 172.21.15.73:0/1470198087 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7740f54b0 con 0x7fc7740f6a00 2023-12-01T22:17:22.964 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:22.961+0000 7fc76a7fc700 1 -- 172.21.15.73:0/1470198087 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc76c01d420 con 0x7fc7740f6a00 2023-12-01T22:17:22.964 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:22.961+0000 7fc77c0b4700 1 -- 172.21.15.73:0/1470198087 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc774109a60 con 0x7fc7740f6a00 2023-12-01T22:17:22.965 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:22.962+0000 7fc76a7fc700 1 -- 172.21.15.73:0/1470198087 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc76c024060 con 0x7fc7740f6a00 2023-12-01T22:17:22.965 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:22.962+0000 7fc77c0b4700 1 -- 172.21.15.73:0/1470198087 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc774045ad0 con 0x7fc7740f6a00 2023-12-01T22:17:22.965 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:22.962+0000 7fc76a7fc700 1 --2- 172.21.15.73:0/1470198087 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fc76002ff60 0x7fc760032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:17:22.965 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:22.962+0000 7fc76a7fc700 1 -- 172.21.15.73:0/1470198087 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc76c0411d0 con 0x7fc7740f6a00 2023-12-01T22:17:22.965 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:22.962+0000 7fc77964f700 1 --2- 172.21.15.73:0/1470198087 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fc76002ff60 0x7fc760032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:17:22.965 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:22.962+0000 7fc77964f700 1 --2- 172.21.15.73:0/1470198087 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fc76002ff60 0x7fc760032420 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7fc7640096a0 tx=0x7fc764006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:17:22.968 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:22.967+0000 7fc76a7fc700 1 -- 172.21.15.73:0/1470198087 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc76c011790 con 0x7fc7740f6a00 2023-12-01T22:17:23.173 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:23.171+0000 7fc77c0b4700 1 -- 172.21.15.73:0/1470198087 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc7740faef0 con 0x7fc7740f6a00 2023-12-01T22:17:23.174 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:23.172+0000 7fc76a7fc700 1 -- 172.21.15.73:0/1470198087 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc76c011790 con 0x7fc7740f6a00 2023-12-01T22:17:23.174 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:17:23.174 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:17:23.175 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:23.173+0000 7fc77c0b4700 1 -- 172.21.15.73:0/1470198087 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fc76002ff60 msgr2=0x7fc760032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:17:23.175 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:23.173+0000 7fc77c0b4700 1 --2- 172.21.15.73:0/1470198087 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fc76002ff60 0x7fc760032420 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7fc7640096a0 tx=0x7fc764006b40).stop 2023-12-01T22:17:23.175 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:23.173+0000 7fc77c0b4700 1 -- 172.21.15.73:0/1470198087 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7740f6a00 msgr2=0x7fc7741093b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:17:23.175 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:23.173+0000 7fc77c0b4700 1 --2- 172.21.15.73:0/1470198087 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7740f6a00 0x7fc7741093b0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7fc76c007bd0 tx=0x7fc76c004d10).stop 2023-12-01T22:17:23.175 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:23.173+0000 7fc77c0b4700 1 -- 172.21.15.73:0/1470198087 shutdown_connections 2023-12-01T22:17:23.176 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:23.173+0000 7fc77c0b4700 1 --2- 172.21.15.73:0/1470198087 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fc76002ff60 0x7fc760032420 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:17:23.176 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:23.173+0000 7fc77c0b4700 1 --2- 172.21.15.73:0/1470198087 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7740f6a00 0x7fc7741093b0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:17:23.176 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:23.173+0000 7fc77c0b4700 1 -- 172.21.15.73:0/1470198087 >> 172.21.15.73:0/1470198087 conn(0x7fc7740f0260 msgr2=0x7fc7740f0d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:17:23.176 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:23.173+0000 7fc77c0b4700 1 -- 172.21.15.73:0/1470198087 shutdown_connections 2023-12-01T22:17:23.176 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:23.173+0000 7fc77c0b4700 1 -- 172.21.15.73:0/1470198087 wait complete. 2023-12-01T22:17:23.176 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:17:24.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:17:23 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:17:22.677344+0000 mgr.smithi029.loxrfu (mgr.14164) 191 : cluster [DBG] pgmap v179: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:17:24.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:17:23 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:17:23.173145+0000 mon.smithi029 (mon.0) 261 : audit [DBG] from='client.? 172.21.15.73:0/1470198087' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:17:24.597 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:17:24.597 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:17:25.750 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:25.748+0000 7fd57d485700 1 -- 172.21.15.73:0/535220753 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd5780f62a0 msgr2=0x7fd5780f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:17:25.750 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:25.748+0000 7fd57d485700 1 --2- 172.21.15.73:0/535220753 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd5780f62a0 0x7fd5780f66c0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 rx=0x7fd568009fb0 tx=0x7fd568009b30).stop 2023-12-01T22:17:25.750 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:25.748+0000 7fd57d485700 1 -- 172.21.15.73:0/535220753 shutdown_connections 2023-12-01T22:17:25.750 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:25.748+0000 7fd57d485700 1 --2- 172.21.15.73:0/535220753 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd5780f62a0 0x7fd5780f66c0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:17:25.750 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:25.748+0000 7fd57d485700 1 -- 172.21.15.73:0/535220753 >> 172.21.15.73:0/535220753 conn(0x7fd5780f1aa0 msgr2=0x7fd5780f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:17:25.750 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:25.748+0000 7fd57d485700 1 -- 172.21.15.73:0/535220753 shutdown_connections 2023-12-01T22:17:25.750 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:25.748+0000 7fd57d485700 1 -- 172.21.15.73:0/535220753 wait complete. 2023-12-01T22:17:25.751 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:25.748+0000 7fd57d485700 1 Processor -- start 2023-12-01T22:17:25.752 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:25.748+0000 7fd57d485700 1 -- start start 2023-12-01T22:17:25.752 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:25.749+0000 7fd57d485700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd5780f62a0 0x7fd5781064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:17:25.752 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:25.749+0000 7fd57d485700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5781069a0 con 0x7fd5780f62a0 2023-12-01T22:17:25.752 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:25.749+0000 7fd577fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd5780f62a0 0x7fd5781064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:17:25.752 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:25.749+0000 7fd577fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd5780f62a0 0x7fd5781064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:40204/0 (socket says 172.21.15.73:40204) 2023-12-01T22:17:25.752 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:25.749+0000 7fd577fff700 1 -- 172.21.15.73:0/487040791 learned_addr learned my addr 172.21.15.73:0/487040791 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:17:25.752 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:25.749+0000 7fd577fff700 1 -- 172.21.15.73:0/487040791 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd568005040 con 0x7fd5780f62a0 2023-12-01T22:17:25.752 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:25.749+0000 7fd577fff700 1 --2- 172.21.15.73:0/487040791 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd5780f62a0 0x7fd5781064b0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7fd568013510 tx=0x7fd56800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:17:25.752 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:25.750+0000 7fd574ff9700 1 -- 172.21.15.73:0/487040791 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd568009320 con 0x7fd5780f62a0 2023-12-01T22:17:25.752 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:25.750+0000 7fd57d485700 1 -- 172.21.15.73:0/487040791 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd578106ba0 con 0x7fd5780f62a0 2023-12-01T22:17:25.752 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:25.750+0000 7fd574ff9700 1 -- 172.21.15.73:0/487040791 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd568013f00 con 0x7fd5780f62a0 2023-12-01T22:17:25.752 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:25.750+0000 7fd574ff9700 1 -- 172.21.15.73:0/487040791 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd568014dc0 con 0x7fd5780f62a0 2023-12-01T22:17:25.752 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:25.750+0000 7fd57d485700 1 -- 172.21.15.73:0/487040791 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd578107030 con 0x7fd5780f62a0 2023-12-01T22:17:25.753 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:25.751+0000 7fd574ff9700 1 -- 172.21.15.73:0/487040791 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd5680095f0 con 0x7fd5780f62a0 2023-12-01T22:17:25.753 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:25.751+0000 7fd57d485700 1 -- 172.21.15.73:0/487040791 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd578045ad0 con 0x7fd5780f62a0 2023-12-01T22:17:25.753 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:25.751+0000 7fd574ff9700 1 --2- 172.21.15.73:0/487040791 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fd56002ff60 0x7fd560032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:17:25.753 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:25.751+0000 7fd574ff9700 1 -- 172.21.15.73:0/487040791 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd568041020 con 0x7fd5780f62a0 2023-12-01T22:17:25.754 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:25.751+0000 7fd5777fe700 1 --2- 172.21.15.73:0/487040791 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fd56002ff60 0x7fd560032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:17:25.754 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:25.751+0000 7fd5777fe700 1 --2- 172.21.15.73:0/487040791 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fd56002ff60 0x7fd560032420 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7fd56c0096a0 tx=0x7fd56c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:17:25.757 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:25.756+0000 7fd574ff9700 1 -- 172.21.15.73:0/487040791 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd568014420 con 0x7fd5780f62a0 2023-12-01T22:17:25.962 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:25.960+0000 7fd57d485700 1 -- 172.21.15.73:0/487040791 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd578107ee0 con 0x7fd5780f62a0 2023-12-01T22:17:25.962 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:25.960+0000 7fd574ff9700 1 -- 172.21.15.73:0/487040791 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd56801c920 con 0x7fd5780f62a0 2023-12-01T22:17:25.962 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:17:25.963 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:17:25.964 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:25.962+0000 7fd57d485700 1 -- 172.21.15.73:0/487040791 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fd56002ff60 msgr2=0x7fd560032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:17:25.964 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:25.962+0000 7fd57d485700 1 --2- 172.21.15.73:0/487040791 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fd56002ff60 0x7fd560032420 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7fd56c0096a0 tx=0x7fd56c006b40).stop 2023-12-01T22:17:25.964 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:25.962+0000 7fd57d485700 1 -- 172.21.15.73:0/487040791 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd5780f62a0 msgr2=0x7fd5781064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:17:25.964 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:25.962+0000 7fd57d485700 1 --2- 172.21.15.73:0/487040791 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd5780f62a0 0x7fd5781064b0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7fd568013510 tx=0x7fd56800b040).stop 2023-12-01T22:17:25.964 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:25.962+0000 7fd57d485700 1 -- 172.21.15.73:0/487040791 shutdown_connections 2023-12-01T22:17:25.964 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:25.962+0000 7fd57d485700 1 --2- 172.21.15.73:0/487040791 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fd56002ff60 0x7fd560032420 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:17:25.964 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:25.962+0000 7fd57d485700 1 --2- 172.21.15.73:0/487040791 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd5780f62a0 0x7fd5781064b0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:17:25.964 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:25.962+0000 7fd57d485700 1 -- 172.21.15.73:0/487040791 >> 172.21.15.73:0/487040791 conn(0x7fd5780f1aa0 msgr2=0x7fd5780f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:17:25.964 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:25.962+0000 7fd57d485700 1 -- 172.21.15.73:0/487040791 shutdown_connections 2023-12-01T22:17:25.964 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:25.962+0000 7fd57d485700 1 -- 172.21.15.73:0/487040791 wait complete. 2023-12-01T22:17:25.964 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:17:26.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:17:25 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:17:24.677720+0000 mgr.smithi029.loxrfu (mgr.14164) 192 : cluster [DBG] pgmap v180: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:17:27.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:17:26 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:17:25.961927+0000 mon.smithi029 (mon.0) 262 : audit [DBG] from='client.? 172.21.15.73:0/487040791' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:17:27.364 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:17:27.365 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:17:28.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:17:27 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:17:26.678123+0000 mgr.smithi029.loxrfu (mgr.14164) 193 : cluster [DBG] pgmap v181: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:17:28.516 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:28.513+0000 7f042c4c8700 1 -- 172.21.15.73:0/3257343462 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f04240f6510 msgr2=0x7f04240f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:17:28.516 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:28.513+0000 7f042c4c8700 1 --2- 172.21.15.73:0/3257343462 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f04240f6510 0x7f04240f6930 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 rx=0x7f0414009fb0 tx=0x7f0414009b30).stop 2023-12-01T22:17:28.517 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:28.513+0000 7f042c4c8700 1 -- 172.21.15.73:0/3257343462 shutdown_connections 2023-12-01T22:17:28.517 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:28.513+0000 7f042c4c8700 1 --2- 172.21.15.73:0/3257343462 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f04240f6510 0x7f04240f6930 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:17:28.517 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:28.513+0000 7f042c4c8700 1 -- 172.21.15.73:0/3257343462 >> 172.21.15.73:0/3257343462 conn(0x7f04240f1c70 msgr2=0x7f04240f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:17:28.517 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:28.513+0000 7f042c4c8700 1 -- 172.21.15.73:0/3257343462 shutdown_connections 2023-12-01T22:17:28.517 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:28.513+0000 7f042c4c8700 1 -- 172.21.15.73:0/3257343462 wait complete. 2023-12-01T22:17:28.517 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:28.514+0000 7f042c4c8700 1 Processor -- start 2023-12-01T22:17:28.517 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:28.514+0000 7f042c4c8700 1 -- start start 2023-12-01T22:17:28.517 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:28.514+0000 7f042c4c8700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f04240f6510 0x7f0424106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:17:28.517 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:28.514+0000 7f042c4c8700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0424106c20 con 0x7f04240f6510 2023-12-01T22:17:28.517 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:28.514+0000 7f042a264700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f04240f6510 0x7f0424106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:17:28.517 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:28.514+0000 7f042a264700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f04240f6510 0x7f0424106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:40214/0 (socket says 172.21.15.73:40214) 2023-12-01T22:17:28.517 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:28.514+0000 7f042a264700 1 -- 172.21.15.73:0/2179072739 learned_addr learned my addr 172.21.15.73:0/2179072739 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:17:28.517 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:28.514+0000 7f042a264700 1 -- 172.21.15.73:0/2179072739 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0414005040 con 0x7f04240f6510 2023-12-01T22:17:28.517 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:28.515+0000 7f042a264700 1 --2- 172.21.15.73:0/2179072739 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f04240f6510 0x7f0424106730 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7f0414014000 tx=0x7f041400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:17:28.517 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:28.515+0000 7f041affd700 1 -- 172.21.15.73:0/2179072739 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f04140095f0 con 0x7f04240f6510 2023-12-01T22:17:28.518 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:28.515+0000 7f042c4c8700 1 -- 172.21.15.73:0/2179072739 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0424106e20 con 0x7f04240f6510 2023-12-01T22:17:28.518 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:28.515+0000 7f041affd700 1 -- 172.21.15.73:0/2179072739 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0414007e70 con 0x7f04240f6510 2023-12-01T22:17:28.518 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:28.515+0000 7f041affd700 1 -- 172.21.15.73:0/2179072739 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0414014ea0 con 0x7f04240f6510 2023-12-01T22:17:28.518 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:28.515+0000 7f042c4c8700 1 -- 172.21.15.73:0/2179072739 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04241072b0 con 0x7f04240f6510 2023-12-01T22:17:28.518 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:28.516+0000 7f041affd700 1 -- 172.21.15.73:0/2179072739 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0414025070 con 0x7f04240f6510 2023-12-01T22:17:28.519 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:28.516+0000 7f042c4c8700 1 -- 172.21.15.73:0/2179072739 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0424045ad0 con 0x7f04240f6510 2023-12-01T22:17:28.519 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:28.516+0000 7f041affd700 1 --2- 172.21.15.73:0/2179072739 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f041002ff60 0x7f0410032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:17:28.519 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:28.516+0000 7f041affd700 1 -- 172.21.15.73:0/2179072739 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0414040970 con 0x7f04240f6510 2023-12-01T22:17:28.519 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:28.516+0000 7f0429a63700 1 --2- 172.21.15.73:0/2179072739 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f041002ff60 0x7f0410032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:17:28.519 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:28.516+0000 7f0429a63700 1 --2- 172.21.15.73:0/2179072739 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f041002ff60 0x7f0410032420 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f04200096a0 tx=0x7f0420006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:17:28.522 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:28.520+0000 7f041affd700 1 -- 172.21.15.73:0/2179072739 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f04140079c0 con 0x7f04240f6510 2023-12-01T22:17:28.724 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:28.723+0000 7f042c4c8700 1 -- 172.21.15.73:0/2179072739 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f04240fbf60 con 0x7f04240f6510 2023-12-01T22:17:28.725 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:28.724+0000 7f041affd700 1 -- 172.21.15.73:0/2179072739 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f041400f300 con 0x7f04240f6510 2023-12-01T22:17:28.725 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:17:28.725 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:17:28.727 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:28.725+0000 7f042c4c8700 1 -- 172.21.15.73:0/2179072739 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f041002ff60 msgr2=0x7f0410032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:17:28.727 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:28.725+0000 7f042c4c8700 1 --2- 172.21.15.73:0/2179072739 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f041002ff60 0x7f0410032420 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f04200096a0 tx=0x7f0420006b40).stop 2023-12-01T22:17:28.727 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:28.725+0000 7f042c4c8700 1 -- 172.21.15.73:0/2179072739 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f04240f6510 msgr2=0x7f0424106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:17:28.727 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:28.725+0000 7f042c4c8700 1 --2- 172.21.15.73:0/2179072739 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f04240f6510 0x7f0424106730 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7f0414014000 tx=0x7f041400b040).stop 2023-12-01T22:17:28.727 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:28.725+0000 7f042c4c8700 1 -- 172.21.15.73:0/2179072739 shutdown_connections 2023-12-01T22:17:28.728 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:28.725+0000 7f042c4c8700 1 --2- 172.21.15.73:0/2179072739 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f041002ff60 0x7f0410032420 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:17:28.728 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:28.725+0000 7f042c4c8700 1 --2- 172.21.15.73:0/2179072739 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f04240f6510 0x7f0424106730 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:17:28.728 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:28.725+0000 7f042c4c8700 1 -- 172.21.15.73:0/2179072739 >> 172.21.15.73:0/2179072739 conn(0x7f04240f1c70 msgr2=0x7f04240f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:17:28.728 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:28.725+0000 7f042c4c8700 1 -- 172.21.15.73:0/2179072739 shutdown_connections 2023-12-01T22:17:28.728 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:28.725+0000 7f042c4c8700 1 -- 172.21.15.73:0/2179072739 wait complete. 2023-12-01T22:17:28.728 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:17:29.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:17:28 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:17:28.724939+0000 mon.smithi029 (mon.0) 263 : audit [DBG] from='client.? 172.21.15.73:0/2179072739' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:17:30.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:17:29 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:17:28.678398+0000 mgr.smithi029.loxrfu (mgr.14164) 194 : cluster [DBG] pgmap v182: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:17:30.157 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:17:30.157 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:17:31.349 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:31.347+0000 7f566158c700 1 -- 172.21.15.73:0/3144080806 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f565c0f51a0 msgr2=0x7f565c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:17:31.349 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:31.347+0000 7f566158c700 1 --2- 172.21.15.73:0/3144080806 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f565c0f51a0 0x7f565c0f55c0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 rx=0x7f564c009fb0 tx=0x7f564c009b30).stop 2023-12-01T22:17:31.349 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:31.347+0000 7f566158c700 1 -- 172.21.15.73:0/3144080806 shutdown_connections 2023-12-01T22:17:31.349 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:31.347+0000 7f566158c700 1 --2- 172.21.15.73:0/3144080806 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f565c0f51a0 0x7f565c0f55c0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:17:31.349 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:31.347+0000 7f566158c700 1 -- 172.21.15.73:0/3144080806 >> 172.21.15.73:0/3144080806 conn(0x7f565c0f09a0 msgr2=0x7f565c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:17:31.350 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:31.347+0000 7f566158c700 1 -- 172.21.15.73:0/3144080806 shutdown_connections 2023-12-01T22:17:31.350 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:31.347+0000 7f566158c700 1 -- 172.21.15.73:0/3144080806 wait complete. 2023-12-01T22:17:31.350 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:31.347+0000 7f566158c700 1 Processor -- start 2023-12-01T22:17:31.350 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:31.348+0000 7f566158c700 1 -- start start 2023-12-01T22:17:31.351 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:31.348+0000 7f566158c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f565c0f51a0 0x7f565c1091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:17:31.352 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:31.348+0000 7f566158c700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f565c060240 con 0x7f565c0f51a0 2023-12-01T22:17:31.352 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:31.348+0000 7f565bfff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f565c0f51a0 0x7f565c1091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:17:31.352 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:31.348+0000 7f565bfff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f565c0f51a0 0x7f565c1091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:40240/0 (socket says 172.21.15.73:40240) 2023-12-01T22:17:31.352 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:31.348+0000 7f565bfff700 1 -- 172.21.15.73:0/1761021441 learned_addr learned my addr 172.21.15.73:0/1761021441 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:17:31.352 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:31.349+0000 7f565bfff700 1 -- 172.21.15.73:0/1761021441 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f564c005040 con 0x7f565c0f51a0 2023-12-01T22:17:31.352 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:31.349+0000 7f565bfff700 1 --2- 172.21.15.73:0/1761021441 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f565c0f51a0 0x7f565c1091b0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7f564c013510 tx=0x7f564c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:17:31.352 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:31.349+0000 7f5658ff9700 1 -- 172.21.15.73:0/1761021441 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f564c009320 con 0x7f565c0f51a0 2023-12-01T22:17:31.352 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:31.349+0000 7f566158c700 1 -- 172.21.15.73:0/1761021441 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f565c060440 con 0x7f565c0f51a0 2023-12-01T22:17:31.352 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:31.349+0000 7f5658ff9700 1 -- 172.21.15.73:0/1761021441 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f564c013f00 con 0x7f565c0f51a0 2023-12-01T22:17:31.352 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:31.349+0000 7f5658ff9700 1 -- 172.21.15.73:0/1761021441 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f564c014dc0 con 0x7f565c0f51a0 2023-12-01T22:17:31.352 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:31.349+0000 7f566158c700 1 -- 172.21.15.73:0/1761021441 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f565c109860 con 0x7f565c0f51a0 2023-12-01T22:17:31.352 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:31.350+0000 7f5658ff9700 1 -- 172.21.15.73:0/1761021441 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f564c01d3d0 con 0x7f565c0f51a0 2023-12-01T22:17:31.352 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:31.350+0000 7f5658ff9700 1 --2- 172.21.15.73:0/1761021441 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f564402ff60 0x7f5644032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:17:31.353 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:31.350+0000 7f5658ff9700 1 -- 172.21.15.73:0/1761021441 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f564c041050 con 0x7f565c0f51a0 2023-12-01T22:17:31.353 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:31.350+0000 7f566158c700 1 -- 172.21.15.73:0/1761021441 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f565c045ad0 con 0x7f565c0f51a0 2023-12-01T22:17:31.353 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:31.350+0000 7f565b7fe700 1 --2- 172.21.15.73:0/1761021441 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f564402ff60 0x7f5644032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:17:31.353 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:31.350+0000 7f565b7fe700 1 --2- 172.21.15.73:0/1761021441 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f564402ff60 0x7f5644032420 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f56500096a0 tx=0x7f5650006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:17:31.356 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:31.355+0000 7f5658ff9700 1 -- 172.21.15.73:0/1761021441 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f564c012920 con 0x7f565c0f51a0 2023-12-01T22:17:31.562 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:31.561+0000 7f566158c700 1 -- 172.21.15.73:0/1761021441 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f565c109fd0 con 0x7f565c0f51a0 2023-12-01T22:17:31.563 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:31.561+0000 7f5658ff9700 1 -- 172.21.15.73:0/1761021441 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f564c02b020 con 0x7f565c0f51a0 2023-12-01T22:17:31.563 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:17:31.563 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:17:31.565 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:31.563+0000 7f566158c700 1 -- 172.21.15.73:0/1761021441 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f564402ff60 msgr2=0x7f5644032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:17:31.565 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:31.563+0000 7f566158c700 1 --2- 172.21.15.73:0/1761021441 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f564402ff60 0x7f5644032420 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f56500096a0 tx=0x7f5650006b40).stop 2023-12-01T22:17:31.565 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:31.563+0000 7f566158c700 1 -- 172.21.15.73:0/1761021441 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f565c0f51a0 msgr2=0x7f565c1091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:17:31.565 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:31.563+0000 7f566158c700 1 --2- 172.21.15.73:0/1761021441 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f565c0f51a0 0x7f565c1091b0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7f564c013510 tx=0x7f564c00b040).stop 2023-12-01T22:17:31.565 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:31.563+0000 7f566158c700 1 -- 172.21.15.73:0/1761021441 shutdown_connections 2023-12-01T22:17:31.565 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:31.563+0000 7f566158c700 1 --2- 172.21.15.73:0/1761021441 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f564402ff60 0x7f5644032420 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:17:31.565 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:31.563+0000 7f566158c700 1 --2- 172.21.15.73:0/1761021441 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f565c0f51a0 0x7f565c1091b0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:17:31.566 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:31.563+0000 7f566158c700 1 -- 172.21.15.73:0/1761021441 >> 172.21.15.73:0/1761021441 conn(0x7f565c0f09a0 msgr2=0x7f565c0f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:17:31.566 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:31.563+0000 7f566158c700 1 -- 172.21.15.73:0/1761021441 shutdown_connections 2023-12-01T22:17:31.566 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:31.563+0000 7f566158c700 1 -- 172.21.15.73:0/1761021441 wait complete. 2023-12-01T22:17:31.566 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:17:31.777 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:17:31 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:17:30.678838+0000 mgr.smithi029.loxrfu (mgr.14164) 195 : cluster [DBG] pgmap v183: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:17:32.776 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:17:32 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:17:31.562749+0000 mon.smithi029 (mon.0) 264 : audit [DBG] from='client.? 172.21.15.73:0/1761021441' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:17:32.981 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:17:32.981 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:17:33.777 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:17:33 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:17:32.679256+0000 mgr.smithi029.loxrfu (mgr.14164) 196 : cluster [DBG] pgmap v184: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:17:34.139 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:34.137+0000 7f68f051a700 1 -- 172.21.15.73:0/602351032 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f68e80f5410 msgr2=0x7f68e80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:17:34.140 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:34.137+0000 7f68f051a700 1 --2- 172.21.15.73:0/602351032 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f68e80f5410 0x7f68e80f5830 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 rx=0x7f68d8009fb0 tx=0x7f68d8009b30).stop 2023-12-01T22:17:34.140 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:34.137+0000 7f68f051a700 1 -- 172.21.15.73:0/602351032 shutdown_connections 2023-12-01T22:17:34.140 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:34.137+0000 7f68f051a700 1 --2- 172.21.15.73:0/602351032 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f68e80f5410 0x7f68e80f5830 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:17:34.140 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:34.137+0000 7f68f051a700 1 -- 172.21.15.73:0/602351032 >> 172.21.15.73:0/602351032 conn(0x7f68e80f0b70 msgr2=0x7f68e80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:17:34.140 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:34.137+0000 7f68f051a700 1 -- 172.21.15.73:0/602351032 shutdown_connections 2023-12-01T22:17:34.140 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:34.137+0000 7f68f051a700 1 -- 172.21.15.73:0/602351032 wait complete. 2023-12-01T22:17:34.140 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:34.138+0000 7f68f051a700 1 Processor -- start 2023-12-01T22:17:34.140 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:34.138+0000 7f68f051a700 1 -- start start 2023-12-01T22:17:34.140 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:34.138+0000 7f68f051a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f68e80f5410 0x7f68e8106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:17:34.140 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:34.138+0000 7f68f051a700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68e8106c10 con 0x7f68e80f5410 2023-12-01T22:17:34.142 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:34.138+0000 7f68ee2b6700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f68e80f5410 0x7f68e8106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:17:34.142 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:34.138+0000 7f68ee2b6700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f68e80f5410 0x7f68e8106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:40264/0 (socket says 172.21.15.73:40264) 2023-12-01T22:17:34.142 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:34.138+0000 7f68ee2b6700 1 -- 172.21.15.73:0/2357207184 learned_addr learned my addr 172.21.15.73:0/2357207184 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:17:34.142 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:34.139+0000 7f68ee2b6700 1 -- 172.21.15.73:0/2357207184 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68d8005040 con 0x7f68e80f5410 2023-12-01T22:17:34.142 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:34.139+0000 7f68ee2b6700 1 --2- 172.21.15.73:0/2357207184 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f68e80f5410 0x7f68e8106720 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7f68d8013510 tx=0x7f68d800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:17:34.142 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:34.139+0000 7f68deffd700 1 -- 172.21.15.73:0/2357207184 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f68d8009320 con 0x7f68e80f5410 2023-12-01T22:17:34.142 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:34.139+0000 7f68f051a700 1 -- 172.21.15.73:0/2357207184 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68e8106e10 con 0x7f68e80f5410 2023-12-01T22:17:34.142 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:34.139+0000 7f68deffd700 1 -- 172.21.15.73:0/2357207184 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f68d8013f00 con 0x7f68e80f5410 2023-12-01T22:17:34.142 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:34.139+0000 7f68f051a700 1 -- 172.21.15.73:0/2357207184 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68e81072a0 con 0x7f68e80f5410 2023-12-01T22:17:34.142 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:34.139+0000 7f68deffd700 1 -- 172.21.15.73:0/2357207184 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f68d8014d90 con 0x7f68e80f5410 2023-12-01T22:17:34.143 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:34.140+0000 7f68deffd700 1 -- 172.21.15.73:0/2357207184 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f68d80095f0 con 0x7f68e80f5410 2023-12-01T22:17:34.143 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:34.140+0000 7f68f051a700 1 -- 172.21.15.73:0/2357207184 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68e8045ad0 con 0x7f68e80f5410 2023-12-01T22:17:34.143 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:34.140+0000 7f68deffd700 1 --2- 172.21.15.73:0/2357207184 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f68d402ff60 0x7f68d4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:17:34.143 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:34.140+0000 7f68deffd700 1 -- 172.21.15.73:0/2357207184 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f68d8041090 con 0x7f68e80f5410 2023-12-01T22:17:34.143 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:34.140+0000 7f68edab5700 1 --2- 172.21.15.73:0/2357207184 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f68d402ff60 0x7f68d4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:17:34.143 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:34.140+0000 7f68edab5700 1 --2- 172.21.15.73:0/2357207184 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f68d402ff60 0x7f68d4032420 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f68e40096a0 tx=0x7f68e4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:17:34.146 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:34.145+0000 7f68deffd700 1 -- 172.21.15.73:0/2357207184 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f68d8014420 con 0x7f68e80f5410 2023-12-01T22:17:34.356 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:34.354+0000 7f68f051a700 1 -- 172.21.15.73:0/2357207184 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f68e81080f0 con 0x7f68e80f5410 2023-12-01T22:17:34.357 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:34.356+0000 7f68deffd700 1 -- 172.21.15.73:0/2357207184 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f68d8010230 con 0x7f68e80f5410 2023-12-01T22:17:34.357 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:17:34.357 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:17:34.358 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:34.356+0000 7f68f051a700 1 -- 172.21.15.73:0/2357207184 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f68d402ff60 msgr2=0x7f68d4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:17:34.358 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:34.357+0000 7f68f051a700 1 --2- 172.21.15.73:0/2357207184 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f68d402ff60 0x7f68d4032420 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f68e40096a0 tx=0x7f68e4006b40).stop 2023-12-01T22:17:34.358 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:34.357+0000 7f68f051a700 1 -- 172.21.15.73:0/2357207184 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f68e80f5410 msgr2=0x7f68e8106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:17:34.359 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:34.357+0000 7f68f051a700 1 --2- 172.21.15.73:0/2357207184 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f68e80f5410 0x7f68e8106720 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7f68d8013510 tx=0x7f68d800b040).stop 2023-12-01T22:17:34.359 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:34.357+0000 7f68f051a700 1 -- 172.21.15.73:0/2357207184 shutdown_connections 2023-12-01T22:17:34.359 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:34.357+0000 7f68f051a700 1 --2- 172.21.15.73:0/2357207184 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f68d402ff60 0x7f68d4032420 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:17:34.359 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:34.357+0000 7f68f051a700 1 --2- 172.21.15.73:0/2357207184 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f68e80f5410 0x7f68e8106720 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:17:34.359 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:34.357+0000 7f68f051a700 1 -- 172.21.15.73:0/2357207184 >> 172.21.15.73:0/2357207184 conn(0x7f68e80f0b70 msgr2=0x7f68e805fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:17:34.359 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:34.357+0000 7f68f051a700 1 -- 172.21.15.73:0/2357207184 shutdown_connections 2023-12-01T22:17:34.359 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:34.357+0000 7f68f051a700 1 -- 172.21.15.73:0/2357207184 wait complete. 2023-12-01T22:17:34.359 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:17:34.776 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:17:34 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:17:34.356383+0000 mon.smithi029 (mon.0) 265 : audit [DBG] from='client.? 172.21.15.73:0/2357207184' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:17:35.752 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:17:35.752 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:17:35.776 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:17:35 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:17:34.679552+0000 mgr.smithi029.loxrfu (mgr.14164) 197 : cluster [DBG] pgmap v185: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:17:36.891 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:36.888+0000 7f883b048700 1 -- 172.21.15.73:0/3217936473 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f88340f5410 msgr2=0x7f88340f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:17:36.891 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:36.888+0000 7f883b048700 1 --2- 172.21.15.73:0/3217936473 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f88340f5410 0x7f88340f5830 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 rx=0x7f8828009fb0 tx=0x7f8828009b30).stop 2023-12-01T22:17:36.891 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:36.888+0000 7f883b048700 1 -- 172.21.15.73:0/3217936473 shutdown_connections 2023-12-01T22:17:36.891 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:36.888+0000 7f883b048700 1 --2- 172.21.15.73:0/3217936473 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f88340f5410 0x7f88340f5830 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:17:36.891 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:36.888+0000 7f883b048700 1 -- 172.21.15.73:0/3217936473 >> 172.21.15.73:0/3217936473 conn(0x7f88340f0b70 msgr2=0x7f88340f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:17:36.892 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:36.888+0000 7f883b048700 1 -- 172.21.15.73:0/3217936473 shutdown_connections 2023-12-01T22:17:36.892 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:36.888+0000 7f883b048700 1 -- 172.21.15.73:0/3217936473 wait complete. 2023-12-01T22:17:36.892 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:36.888+0000 7f883b048700 1 Processor -- start 2023-12-01T22:17:36.892 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:36.888+0000 7f883b048700 1 -- start start 2023-12-01T22:17:36.892 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:36.888+0000 7f883b048700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f88340f5410 0x7f8834106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:17:36.892 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:36.888+0000 7f883b048700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8834106c20 con 0x7f88340f5410 2023-12-01T22:17:36.892 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:36.889+0000 7f8838de4700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f88340f5410 0x7f8834106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:17:36.892 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:36.889+0000 7f8838de4700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f88340f5410 0x7f8834106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:43888/0 (socket says 172.21.15.73:43888) 2023-12-01T22:17:36.892 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:36.889+0000 7f8838de4700 1 -- 172.21.15.73:0/847458651 learned_addr learned my addr 172.21.15.73:0/847458651 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:17:36.892 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:36.889+0000 7f8838de4700 1 -- 172.21.15.73:0/847458651 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8828005040 con 0x7f88340f5410 2023-12-01T22:17:36.892 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:36.889+0000 7f8838de4700 1 --2- 172.21.15.73:0/847458651 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f88340f5410 0x7f8834106730 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7f8828013510 tx=0x7f882800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:17:36.892 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:36.889+0000 7f88317fa700 1 -- 172.21.15.73:0/847458651 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8828009320 con 0x7f88340f5410 2023-12-01T22:17:36.892 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:36.889+0000 7f883b048700 1 -- 172.21.15.73:0/847458651 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8834106e20 con 0x7f88340f5410 2023-12-01T22:17:36.892 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:36.889+0000 7f88317fa700 1 -- 172.21.15.73:0/847458651 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8828013f00 con 0x7f88340f5410 2023-12-01T22:17:36.893 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:36.889+0000 7f88317fa700 1 -- 172.21.15.73:0/847458651 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8828014d90 con 0x7f88340f5410 2023-12-01T22:17:36.893 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:36.889+0000 7f883b048700 1 -- 172.21.15.73:0/847458651 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88341072b0 con 0x7f88340f5410 2023-12-01T22:17:36.893 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:36.890+0000 7f88317fa700 1 -- 172.21.15.73:0/847458651 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8828014420 con 0x7f88340f5410 2023-12-01T22:17:36.894 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:36.890+0000 7f883b048700 1 -- 172.21.15.73:0/847458651 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8834045ad0 con 0x7f88340f5410 2023-12-01T22:17:36.894 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:36.890+0000 7f88317fa700 1 --2- 172.21.15.73:0/847458651 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f882402ff60 0x7f8824032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:17:36.894 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:36.890+0000 7f88317fa700 1 -- 172.21.15.73:0/847458651 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8828040ff0 con 0x7f88340f5410 2023-12-01T22:17:36.894 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:36.890+0000 7f8833fff700 1 --2- 172.21.15.73:0/847458651 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f882402ff60 0x7f8824032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:17:36.894 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:36.891+0000 7f8833fff700 1 --2- 172.21.15.73:0/847458651 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f882402ff60 0x7f8824032420 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f88200096a0 tx=0x7f8820006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:17:36.896 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:36.895+0000 7f88317fa700 1 -- 172.21.15.73:0/847458651 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f88280095f0 con 0x7f88340f5410 2023-12-01T22:17:37.107 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:37.105+0000 7f883b048700 1 -- 172.21.15.73:0/847458651 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f88340fbf90 con 0x7f88340f5410 2023-12-01T22:17:37.107 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:37.105+0000 7f88317fa700 1 -- 172.21.15.73:0/847458651 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f882801db70 con 0x7f88340f5410 2023-12-01T22:17:37.107 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:17:37.107 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:17:37.109 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:37.107+0000 7f883b048700 1 -- 172.21.15.73:0/847458651 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f882402ff60 msgr2=0x7f8824032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:17:37.109 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:37.107+0000 7f883b048700 1 --2- 172.21.15.73:0/847458651 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f882402ff60 0x7f8824032420 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f88200096a0 tx=0x7f8820006b40).stop 2023-12-01T22:17:37.109 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:37.107+0000 7f883b048700 1 -- 172.21.15.73:0/847458651 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f88340f5410 msgr2=0x7f8834106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:17:37.109 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:37.107+0000 7f883b048700 1 --2- 172.21.15.73:0/847458651 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f88340f5410 0x7f8834106730 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7f8828013510 tx=0x7f882800b040).stop 2023-12-01T22:17:37.109 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:37.107+0000 7f883b048700 1 -- 172.21.15.73:0/847458651 shutdown_connections 2023-12-01T22:17:37.109 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:37.107+0000 7f883b048700 1 --2- 172.21.15.73:0/847458651 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f882402ff60 0x7f8824032420 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:17:37.109 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:37.107+0000 7f883b048700 1 --2- 172.21.15.73:0/847458651 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f88340f5410 0x7f8834106730 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:17:37.109 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:37.107+0000 7f883b048700 1 -- 172.21.15.73:0/847458651 >> 172.21.15.73:0/847458651 conn(0x7f88340f0b70 msgr2=0x7f88340f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:17:37.109 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:37.107+0000 7f883b048700 1 -- 172.21.15.73:0/847458651 shutdown_connections 2023-12-01T22:17:37.109 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:37.107+0000 7f883b048700 1 -- 172.21.15.73:0/847458651 wait complete. 2023-12-01T22:17:37.110 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:17:38.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:17:37 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:17:36.679996+0000 mgr.smithi029.loxrfu (mgr.14164) 198 : cluster [DBG] pgmap v186: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:17:38.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:17:37 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:17:37.106868+0000 mon.smithi029 (mon.0) 266 : audit [DBG] from='client.? 172.21.15.73:0/847458651' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:17:38.497 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:17:38.497 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:17:39.655 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:39.653+0000 7fe41943c700 1 -- 172.21.15.73:0/2790688110 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe414060280 msgr2=0x7fe4140f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:17:39.655 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:39.653+0000 7fe41943c700 1 --2- 172.21.15.73:0/2790688110 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe414060280 0x7fe4140f90f0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 rx=0x7fe408009fb0 tx=0x7fe408009b30).stop 2023-12-01T22:17:39.655 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:39.653+0000 7fe41943c700 1 -- 172.21.15.73:0/2790688110 shutdown_connections 2023-12-01T22:17:39.655 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:39.653+0000 7fe41943c700 1 --2- 172.21.15.73:0/2790688110 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe414060280 0x7fe4140f90f0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:17:39.655 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:39.653+0000 7fe41943c700 1 -- 172.21.15.73:0/2790688110 >> 172.21.15.73:0/2790688110 conn(0x7fe4140f0b70 msgr2=0x7fe4140f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:17:39.655 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:39.653+0000 7fe41943c700 1 -- 172.21.15.73:0/2790688110 shutdown_connections 2023-12-01T22:17:39.655 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:39.653+0000 7fe41943c700 1 -- 172.21.15.73:0/2790688110 wait complete. 2023-12-01T22:17:39.656 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:39.654+0000 7fe41943c700 1 Processor -- start 2023-12-01T22:17:39.656 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:39.654+0000 7fe41943c700 1 -- start start 2023-12-01T22:17:39.656 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:39.654+0000 7fe41943c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe414060280 0x7fe4141088f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:17:39.656 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:39.654+0000 7fe41943c700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe414108de0 con 0x7fe414060280 2023-12-01T22:17:39.656 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:39.654+0000 7fe412ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe414060280 0x7fe4141088f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:17:39.656 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:39.654+0000 7fe412ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe414060280 0x7fe4141088f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:43912/0 (socket says 172.21.15.73:43912) 2023-12-01T22:17:39.656 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:39.654+0000 7fe412ffd700 1 -- 172.21.15.73:0/2605243571 learned_addr learned my addr 172.21.15.73:0/2605243571 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:17:39.657 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:39.655+0000 7fe412ffd700 1 -- 172.21.15.73:0/2605243571 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe408005040 con 0x7fe414060280 2023-12-01T22:17:39.657 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:39.655+0000 7fe412ffd700 1 --2- 172.21.15.73:0/2605243571 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe414060280 0x7fe4141088f0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7fe408013510 tx=0x7fe40800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:17:39.657 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:39.655+0000 7fe3fb7fe700 1 -- 172.21.15.73:0/2605243571 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe408009320 con 0x7fe414060280 2023-12-01T22:17:39.657 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:39.655+0000 7fe41943c700 1 -- 172.21.15.73:0/2605243571 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe414108fe0 con 0x7fe414060280 2023-12-01T22:17:39.657 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:39.655+0000 7fe3fb7fe700 1 -- 172.21.15.73:0/2605243571 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe408013f00 con 0x7fe414060280 2023-12-01T22:17:39.658 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:39.655+0000 7fe3fb7fe700 1 -- 172.21.15.73:0/2605243571 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe408014dc0 con 0x7fe414060280 2023-12-01T22:17:39.658 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:39.655+0000 7fe41943c700 1 -- 172.21.15.73:0/2605243571 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe414109470 con 0x7fe414060280 2023-12-01T22:17:39.658 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:39.656+0000 7fe3fb7fe700 1 -- 172.21.15.73:0/2605243571 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe40801d3d0 con 0x7fe414060280 2023-12-01T22:17:39.659 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:39.656+0000 7fe3fb7fe700 1 --2- 172.21.15.73:0/2605243571 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fe3fc02ff60 0x7fe3fc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:17:39.659 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:39.656+0000 7fe3fb7fe700 1 -- 172.21.15.73:0/2605243571 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe408041f10 con 0x7fe414060280 2023-12-01T22:17:39.659 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:39.656+0000 7fe4127fc700 1 --2- 172.21.15.73:0/2605243571 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fe3fc02ff60 0x7fe3fc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:17:39.659 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:39.657+0000 7fe41943c700 1 -- 172.21.15.73:0/2605243571 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe414045ad0 con 0x7fe414060280 2023-12-01T22:17:39.659 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:39.657+0000 7fe4127fc700 1 --2- 172.21.15.73:0/2605243571 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fe3fc02ff60 0x7fe3fc032420 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7fe4040096a0 tx=0x7fe404006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:17:39.663 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:39.662+0000 7fe3fb7fe700 1 -- 172.21.15.73:0/2605243571 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe408026490 con 0x7fe414060280 2023-12-01T22:17:39.870 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:39.868+0000 7fe41943c700 1 -- 172.21.15.73:0/2605243571 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe41410a140 con 0x7fe414060280 2023-12-01T22:17:39.870 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:39.869+0000 7fe3fb7fe700 1 -- 172.21.15.73:0/2605243571 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe4080126f0 con 0x7fe414060280 2023-12-01T22:17:39.870 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:17:39.870 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:17:39.872 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:39.870+0000 7fe41943c700 1 -- 172.21.15.73:0/2605243571 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fe3fc02ff60 msgr2=0x7fe3fc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:17:39.872 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:39.870+0000 7fe41943c700 1 --2- 172.21.15.73:0/2605243571 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fe3fc02ff60 0x7fe3fc032420 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7fe4040096a0 tx=0x7fe404006b40).stop 2023-12-01T22:17:39.872 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:39.870+0000 7fe41943c700 1 -- 172.21.15.73:0/2605243571 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe414060280 msgr2=0x7fe4141088f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:17:39.872 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:39.870+0000 7fe41943c700 1 --2- 172.21.15.73:0/2605243571 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe414060280 0x7fe4141088f0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7fe408013510 tx=0x7fe40800b040).stop 2023-12-01T22:17:39.872 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:39.870+0000 7fe41943c700 1 -- 172.21.15.73:0/2605243571 shutdown_connections 2023-12-01T22:17:39.872 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:39.870+0000 7fe41943c700 1 --2- 172.21.15.73:0/2605243571 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fe3fc02ff60 0x7fe3fc032420 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:17:39.872 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:39.870+0000 7fe41943c700 1 --2- 172.21.15.73:0/2605243571 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe414060280 0x7fe4141088f0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:17:39.873 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:39.870+0000 7fe41943c700 1 -- 172.21.15.73:0/2605243571 >> 172.21.15.73:0/2605243571 conn(0x7fe4140f0b70 msgr2=0x7fe4140f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:17:39.873 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:39.870+0000 7fe41943c700 1 -- 172.21.15.73:0/2605243571 shutdown_connections 2023-12-01T22:17:39.873 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:39.870+0000 7fe41943c700 1 -- 172.21.15.73:0/2605243571 wait complete. 2023-12-01T22:17:39.873 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:17:40.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:17:39 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:17:38.680329+0000 mgr.smithi029.loxrfu (mgr.14164) 199 : cluster [DBG] pgmap v187: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:17:41.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:17:40 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:17:39.870209+0000 mon.smithi029 (mon.0) 267 : audit [DBG] from='client.? 172.21.15.73:0/2605243571' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:17:41.276 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:17:41.276 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:17:42.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:17:41 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:17:40.680659+0000 mgr.smithi029.loxrfu (mgr.14164) 200 : cluster [DBG] pgmap v188: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:17:42.473 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:42.470+0000 7f92b285b700 1 -- 172.21.15.73:0/813428984 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92ac0f6480 msgr2=0x7f92ac0f68a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:17:42.473 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:42.470+0000 7f92b285b700 1 --2- 172.21.15.73:0/813428984 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92ac0f6480 0x7f92ac0f68a0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 rx=0x7f92a0009fb0 tx=0x7f92a0009b30).stop 2023-12-01T22:17:42.473 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:42.470+0000 7f92b285b700 1 -- 172.21.15.73:0/813428984 shutdown_connections 2023-12-01T22:17:42.473 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:42.470+0000 7f92b285b700 1 --2- 172.21.15.73:0/813428984 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92ac0f6480 0x7f92ac0f68a0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:17:42.473 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:42.470+0000 7f92b285b700 1 -- 172.21.15.73:0/813428984 >> 172.21.15.73:0/813428984 conn(0x7f92ac0f1c00 msgr2=0x7f92ac0f4020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:17:42.473 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:42.470+0000 7f92b285b700 1 -- 172.21.15.73:0/813428984 shutdown_connections 2023-12-01T22:17:42.473 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:42.470+0000 7f92b285b700 1 -- 172.21.15.73:0/813428984 wait complete. 2023-12-01T22:17:42.474 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:42.471+0000 7f92b285b700 1 Processor -- start 2023-12-01T22:17:42.474 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:42.471+0000 7f92b285b700 1 -- start start 2023-12-01T22:17:42.474 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:42.471+0000 7f92b285b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92ac0f6480 0x7f92ac1066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:17:42.474 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:42.471+0000 7f92b285b700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92ac106bc0 con 0x7f92ac0f6480 2023-12-01T22:17:42.474 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:42.471+0000 7f92abfff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92ac0f6480 0x7f92ac1066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:17:42.474 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:42.471+0000 7f92abfff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92ac0f6480 0x7f92ac1066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:43934/0 (socket says 172.21.15.73:43934) 2023-12-01T22:17:42.474 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:42.471+0000 7f92abfff700 1 -- 172.21.15.73:0/1680391365 learned_addr learned my addr 172.21.15.73:0/1680391365 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:17:42.474 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:42.471+0000 7f92abfff700 1 -- 172.21.15.73:0/1680391365 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92a0005040 con 0x7f92ac0f6480 2023-12-01T22:17:42.474 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:42.472+0000 7f92abfff700 1 --2- 172.21.15.73:0/1680391365 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92ac0f6480 0x7f92ac1066d0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7f92a0013510 tx=0x7f92a000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:17:42.475 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:42.472+0000 7f92a8ff9700 1 -- 172.21.15.73:0/1680391365 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f92a0009320 con 0x7f92ac0f6480 2023-12-01T22:17:42.475 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:42.472+0000 7f92b285b700 1 -- 172.21.15.73:0/1680391365 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92ac106dc0 con 0x7f92ac0f6480 2023-12-01T22:17:42.475 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:42.472+0000 7f92a8ff9700 1 -- 172.21.15.73:0/1680391365 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f92a0013f00 con 0x7f92ac0f6480 2023-12-01T22:17:42.475 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:42.472+0000 7f92a8ff9700 1 -- 172.21.15.73:0/1680391365 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f92a0014d90 con 0x7f92ac0f6480 2023-12-01T22:17:42.476 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:42.472+0000 7f92b285b700 1 -- 172.21.15.73:0/1680391365 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92ac107250 con 0x7f92ac0f6480 2023-12-01T22:17:42.476 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:42.473+0000 7f92a8ff9700 1 -- 172.21.15.73:0/1680391365 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f92a00095f0 con 0x7f92ac0f6480 2023-12-01T22:17:42.476 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:42.473+0000 7f92b285b700 1 -- 172.21.15.73:0/1680391365 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92ac045ad0 con 0x7f92ac0f6480 2023-12-01T22:17:42.476 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:42.473+0000 7f92a8ff9700 1 --2- 172.21.15.73:0/1680391365 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f929402ff60 0x7f9294032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:17:42.476 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:42.473+0000 7f92a8ff9700 1 -- 172.21.15.73:0/1680391365 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f92a00410a0 con 0x7f92ac0f6480 2023-12-01T22:17:42.476 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:42.473+0000 7f92ab7fe700 1 --2- 172.21.15.73:0/1680391365 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f929402ff60 0x7f9294032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:17:42.476 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:42.473+0000 7f92ab7fe700 1 --2- 172.21.15.73:0/1680391365 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f929402ff60 0x7f9294032420 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f929c0096a0 tx=0x7f929c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:17:42.479 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:42.478+0000 7f92a8ff9700 1 -- 172.21.15.73:0/1680391365 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f92a0010230 con 0x7f92ac0f6480 2023-12-01T22:17:42.683 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:42.682+0000 7f92b285b700 1 -- 172.21.15.73:0/1680391365 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f92ac1080a0 con 0x7f92ac0f6480 2023-12-01T22:17:42.684 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:42.682+0000 7f92a8ff9700 1 -- 172.21.15.73:0/1680391365 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f92a0010230 con 0x7f92ac0f6480 2023-12-01T22:17:42.685 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:17:42.685 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:17:42.686 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:42.684+0000 7f92b285b700 1 -- 172.21.15.73:0/1680391365 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f929402ff60 msgr2=0x7f9294032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:17:42.686 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:42.684+0000 7f92b285b700 1 --2- 172.21.15.73:0/1680391365 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f929402ff60 0x7f9294032420 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f929c0096a0 tx=0x7f929c006b40).stop 2023-12-01T22:17:42.686 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:42.684+0000 7f92b285b700 1 -- 172.21.15.73:0/1680391365 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92ac0f6480 msgr2=0x7f92ac1066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:17:42.686 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:42.684+0000 7f92b285b700 1 --2- 172.21.15.73:0/1680391365 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92ac0f6480 0x7f92ac1066d0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7f92a0013510 tx=0x7f92a000b040).stop 2023-12-01T22:17:42.686 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:42.684+0000 7f92b285b700 1 -- 172.21.15.73:0/1680391365 shutdown_connections 2023-12-01T22:17:42.686 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:42.684+0000 7f92b285b700 1 --2- 172.21.15.73:0/1680391365 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f929402ff60 0x7f9294032420 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:17:42.686 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:42.684+0000 7f92b285b700 1 --2- 172.21.15.73:0/1680391365 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92ac0f6480 0x7f92ac1066d0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:17:42.687 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:42.684+0000 7f92b285b700 1 -- 172.21.15.73:0/1680391365 >> 172.21.15.73:0/1680391365 conn(0x7f92ac0f1c00 msgr2=0x7f92ac0f2720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:17:42.687 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:42.684+0000 7f92b285b700 1 -- 172.21.15.73:0/1680391365 shutdown_connections 2023-12-01T22:17:42.687 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:42.684+0000 7f92b285b700 1 -- 172.21.15.73:0/1680391365 wait complete. 2023-12-01T22:17:42.687 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:17:43.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:17:42 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:17:42.683802+0000 mon.smithi029 (mon.0) 268 : audit [DBG] from='client.? 172.21.15.73:0/1680391365' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:17:44.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:17:43 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:17:42.681009+0000 mgr.smithi029.loxrfu (mgr.14164) 201 : cluster [DBG] pgmap v189: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:17:44.064 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:17:44.064 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:17:45.191 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:45.189+0000 7f2162ccb700 1 -- 172.21.15.73:0/1150163593 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f215c0f6510 msgr2=0x7f215c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:17:45.191 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:45.189+0000 7f2162ccb700 1 --2- 172.21.15.73:0/1150163593 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f215c0f6510 0x7f215c0f6930 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 rx=0x7f2150009fb0 tx=0x7f2150009b30).stop 2023-12-01T22:17:45.191 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:45.189+0000 7f2162ccb700 1 -- 172.21.15.73:0/1150163593 shutdown_connections 2023-12-01T22:17:45.191 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:45.189+0000 7f2162ccb700 1 --2- 172.21.15.73:0/1150163593 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f215c0f6510 0x7f215c0f6930 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:17:45.191 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:45.189+0000 7f2162ccb700 1 -- 172.21.15.73:0/1150163593 >> 172.21.15.73:0/1150163593 conn(0x7f215c0f1c70 msgr2=0x7f215c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:17:45.191 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:45.189+0000 7f2162ccb700 1 -- 172.21.15.73:0/1150163593 shutdown_connections 2023-12-01T22:17:45.191 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:45.189+0000 7f2162ccb700 1 -- 172.21.15.73:0/1150163593 wait complete. 2023-12-01T22:17:45.193 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:45.190+0000 7f2162ccb700 1 Processor -- start 2023-12-01T22:17:45.193 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:45.190+0000 7f2162ccb700 1 -- start start 2023-12-01T22:17:45.193 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:45.190+0000 7f2162ccb700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f215c0f6510 0x7f215c106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:17:45.193 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:45.190+0000 7f2162ccb700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f215c106c20 con 0x7f215c0f6510 2023-12-01T22:17:45.193 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:45.190+0000 7f2160a67700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f215c0f6510 0x7f215c106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:17:45.193 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:45.190+0000 7f2160a67700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f215c0f6510 0x7f215c106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:35118/0 (socket says 172.21.15.73:35118) 2023-12-01T22:17:45.193 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:45.190+0000 7f2160a67700 1 -- 172.21.15.73:0/3098588856 learned_addr learned my addr 172.21.15.73:0/3098588856 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:17:45.193 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:45.191+0000 7f2160a67700 1 -- 172.21.15.73:0/3098588856 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2150005040 con 0x7f215c0f6510 2023-12-01T22:17:45.193 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:45.191+0000 7f2160a67700 1 --2- 172.21.15.73:0/3098588856 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f215c0f6510 0x7f215c106730 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7f2150013510 tx=0x7f215000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:17:45.193 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:45.191+0000 7f21597fa700 1 -- 172.21.15.73:0/3098588856 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2150009320 con 0x7f215c0f6510 2023-12-01T22:17:45.193 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:45.191+0000 7f2162ccb700 1 -- 172.21.15.73:0/3098588856 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f215c106e20 con 0x7f215c0f6510 2023-12-01T22:17:45.193 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:45.191+0000 7f21597fa700 1 -- 172.21.15.73:0/3098588856 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2150013f00 con 0x7f215c0f6510 2023-12-01T22:17:45.194 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:45.191+0000 7f21597fa700 1 -- 172.21.15.73:0/3098588856 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2150014d90 con 0x7f215c0f6510 2023-12-01T22:17:45.194 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:45.191+0000 7f2162ccb700 1 -- 172.21.15.73:0/3098588856 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f215c1072b0 con 0x7f215c0f6510 2023-12-01T22:17:45.194 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:45.192+0000 7f21597fa700 1 -- 172.21.15.73:0/3098588856 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f21500263a0 con 0x7f215c0f6510 2023-12-01T22:17:45.195 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:45.192+0000 7f2162ccb700 1 -- 172.21.15.73:0/3098588856 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f215c045ad0 con 0x7f215c0f6510 2023-12-01T22:17:45.195 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:45.192+0000 7f21597fa700 1 --2- 172.21.15.73:0/3098588856 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f214c02ff60 0x7f214c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:17:45.195 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:45.192+0000 7f21597fa700 1 -- 172.21.15.73:0/3098588856 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f21500412d0 con 0x7f215c0f6510 2023-12-01T22:17:45.195 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:45.192+0000 7f215bfff700 1 --2- 172.21.15.73:0/3098588856 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f214c02ff60 0x7f214c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:17:45.195 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:45.193+0000 7f215bfff700 1 --2- 172.21.15.73:0/3098588856 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f214c02ff60 0x7f214c032420 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f21480096a0 tx=0x7f2148006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:17:45.198 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:45.197+0000 7f21597fa700 1 -- 172.21.15.73:0/3098588856 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f215001cb10 con 0x7f215c0f6510 2023-12-01T22:17:45.408 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:45.406+0000 7f2162ccb700 1 -- 172.21.15.73:0/3098588856 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f215c0fbf60 con 0x7f215c0f6510 2023-12-01T22:17:45.409 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:45.407+0000 7f21597fa700 1 -- 172.21.15.73:0/3098588856 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f215001cb10 con 0x7f215c0f6510 2023-12-01T22:17:45.409 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:17:45.409 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:17:45.410 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:45.409+0000 7f2162ccb700 1 -- 172.21.15.73:0/3098588856 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f214c02ff60 msgr2=0x7f214c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:17:45.411 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:45.409+0000 7f2162ccb700 1 --2- 172.21.15.73:0/3098588856 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f214c02ff60 0x7f214c032420 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f21480096a0 tx=0x7f2148006b40).stop 2023-12-01T22:17:45.411 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:45.409+0000 7f2162ccb700 1 -- 172.21.15.73:0/3098588856 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f215c0f6510 msgr2=0x7f215c106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:17:45.411 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:45.409+0000 7f2162ccb700 1 --2- 172.21.15.73:0/3098588856 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f215c0f6510 0x7f215c106730 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7f2150013510 tx=0x7f215000b040).stop 2023-12-01T22:17:45.411 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:45.409+0000 7f2162ccb700 1 -- 172.21.15.73:0/3098588856 shutdown_connections 2023-12-01T22:17:45.411 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:45.409+0000 7f2162ccb700 1 --2- 172.21.15.73:0/3098588856 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f214c02ff60 0x7f214c032420 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:17:45.411 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:45.409+0000 7f2162ccb700 1 --2- 172.21.15.73:0/3098588856 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f215c0f6510 0x7f215c106730 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:17:45.411 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:45.409+0000 7f2162ccb700 1 -- 172.21.15.73:0/3098588856 >> 172.21.15.73:0/3098588856 conn(0x7f215c0f1c70 msgr2=0x7f215c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:17:45.411 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:45.409+0000 7f2162ccb700 1 -- 172.21.15.73:0/3098588856 shutdown_connections 2023-12-01T22:17:45.411 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:45.409+0000 7f2162ccb700 1 -- 172.21.15.73:0/3098588856 wait complete. 2023-12-01T22:17:45.412 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:17:46.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:17:45 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:17:44.681445+0000 mgr.smithi029.loxrfu (mgr.14164) 202 : cluster [DBG] pgmap v190: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:17:46.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:17:45 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:17:45.408216+0000 mon.smithi029 (mon.0) 269 : audit [DBG] from='client.? 172.21.15.73:0/3098588856' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:17:46.790 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:17:46.790 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:17:47.908 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:47.906+0000 7fac256ea700 1 -- 172.21.15.73:0/1203166341 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac200f6510 msgr2=0x7fac200f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:17:47.908 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:47.906+0000 7fac256ea700 1 --2- 172.21.15.73:0/1203166341 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac200f6510 0x7fac200f6930 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 rx=0x7fac14009fb0 tx=0x7fac14009b30).stop 2023-12-01T22:17:47.908 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:47.906+0000 7fac256ea700 1 -- 172.21.15.73:0/1203166341 shutdown_connections 2023-12-01T22:17:47.908 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:47.906+0000 7fac256ea700 1 --2- 172.21.15.73:0/1203166341 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac200f6510 0x7fac200f6930 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:17:47.909 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:47.906+0000 7fac256ea700 1 -- 172.21.15.73:0/1203166341 >> 172.21.15.73:0/1203166341 conn(0x7fac200f1c70 msgr2=0x7fac200f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:17:47.909 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:47.906+0000 7fac256ea700 1 -- 172.21.15.73:0/1203166341 shutdown_connections 2023-12-01T22:17:47.909 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:47.906+0000 7fac256ea700 1 -- 172.21.15.73:0/1203166341 wait complete. 2023-12-01T22:17:47.911 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:47.907+0000 7fac256ea700 1 Processor -- start 2023-12-01T22:17:47.911 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:47.907+0000 7fac256ea700 1 -- start start 2023-12-01T22:17:47.911 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:47.907+0000 7fac256ea700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac200f6510 0x7fac2006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:17:47.911 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:47.907+0000 7fac256ea700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac20069e20 con 0x7fac200f6510 2023-12-01T22:17:47.911 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:47.907+0000 7fac1effd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac200f6510 0x7fac2006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:17:47.911 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:47.907+0000 7fac1effd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac200f6510 0x7fac2006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:35136/0 (socket says 172.21.15.73:35136) 2023-12-01T22:17:47.911 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:47.907+0000 7fac1effd700 1 -- 172.21.15.73:0/455035227 learned_addr learned my addr 172.21.15.73:0/455035227 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:17:47.911 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:47.907+0000 7fac1effd700 1 -- 172.21.15.73:0/455035227 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac14005040 con 0x7fac200f6510 2023-12-01T22:17:47.912 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:47.908+0000 7fac1effd700 1 --2- 172.21.15.73:0/455035227 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac200f6510 0x7fac2006b9a0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7fac14013510 tx=0x7fac1400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:17:47.912 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:47.908+0000 7fac077fe700 1 -- 172.21.15.73:0/455035227 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fac14009320 con 0x7fac200f6510 2023-12-01T22:17:47.912 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:47.908+0000 7fac256ea700 1 -- 172.21.15.73:0/455035227 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac20069fc0 con 0x7fac200f6510 2023-12-01T22:17:47.912 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:47.908+0000 7fac077fe700 1 -- 172.21.15.73:0/455035227 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fac14013f00 con 0x7fac200f6510 2023-12-01T22:17:47.912 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:47.908+0000 7fac077fe700 1 -- 172.21.15.73:0/455035227 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fac14014d90 con 0x7fac200f6510 2023-12-01T22:17:47.912 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:47.908+0000 7fac256ea700 1 -- 172.21.15.73:0/455035227 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac2006a450 con 0x7fac200f6510 2023-12-01T22:17:47.912 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:47.909+0000 7fac077fe700 1 -- 172.21.15.73:0/455035227 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fac140263a0 con 0x7fac200f6510 2023-12-01T22:17:47.912 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:47.909+0000 7fac256ea700 1 -- 172.21.15.73:0/455035227 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac20059100 con 0x7fac200f6510 2023-12-01T22:17:47.912 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:47.909+0000 7fac077fe700 1 --2- 172.21.15.73:0/455035227 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fac0802ff60 0x7fac08032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:17:47.912 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:47.909+0000 7fac077fe700 1 -- 172.21.15.73:0/455035227 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fac140413d0 con 0x7fac200f6510 2023-12-01T22:17:47.912 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:47.909+0000 7fac1e7fc700 1 --2- 172.21.15.73:0/455035227 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fac0802ff60 0x7fac08032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:17:47.912 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:47.909+0000 7fac1e7fc700 1 --2- 172.21.15.73:0/455035227 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fac0802ff60 0x7fac08032420 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7fac100096a0 tx=0x7fac10006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:17:47.915 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:47.913+0000 7fac077fe700 1 -- 172.21.15.73:0/455035227 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fac140126f0 con 0x7fac200f6510 2023-12-01T22:17:48.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:17:47 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:17:46.681845+0000 mgr.smithi029.loxrfu (mgr.14164) 203 : cluster [DBG] pgmap v191: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:17:48.126 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:48.124+0000 7fac256ea700 1 -- 172.21.15.73:0/455035227 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fac200fbf60 con 0x7fac200f6510 2023-12-01T22:17:48.127 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:48.126+0000 7fac077fe700 1 -- 172.21.15.73:0/455035227 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fac14012330 con 0x7fac200f6510 2023-12-01T22:17:48.127 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:17:48.127 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:17:48.128 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:48.127+0000 7fac256ea700 1 -- 172.21.15.73:0/455035227 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fac0802ff60 msgr2=0x7fac08032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:17:48.128 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:48.127+0000 7fac256ea700 1 --2- 172.21.15.73:0/455035227 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fac0802ff60 0x7fac08032420 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7fac100096a0 tx=0x7fac10006b40).stop 2023-12-01T22:17:48.128 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:48.127+0000 7fac256ea700 1 -- 172.21.15.73:0/455035227 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac200f6510 msgr2=0x7fac2006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:17:48.129 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:48.127+0000 7fac256ea700 1 --2- 172.21.15.73:0/455035227 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac200f6510 0x7fac2006b9a0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7fac14013510 tx=0x7fac1400b040).stop 2023-12-01T22:17:48.129 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:48.127+0000 7fac256ea700 1 -- 172.21.15.73:0/455035227 shutdown_connections 2023-12-01T22:17:48.129 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:48.127+0000 7fac256ea700 1 --2- 172.21.15.73:0/455035227 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fac0802ff60 0x7fac08032420 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:17:48.129 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:48.127+0000 7fac256ea700 1 --2- 172.21.15.73:0/455035227 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac200f6510 0x7fac2006b9a0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:17:48.129 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:48.127+0000 7fac256ea700 1 -- 172.21.15.73:0/455035227 >> 172.21.15.73:0/455035227 conn(0x7fac200f1c70 msgr2=0x7fac200f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:17:48.129 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:48.127+0000 7fac256ea700 1 -- 172.21.15.73:0/455035227 shutdown_connections 2023-12-01T22:17:48.129 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:48.127+0000 7fac256ea700 1 -- 172.21.15.73:0/455035227 wait complete. 2023-12-01T22:17:48.129 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:17:49.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:17:48 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:17:48.126322+0000 mon.smithi029 (mon.0) 270 : audit [DBG] from='client.? 172.21.15.73:0/455035227' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:17:49.507 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:17:49.507 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:17:50.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:17:49 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:17:48.682207+0000 mgr.smithi029.loxrfu (mgr.14164) 204 : cluster [DBG] pgmap v192: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:17:50.690 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:50.689+0000 7fb2ccc45700 1 -- 172.21.15.73:0/3095618709 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2c80f51a0 msgr2=0x7fb2c80f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:17:50.690 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:50.689+0000 7fb2ccc45700 1 --2- 172.21.15.73:0/3095618709 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2c80f51a0 0x7fb2c80f55c0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 rx=0x7fb2b8009fb0 tx=0x7fb2b8009b30).stop 2023-12-01T22:17:50.690 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:50.689+0000 7fb2ccc45700 1 -- 172.21.15.73:0/3095618709 shutdown_connections 2023-12-01T22:17:50.691 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:50.689+0000 7fb2ccc45700 1 --2- 172.21.15.73:0/3095618709 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2c80f51a0 0x7fb2c80f55c0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:17:50.691 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:50.689+0000 7fb2ccc45700 1 -- 172.21.15.73:0/3095618709 >> 172.21.15.73:0/3095618709 conn(0x7fb2c80f09a0 msgr2=0x7fb2c80f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:17:50.691 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:50.689+0000 7fb2ccc45700 1 -- 172.21.15.73:0/3095618709 shutdown_connections 2023-12-01T22:17:50.691 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:50.689+0000 7fb2ccc45700 1 -- 172.21.15.73:0/3095618709 wait complete. 2023-12-01T22:17:50.691 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:50.690+0000 7fb2ccc45700 1 Processor -- start 2023-12-01T22:17:50.691 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:50.690+0000 7fb2ccc45700 1 -- start start 2023-12-01T22:17:50.692 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:50.690+0000 7fb2ccc45700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2c80f51a0 0x7fb2c81086c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:17:50.692 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:50.690+0000 7fb2ccc45700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2c8108bb0 con 0x7fb2c80f51a0 2023-12-01T22:17:50.693 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:50.690+0000 7fb2c77fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2c80f51a0 0x7fb2c81086c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:17:50.693 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:50.690+0000 7fb2c77fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2c80f51a0 0x7fb2c81086c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:35152/0 (socket says 172.21.15.73:35152) 2023-12-01T22:17:50.693 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:50.690+0000 7fb2c77fe700 1 -- 172.21.15.73:0/4117225777 learned_addr learned my addr 172.21.15.73:0/4117225777 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:17:50.693 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:50.691+0000 7fb2c77fe700 1 -- 172.21.15.73:0/4117225777 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2b8005040 con 0x7fb2c80f51a0 2023-12-01T22:17:50.693 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:50.691+0000 7fb2c77fe700 1 --2- 172.21.15.73:0/4117225777 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2c80f51a0 0x7fb2c81086c0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7fb2b8013510 tx=0x7fb2b800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:17:50.693 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:50.691+0000 7fb2affff700 1 -- 172.21.15.73:0/4117225777 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb2b8009320 con 0x7fb2c80f51a0 2023-12-01T22:17:50.693 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:50.691+0000 7fb2ccc45700 1 -- 172.21.15.73:0/4117225777 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2c8108db0 con 0x7fb2c80f51a0 2023-12-01T22:17:50.693 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:50.691+0000 7fb2affff700 1 -- 172.21.15.73:0/4117225777 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb2b8013f00 con 0x7fb2c80f51a0 2023-12-01T22:17:50.693 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:50.691+0000 7fb2affff700 1 -- 172.21.15.73:0/4117225777 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb2b8014dc0 con 0x7fb2c80f51a0 2023-12-01T22:17:50.693 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:50.691+0000 7fb2ccc45700 1 -- 172.21.15.73:0/4117225777 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2c8109240 con 0x7fb2c80f51a0 2023-12-01T22:17:50.693 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:50.692+0000 7fb2affff700 1 -- 172.21.15.73:0/4117225777 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb2b8014420 con 0x7fb2c80f51a0 2023-12-01T22:17:50.693 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:50.692+0000 7fb2ccc45700 1 -- 172.21.15.73:0/4117225777 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2c8045ad0 con 0x7fb2c80f51a0 2023-12-01T22:17:50.693 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:50.692+0000 7fb2affff700 1 --2- 172.21.15.73:0/4117225777 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fb2b002ff60 0x7fb2b0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:17:50.693 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:50.692+0000 7fb2affff700 1 -- 172.21.15.73:0/4117225777 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb2b80410d0 con 0x7fb2c80f51a0 2023-12-01T22:17:50.694 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:50.692+0000 7fb2c6ffd700 1 --2- 172.21.15.73:0/4117225777 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fb2b002ff60 0x7fb2b0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:17:50.694 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:50.692+0000 7fb2c6ffd700 1 --2- 172.21.15.73:0/4117225777 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fb2b002ff60 0x7fb2b0032420 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7fb2bc0096a0 tx=0x7fb2bc006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:17:50.697 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:50.697+0000 7fb2affff700 1 -- 172.21.15.73:0/4117225777 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb2b80095f0 con 0x7fb2c80f51a0 2023-12-01T22:17:50.902 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:50.902+0000 7fb2ccc45700 1 -- 172.21.15.73:0/4117225777 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb2c810a0f0 con 0x7fb2c80f51a0 2023-12-01T22:17:50.903 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:50.903+0000 7fb2affff700 1 -- 172.21.15.73:0/4117225777 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb2b801cb10 con 0x7fb2c80f51a0 2023-12-01T22:17:50.903 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:17:50.903 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:17:50.905 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:50.904+0000 7fb2ccc45700 1 -- 172.21.15.73:0/4117225777 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fb2b002ff60 msgr2=0x7fb2b0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:17:50.905 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:50.904+0000 7fb2ccc45700 1 --2- 172.21.15.73:0/4117225777 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fb2b002ff60 0x7fb2b0032420 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7fb2bc0096a0 tx=0x7fb2bc006b40).stop 2023-12-01T22:17:50.905 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:50.904+0000 7fb2ccc45700 1 -- 172.21.15.73:0/4117225777 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2c80f51a0 msgr2=0x7fb2c81086c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:17:50.905 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:50.904+0000 7fb2ccc45700 1 --2- 172.21.15.73:0/4117225777 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2c80f51a0 0x7fb2c81086c0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7fb2b8013510 tx=0x7fb2b800b040).stop 2023-12-01T22:17:50.905 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:50.904+0000 7fb2ccc45700 1 -- 172.21.15.73:0/4117225777 shutdown_connections 2023-12-01T22:17:50.905 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:50.904+0000 7fb2ccc45700 1 --2- 172.21.15.73:0/4117225777 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fb2b002ff60 0x7fb2b0032420 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:17:50.905 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:50.904+0000 7fb2ccc45700 1 --2- 172.21.15.73:0/4117225777 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2c80f51a0 0x7fb2c81086c0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:17:50.905 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:50.904+0000 7fb2ccc45700 1 -- 172.21.15.73:0/4117225777 >> 172.21.15.73:0/4117225777 conn(0x7fb2c80f09a0 msgr2=0x7fb2c805fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:17:50.906 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:50.904+0000 7fb2ccc45700 1 -- 172.21.15.73:0/4117225777 shutdown_connections 2023-12-01T22:17:50.906 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:50.904+0000 7fb2ccc45700 1 -- 172.21.15.73:0/4117225777 wait complete. 2023-12-01T22:17:50.906 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:17:51.776 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:17:51 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:17:50.682638+0000 mgr.smithi029.loxrfu (mgr.14164) 205 : cluster [DBG] pgmap v193: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:17:51.777 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:17:51 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:17:50.902882+0000 mon.smithi029 (mon.0) 271 : audit [DBG] from='client.? 172.21.15.73:0/4117225777' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:17:52.293 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:17:52.294 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:17:53.455 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:53.453+0000 7f45849f7700 1 -- 172.21.15.73:0/2481344854 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45800f6510 msgr2=0x7f45800f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:17:53.456 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:53.453+0000 7f45849f7700 1 --2- 172.21.15.73:0/2481344854 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45800f6510 0x7f45800f6930 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 rx=0x7f4574009fb0 tx=0x7f4574009b30).stop 2023-12-01T22:17:53.456 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:53.453+0000 7f45849f7700 1 -- 172.21.15.73:0/2481344854 shutdown_connections 2023-12-01T22:17:53.456 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:53.453+0000 7f45849f7700 1 --2- 172.21.15.73:0/2481344854 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45800f6510 0x7f45800f6930 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:17:53.456 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:53.453+0000 7f45849f7700 1 -- 172.21.15.73:0/2481344854 >> 172.21.15.73:0/2481344854 conn(0x7f45800f1c70 msgr2=0x7f45800f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:17:53.456 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:53.453+0000 7f45849f7700 1 -- 172.21.15.73:0/2481344854 shutdown_connections 2023-12-01T22:17:53.456 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:53.453+0000 7f45849f7700 1 -- 172.21.15.73:0/2481344854 wait complete. 2023-12-01T22:17:53.456 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:53.453+0000 7f45849f7700 1 Processor -- start 2023-12-01T22:17:53.456 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:53.453+0000 7f45849f7700 1 -- start start 2023-12-01T22:17:53.456 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:53.454+0000 7f45849f7700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45800f6510 0x7f4580106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:17:53.456 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:53.454+0000 7f45849f7700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4580106c20 con 0x7f45800f6510 2023-12-01T22:17:53.456 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:53.454+0000 7f457e59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45800f6510 0x7f4580106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:17:53.456 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:53.454+0000 7f457e59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45800f6510 0x7f4580106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:35180/0 (socket says 172.21.15.73:35180) 2023-12-01T22:17:53.456 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:53.454+0000 7f457e59c700 1 -- 172.21.15.73:0/2899158234 learned_addr learned my addr 172.21.15.73:0/2899158234 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:17:53.456 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:53.454+0000 7f457e59c700 1 -- 172.21.15.73:0/2899158234 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4574005040 con 0x7f45800f6510 2023-12-01T22:17:53.457 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:53.454+0000 7f457e59c700 1 --2- 172.21.15.73:0/2899158234 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45800f6510 0x7f4580106730 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7f4574013510 tx=0x7f457400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:17:53.457 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:53.454+0000 7f456effd700 1 -- 172.21.15.73:0/2899158234 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4574009320 con 0x7f45800f6510 2023-12-01T22:17:53.457 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:53.455+0000 7f45849f7700 1 -- 172.21.15.73:0/2899158234 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4580106e20 con 0x7f45800f6510 2023-12-01T22:17:53.457 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:53.455+0000 7f456effd700 1 -- 172.21.15.73:0/2899158234 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4574013f00 con 0x7f45800f6510 2023-12-01T22:17:53.457 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:53.455+0000 7f456effd700 1 -- 172.21.15.73:0/2899158234 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4574014d90 con 0x7f45800f6510 2023-12-01T22:17:53.457 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:53.455+0000 7f45849f7700 1 -- 172.21.15.73:0/2899158234 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45801072b0 con 0x7f45800f6510 2023-12-01T22:17:53.458 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:53.455+0000 7f456effd700 1 -- 172.21.15.73:0/2899158234 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f45740263a0 con 0x7f45800f6510 2023-12-01T22:17:53.458 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:53.455+0000 7f456effd700 1 --2- 172.21.15.73:0/2899158234 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f456802ff60 0x7f4568032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:17:53.458 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:53.456+0000 7f456effd700 1 -- 172.21.15.73:0/2899158234 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f45740410f0 con 0x7f45800f6510 2023-12-01T22:17:53.458 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:53.455+0000 7f45849f7700 1 -- 172.21.15.73:0/2899158234 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4580045ad0 con 0x7f45800f6510 2023-12-01T22:17:53.458 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:53.456+0000 7f457dd9b700 1 --2- 172.21.15.73:0/2899158234 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f456802ff60 0x7f4568032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:17:53.458 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:53.456+0000 7f457dd9b700 1 --2- 172.21.15.73:0/2899158234 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f456802ff60 0x7f4568032420 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f45700096a0 tx=0x7f4570006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:17:53.461 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:53.460+0000 7f456effd700 1 -- 172.21.15.73:0/2899158234 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4574012920 con 0x7f45800f6510 2023-12-01T22:17:53.667 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:53.666+0000 7f45849f7700 1 -- 172.21.15.73:0/2899158234 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f45800fbf60 con 0x7f45800f6510 2023-12-01T22:17:53.668 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:53.667+0000 7f456effd700 1 -- 172.21.15.73:0/2899158234 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f457402b020 con 0x7f45800f6510 2023-12-01T22:17:53.668 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:17:53.668 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:17:53.669 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:53.668+0000 7f45849f7700 1 -- 172.21.15.73:0/2899158234 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f456802ff60 msgr2=0x7f4568032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:17:53.669 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:53.668+0000 7f45849f7700 1 --2- 172.21.15.73:0/2899158234 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f456802ff60 0x7f4568032420 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f45700096a0 tx=0x7f4570006b40).stop 2023-12-01T22:17:53.669 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:53.668+0000 7f45849f7700 1 -- 172.21.15.73:0/2899158234 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45800f6510 msgr2=0x7f4580106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:17:53.670 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:53.668+0000 7f45849f7700 1 --2- 172.21.15.73:0/2899158234 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45800f6510 0x7f4580106730 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7f4574013510 tx=0x7f457400b040).stop 2023-12-01T22:17:53.670 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:53.669+0000 7f45849f7700 1 -- 172.21.15.73:0/2899158234 shutdown_connections 2023-12-01T22:17:53.670 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:53.669+0000 7f45849f7700 1 --2- 172.21.15.73:0/2899158234 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f456802ff60 0x7f4568032420 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:17:53.670 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:53.669+0000 7f45849f7700 1 --2- 172.21.15.73:0/2899158234 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45800f6510 0x7f4580106730 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:17:53.670 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:53.669+0000 7f45849f7700 1 -- 172.21.15.73:0/2899158234 >> 172.21.15.73:0/2899158234 conn(0x7f45800f1c70 msgr2=0x7f45800f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:17:53.670 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:53.669+0000 7f45849f7700 1 -- 172.21.15.73:0/2899158234 shutdown_connections 2023-12-01T22:17:53.670 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:53.669+0000 7f45849f7700 1 -- 172.21.15.73:0/2899158234 wait complete. 2023-12-01T22:17:53.670 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:17:54.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:17:53 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:17:52.683024+0000 mgr.smithi029.loxrfu (mgr.14164) 206 : cluster [DBG] pgmap v194: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:17:54.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:17:53 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:17:53.667443+0000 mon.smithi029 (mon.0) 272 : audit [DBG] from='client.? 172.21.15.73:0/2899158234' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:17:55.126 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:17:55.126 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:17:56.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:17:55 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:17:54.683426+0000 mgr.smithi029.loxrfu (mgr.14164) 207 : cluster [DBG] pgmap v195: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:17:56.364 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:56.362+0000 7f10f1cd0700 1 -- 172.21.15.73:0/2751496194 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f10ec060280 msgr2=0x7f10ec0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:17:56.364 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:56.362+0000 7f10f1cd0700 1 --2- 172.21.15.73:0/2751496194 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f10ec060280 0x7f10ec0f90f0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 rx=0x7f10d4009fb0 tx=0x7f10d4009b30).stop 2023-12-01T22:17:56.364 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:56.362+0000 7f10f1cd0700 1 -- 172.21.15.73:0/2751496194 shutdown_connections 2023-12-01T22:17:56.364 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:56.362+0000 7f10f1cd0700 1 --2- 172.21.15.73:0/2751496194 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f10ec060280 0x7f10ec0f90f0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:17:56.364 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:56.362+0000 7f10f1cd0700 1 -- 172.21.15.73:0/2751496194 >> 172.21.15.73:0/2751496194 conn(0x7f10ec0f0b70 msgr2=0x7f10ec0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:17:56.365 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:56.362+0000 7f10f1cd0700 1 -- 172.21.15.73:0/2751496194 shutdown_connections 2023-12-01T22:17:56.365 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:56.362+0000 7f10f1cd0700 1 -- 172.21.15.73:0/2751496194 wait complete. 2023-12-01T22:17:56.365 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:56.363+0000 7f10f1cd0700 1 Processor -- start 2023-12-01T22:17:56.365 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:56.363+0000 7f10f1cd0700 1 -- start start 2023-12-01T22:17:56.367 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:56.363+0000 7f10f1cd0700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f10ec060280 0x7f10ec1066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:17:56.367 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:56.364+0000 7f10f1cd0700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10ec106bd0 con 0x7f10ec060280 2023-12-01T22:17:56.367 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:56.364+0000 7f10eb7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f10ec060280 0x7f10ec1066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:17:56.367 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:56.364+0000 7f10eb7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f10ec060280 0x7f10ec1066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:53354/0 (socket says 172.21.15.73:53354) 2023-12-01T22:17:56.367 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:56.364+0000 7f10eb7fe700 1 -- 172.21.15.73:0/254506426 learned_addr learned my addr 172.21.15.73:0/254506426 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:17:56.367 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:56.364+0000 7f10eb7fe700 1 -- 172.21.15.73:0/254506426 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10d4005040 con 0x7f10ec060280 2023-12-01T22:17:56.367 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:56.364+0000 7f10eb7fe700 1 --2- 172.21.15.73:0/254506426 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f10ec060280 0x7f10ec1066e0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7f10d4013510 tx=0x7f10d400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:17:56.367 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:56.364+0000 7f10e3fff700 1 -- 172.21.15.73:0/254506426 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f10d4009320 con 0x7f10ec060280 2023-12-01T22:17:56.367 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:56.364+0000 7f10f1cd0700 1 -- 172.21.15.73:0/254506426 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10ec106dd0 con 0x7f10ec060280 2023-12-01T22:17:56.367 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:56.364+0000 7f10e3fff700 1 -- 172.21.15.73:0/254506426 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f10d4013f00 con 0x7f10ec060280 2023-12-01T22:17:56.368 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:56.364+0000 7f10e3fff700 1 -- 172.21.15.73:0/254506426 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f10d4014dc0 con 0x7f10ec060280 2023-12-01T22:17:56.368 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:56.364+0000 7f10f1cd0700 1 -- 172.21.15.73:0/254506426 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10ec107260 con 0x7f10ec060280 2023-12-01T22:17:56.368 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:56.365+0000 7f10e3fff700 1 -- 172.21.15.73:0/254506426 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f10d40263a0 con 0x7f10ec060280 2023-12-01T22:17:56.368 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:56.365+0000 7f10f1cd0700 1 -- 172.21.15.73:0/254506426 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10ec045ad0 con 0x7f10ec060280 2023-12-01T22:17:56.368 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:56.365+0000 7f10e3fff700 1 --2- 172.21.15.73:0/254506426 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f10d802ffb0 0x7f10d8032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:17:56.368 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:56.365+0000 7f10e3fff700 1 -- 172.21.15.73:0/254506426 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f10d40412d0 con 0x7f10ec060280 2023-12-01T22:17:56.368 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:56.365+0000 7f10eaffd700 1 --2- 172.21.15.73:0/254506426 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f10d802ffb0 0x7f10d8032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:17:56.368 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:56.366+0000 7f10eaffd700 1 --2- 172.21.15.73:0/254506426 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f10d802ffb0 0x7f10d8032470 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f10dc0096a0 tx=0x7f10dc006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:17:56.371 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:56.370+0000 7f10e3fff700 1 -- 172.21.15.73:0/254506426 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f10d401d380 con 0x7f10ec060280 2023-12-01T22:17:56.575 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:56.574+0000 7f10f1cd0700 1 -- 172.21.15.73:0/254506426 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f10ec108110 con 0x7f10ec060280 2023-12-01T22:17:56.576 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:56.575+0000 7f10e3fff700 1 -- 172.21.15.73:0/254506426 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f10d4012330 con 0x7f10ec060280 2023-12-01T22:17:56.576 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:17:56.576 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:17:56.577 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:56.577+0000 7f10f1cd0700 1 -- 172.21.15.73:0/254506426 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f10d802ffb0 msgr2=0x7f10d8032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:17:56.577 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:56.577+0000 7f10f1cd0700 1 --2- 172.21.15.73:0/254506426 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f10d802ffb0 0x7f10d8032470 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f10dc0096a0 tx=0x7f10dc006b40).stop 2023-12-01T22:17:56.577 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:56.577+0000 7f10f1cd0700 1 -- 172.21.15.73:0/254506426 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f10ec060280 msgr2=0x7f10ec1066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:17:56.577 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:56.577+0000 7f10f1cd0700 1 --2- 172.21.15.73:0/254506426 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f10ec060280 0x7f10ec1066e0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7f10d4013510 tx=0x7f10d400b040).stop 2023-12-01T22:17:56.578 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:56.577+0000 7f10f1cd0700 1 -- 172.21.15.73:0/254506426 shutdown_connections 2023-12-01T22:17:56.578 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:56.577+0000 7f10f1cd0700 1 --2- 172.21.15.73:0/254506426 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f10d802ffb0 0x7f10d8032470 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:17:56.578 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:56.577+0000 7f10f1cd0700 1 --2- 172.21.15.73:0/254506426 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f10ec060280 0x7f10ec1066e0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:17:56.578 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:56.577+0000 7f10f1cd0700 1 -- 172.21.15.73:0/254506426 >> 172.21.15.73:0/254506426 conn(0x7f10ec0f0b70 msgr2=0x7f10ec0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:17:56.578 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:56.577+0000 7f10f1cd0700 1 -- 172.21.15.73:0/254506426 shutdown_connections 2023-12-01T22:17:56.578 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:56.577+0000 7f10f1cd0700 1 -- 172.21.15.73:0/254506426 wait complete. 2023-12-01T22:17:56.578 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:17:57.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:17:56 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:17:56.575585+0000 mon.smithi029 (mon.0) 273 : audit [DBG] from='client.? 172.21.15.73:0/254506426' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:17:57.978 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:17:57.979 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:17:58.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:17:57 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:17:56.683888+0000 mgr.smithi029.loxrfu (mgr.14164) 208 : cluster [DBG] pgmap v196: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:17:59.181 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:59.179+0000 7f30ac2a9700 1 -- 172.21.15.73:0/653490860 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30a40f6510 msgr2=0x7f30a40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:17:59.181 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:59.179+0000 7f30ac2a9700 1 --2- 172.21.15.73:0/653490860 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30a40f6510 0x7f30a40f6930 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 rx=0x7f3098009fb0 tx=0x7f3098009b30).stop 2023-12-01T22:17:59.181 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:59.180+0000 7f30ac2a9700 1 -- 172.21.15.73:0/653490860 shutdown_connections 2023-12-01T22:17:59.182 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:59.180+0000 7f30ac2a9700 1 --2- 172.21.15.73:0/653490860 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30a40f6510 0x7f30a40f6930 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:17:59.182 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:59.180+0000 7f30ac2a9700 1 -- 172.21.15.73:0/653490860 >> 172.21.15.73:0/653490860 conn(0x7f30a40f1c70 msgr2=0x7f30a40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:17:59.182 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:59.180+0000 7f30ac2a9700 1 -- 172.21.15.73:0/653490860 shutdown_connections 2023-12-01T22:17:59.182 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:59.180+0000 7f30ac2a9700 1 -- 172.21.15.73:0/653490860 wait complete. 2023-12-01T22:17:59.182 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:59.180+0000 7f30ac2a9700 1 Processor -- start 2023-12-01T22:17:59.182 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:59.180+0000 7f30ac2a9700 1 -- start start 2023-12-01T22:17:59.182 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:59.180+0000 7f30ac2a9700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30a40f6510 0x7f30a4106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:17:59.182 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:59.180+0000 7f30ac2a9700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30a4106c20 con 0x7f30a40f6510 2023-12-01T22:17:59.182 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:59.181+0000 7f30aa045700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30a40f6510 0x7f30a4106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:17:59.182 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:59.181+0000 7f30aa045700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30a40f6510 0x7f30a4106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:53378/0 (socket says 172.21.15.73:53378) 2023-12-01T22:17:59.182 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:59.181+0000 7f30aa045700 1 -- 172.21.15.73:0/618856307 learned_addr learned my addr 172.21.15.73:0/618856307 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:17:59.184 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:59.181+0000 7f30aa045700 1 -- 172.21.15.73:0/618856307 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3098005040 con 0x7f30a40f6510 2023-12-01T22:17:59.184 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:59.181+0000 7f30aa045700 1 --2- 172.21.15.73:0/618856307 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30a40f6510 0x7f30a4106730 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7f3098013510 tx=0x7f309800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:17:59.184 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:59.181+0000 7f3096ffd700 1 -- 172.21.15.73:0/618856307 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3098009320 con 0x7f30a40f6510 2023-12-01T22:17:59.184 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:59.181+0000 7f3096ffd700 1 -- 172.21.15.73:0/618856307 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3098013f00 con 0x7f30a40f6510 2023-12-01T22:17:59.184 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:59.181+0000 7f30ac2a9700 1 -- 172.21.15.73:0/618856307 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30a4106e20 con 0x7f30a40f6510 2023-12-01T22:17:59.184 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:59.181+0000 7f3096ffd700 1 -- 172.21.15.73:0/618856307 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3098014d90 con 0x7f30a40f6510 2023-12-01T22:17:59.184 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:59.181+0000 7f30ac2a9700 1 -- 172.21.15.73:0/618856307 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30a41072b0 con 0x7f30a40f6510 2023-12-01T22:17:59.184 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:59.182+0000 7f3096ffd700 1 -- 172.21.15.73:0/618856307 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f30980263a0 con 0x7f30a40f6510 2023-12-01T22:17:59.184 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:59.182+0000 7f30ac2a9700 1 -- 172.21.15.73:0/618856307 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30a4045ad0 con 0x7f30a40f6510 2023-12-01T22:17:59.184 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:59.182+0000 7f3096ffd700 1 --2- 172.21.15.73:0/618856307 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f309002ff60 0x7f3090032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:17:59.184 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:59.182+0000 7f3096ffd700 1 -- 172.21.15.73:0/618856307 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3098041110 con 0x7f30a40f6510 2023-12-01T22:17:59.184 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:59.183+0000 7f30a9844700 1 --2- 172.21.15.73:0/618856307 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f309002ff60 0x7f3090032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:17:59.185 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:59.183+0000 7f30a9844700 1 --2- 172.21.15.73:0/618856307 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f309002ff60 0x7f3090032420 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f30a00096a0 tx=0x7f30a0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:17:59.188 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:59.187+0000 7f3096ffd700 1 -- 172.21.15.73:0/618856307 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f30980126f0 con 0x7f30a40f6510 2023-12-01T22:17:59.395 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:59.394+0000 7f30ac2a9700 1 -- 172.21.15.73:0/618856307 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f30a40fbf60 con 0x7f30a40f6510 2023-12-01T22:17:59.395 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:59.395+0000 7f3096ffd700 1 -- 172.21.15.73:0/618856307 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3098012330 con 0x7f30a40f6510 2023-12-01T22:17:59.395 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:17:59.396 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:17:59.397 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:59.396+0000 7f30ac2a9700 1 -- 172.21.15.73:0/618856307 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f309002ff60 msgr2=0x7f3090032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:17:59.397 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:59.396+0000 7f30ac2a9700 1 --2- 172.21.15.73:0/618856307 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f309002ff60 0x7f3090032420 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f30a00096a0 tx=0x7f30a0006b40).stop 2023-12-01T22:17:59.397 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:59.396+0000 7f30ac2a9700 1 -- 172.21.15.73:0/618856307 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30a40f6510 msgr2=0x7f30a4106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:17:59.397 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:59.396+0000 7f30ac2a9700 1 --2- 172.21.15.73:0/618856307 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30a40f6510 0x7f30a4106730 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7f3098013510 tx=0x7f309800b040).stop 2023-12-01T22:17:59.397 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:59.396+0000 7f30ac2a9700 1 -- 172.21.15.73:0/618856307 shutdown_connections 2023-12-01T22:17:59.397 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:59.396+0000 7f30ac2a9700 1 --2- 172.21.15.73:0/618856307 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f309002ff60 0x7f3090032420 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:17:59.397 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:59.396+0000 7f30ac2a9700 1 --2- 172.21.15.73:0/618856307 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30a40f6510 0x7f30a4106730 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:17:59.397 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:59.396+0000 7f30ac2a9700 1 -- 172.21.15.73:0/618856307 >> 172.21.15.73:0/618856307 conn(0x7f30a40f1c70 msgr2=0x7f30a40f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:17:59.398 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:59.396+0000 7f30ac2a9700 1 -- 172.21.15.73:0/618856307 shutdown_connections 2023-12-01T22:17:59.398 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:17:59.396+0000 7f30ac2a9700 1 -- 172.21.15.73:0/618856307 wait complete. 2023-12-01T22:17:59.398 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:18:00.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:17:59 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:17:58.684384+0000 mgr.smithi029.loxrfu (mgr.14164) 209 : cluster [DBG] pgmap v197: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:18:00.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:17:59 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:17:59.395119+0000 mon.smithi029 (mon.0) 274 : audit [DBG] from='client.? 172.21.15.73:0/618856307' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:18:00.743 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:18:00.743 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:18:01.776 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:18:01 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:18:00.684872+0000 mgr.smithi029.loxrfu (mgr.14164) 210 : cluster [DBG] pgmap v198: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:18:01.902 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:01.899+0000 7f2fc9ffb700 1 -- 172.21.15.73:0/2097758802 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2fc00138a0 con 0x7f2fcc0f6510 2023-12-01T22:18:01.902 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:01.900+0000 7f2fd14dc700 1 -- 172.21.15.73:0/2097758802 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2fcc0f6510 msgr2=0x7f2fcc0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:18:01.902 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:01.900+0000 7f2fd14dc700 1 --2- 172.21.15.73:0/2097758802 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2fcc0f6510 0x7f2fcc0f6930 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 rx=0x7f2fc0009fb0 tx=0x7f2fc0009b30).stop 2023-12-01T22:18:01.902 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:01.900+0000 7f2fd14dc700 1 -- 172.21.15.73:0/2097758802 shutdown_connections 2023-12-01T22:18:01.902 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:01.900+0000 7f2fd14dc700 1 --2- 172.21.15.73:0/2097758802 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2fcc0f6510 0x7f2fcc0f6930 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:18:01.902 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:01.900+0000 7f2fd14dc700 1 -- 172.21.15.73:0/2097758802 >> 172.21.15.73:0/2097758802 conn(0x7f2fcc0f1c70 msgr2=0x7f2fcc0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:18:01.902 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:01.900+0000 7f2fd14dc700 1 -- 172.21.15.73:0/2097758802 shutdown_connections 2023-12-01T22:18:01.902 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:01.900+0000 7f2fd14dc700 1 -- 172.21.15.73:0/2097758802 wait complete. 2023-12-01T22:18:01.904 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:01.901+0000 7f2fd14dc700 1 Processor -- start 2023-12-01T22:18:01.904 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:01.901+0000 7f2fd14dc700 1 -- start start 2023-12-01T22:18:01.904 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:01.901+0000 7f2fd14dc700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2fcc104630 0x7f2fcc104a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:18:01.904 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:01.901+0000 7f2fd14dc700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fcc104f40 con 0x7f2fcc104630 2023-12-01T22:18:01.904 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:01.901+0000 7f2fcaffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2fcc104630 0x7f2fcc104a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:18:01.904 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:01.901+0000 7f2fcaffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2fcc104630 0x7f2fcc104a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:53402/0 (socket says 172.21.15.73:53402) 2023-12-01T22:18:01.904 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:01.901+0000 7f2fcaffd700 1 -- 172.21.15.73:0/297634972 learned_addr learned my addr 172.21.15.73:0/297634972 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:18:01.904 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:01.901+0000 7f2fcaffd700 1 -- 172.21.15.73:0/297634972 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fc0005040 con 0x7f2fcc104630 2023-12-01T22:18:01.904 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:01.902+0000 7f2fcaffd700 1 --2- 172.21.15.73:0/297634972 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2fcc104630 0x7f2fcc104a50 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7f2fc0009f80 tx=0x7f2fc0009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:18:01.904 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:01.902+0000 7f2fb37fe700 1 -- 172.21.15.73:0/297634972 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2fc00136c0 con 0x7f2fcc104630 2023-12-01T22:18:01.904 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:01.902+0000 7f2fd14dc700 1 -- 172.21.15.73:0/297634972 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fcc105140 con 0x7f2fcc104630 2023-12-01T22:18:01.905 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:01.902+0000 7f2fb37fe700 1 -- 172.21.15.73:0/297634972 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2fc00095f0 con 0x7f2fcc104630 2023-12-01T22:18:01.905 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:01.902+0000 7f2fb37fe700 1 -- 172.21.15.73:0/297634972 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2fc0014420 con 0x7f2fcc104630 2023-12-01T22:18:01.905 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:01.902+0000 7f2fd14dc700 1 -- 172.21.15.73:0/297634972 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fcc105c90 con 0x7f2fcc104630 2023-12-01T22:18:01.905 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:01.903+0000 7f2fb37fe700 1 -- 172.21.15.73:0/297634972 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2fc0025060 con 0x7f2fcc104630 2023-12-01T22:18:01.906 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:01.903+0000 7f2fb37fe700 1 --2- 172.21.15.73:0/297634972 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f2fb40343b0 0x7f2fb4036870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:18:01.906 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:01.903+0000 7f2fb37fe700 1 -- 172.21.15.73:0/297634972 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2fc0040a60 con 0x7f2fcc104630 2023-12-01T22:18:01.906 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:01.903+0000 7f2fd14dc700 1 -- 172.21.15.73:0/297634972 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fcc059100 con 0x7f2fcc104630 2023-12-01T22:18:01.906 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:01.903+0000 7f2fca7fc700 1 --2- 172.21.15.73:0/297634972 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f2fb40343b0 0x7f2fb4036870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:18:01.906 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:01.903+0000 7f2fca7fc700 1 --2- 172.21.15.73:0/297634972 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f2fb40343b0 0x7f2fb4036870 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f2fbc0096a0 tx=0x7f2fbc006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:18:01.909 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:01.908+0000 7f2fb37fe700 1 -- 172.21.15.73:0/297634972 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2fc0012920 con 0x7f2fcc104630 2023-12-01T22:18:02.115 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:02.113+0000 7f2fd14dc700 1 -- 172.21.15.73:0/297634972 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2fcc0fe470 con 0x7f2fcc104630 2023-12-01T22:18:02.115 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:02.114+0000 7f2fb37fe700 1 -- 172.21.15.73:0/297634972 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2fc0012330 con 0x7f2fcc104630 2023-12-01T22:18:02.115 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:18:02.115 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:18:02.117 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:02.115+0000 7f2fd14dc700 1 -- 172.21.15.73:0/297634972 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f2fb40343b0 msgr2=0x7f2fb4036870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:18:02.117 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:02.115+0000 7f2fd14dc700 1 --2- 172.21.15.73:0/297634972 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f2fb40343b0 0x7f2fb4036870 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f2fbc0096a0 tx=0x7f2fbc006b40).stop 2023-12-01T22:18:02.117 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:02.115+0000 7f2fd14dc700 1 -- 172.21.15.73:0/297634972 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2fcc104630 msgr2=0x7f2fcc104a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:18:02.117 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:02.115+0000 7f2fd14dc700 1 --2- 172.21.15.73:0/297634972 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2fcc104630 0x7f2fcc104a50 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7f2fc0009f80 tx=0x7f2fc0009b30).stop 2023-12-01T22:18:02.117 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:02.115+0000 7f2fd14dc700 1 -- 172.21.15.73:0/297634972 shutdown_connections 2023-12-01T22:18:02.117 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:02.115+0000 7f2fd14dc700 1 --2- 172.21.15.73:0/297634972 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f2fb40343b0 0x7f2fb4036870 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:18:02.117 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:02.115+0000 7f2fd14dc700 1 --2- 172.21.15.73:0/297634972 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2fcc104630 0x7f2fcc104a50 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:18:02.117 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:02.115+0000 7f2fd14dc700 1 -- 172.21.15.73:0/297634972 >> 172.21.15.73:0/297634972 conn(0x7f2fcc0f1c70 msgr2=0x7f2fcc0f2870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:18:02.117 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:02.115+0000 7f2fd14dc700 1 -- 172.21.15.73:0/297634972 shutdown_connections 2023-12-01T22:18:02.117 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:02.115+0000 7f2fd14dc700 1 -- 172.21.15.73:0/297634972 wait complete. 2023-12-01T22:18:02.118 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:18:02.776 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:18:02 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:18:02.115036+0000 mon.smithi029 (mon.0) 275 : audit [DBG] from='client.? 172.21.15.73:0/297634972' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:18:03.514 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:18:03.514 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:18:03.776 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:18:03 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:18:02.685342+0000 mgr.smithi029.loxrfu (mgr.14164) 211 : cluster [DBG] pgmap v199: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:18:04.762 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:04.759+0000 7f324bfff700 1 -- 172.21.15.73:0/1512449661 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f324c0601e0 msgr2=0x7f324c0f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:18:04.762 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:04.759+0000 7f324bfff700 1 --2- 172.21.15.73:0/1512449661 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f324c0601e0 0x7f324c0f8f20 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 rx=0x7f323c009fb0 tx=0x7f323c009b30).stop 2023-12-01T22:18:04.762 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:04.759+0000 7f324bfff700 1 -- 172.21.15.73:0/1512449661 shutdown_connections 2023-12-01T22:18:04.763 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:04.759+0000 7f324bfff700 1 --2- 172.21.15.73:0/1512449661 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f324c0601e0 0x7f324c0f8f20 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:18:04.763 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:04.759+0000 7f324bfff700 1 -- 172.21.15.73:0/1512449661 >> 172.21.15.73:0/1512449661 conn(0x7f324c0f09a0 msgr2=0x7f324c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:18:04.763 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:04.760+0000 7f324bfff700 1 -- 172.21.15.73:0/1512449661 shutdown_connections 2023-12-01T22:18:04.763 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:04.760+0000 7f324bfff700 1 -- 172.21.15.73:0/1512449661 wait complete. 2023-12-01T22:18:04.763 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:04.760+0000 7f324bfff700 1 Processor -- start 2023-12-01T22:18:04.763 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:04.760+0000 7f324bfff700 1 -- start start 2023-12-01T22:18:04.763 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:04.760+0000 7f324bfff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f324c0601e0 0x7f324c106500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:18:04.763 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:04.760+0000 7f324bfff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f324c1069f0 con 0x7f324c0601e0 2023-12-01T22:18:04.763 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:04.760+0000 7f324affd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f324c0601e0 0x7f324c106500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:18:04.763 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:04.761+0000 7f324affd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f324c0601e0 0x7f324c106500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:39910/0 (socket says 172.21.15.73:39910) 2023-12-01T22:18:04.763 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:04.761+0000 7f324affd700 1 -- 172.21.15.73:0/3491050723 learned_addr learned my addr 172.21.15.73:0/3491050723 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:18:04.763 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:04.761+0000 7f324affd700 1 -- 172.21.15.73:0/3491050723 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f323c005040 con 0x7f324c0601e0 2023-12-01T22:18:04.763 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:04.761+0000 7f324affd700 1 --2- 172.21.15.73:0/3491050723 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f324c0601e0 0x7f324c106500 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7f323c013510 tx=0x7f323c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:18:04.764 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:04.761+0000 7f32337fe700 1 -- 172.21.15.73:0/3491050723 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f323c009320 con 0x7f324c0601e0 2023-12-01T22:18:04.765 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:04.761+0000 7f324bfff700 1 -- 172.21.15.73:0/3491050723 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f324c106bf0 con 0x7f324c0601e0 2023-12-01T22:18:04.765 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:04.761+0000 7f32337fe700 1 -- 172.21.15.73:0/3491050723 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f323c013f00 con 0x7f324c0601e0 2023-12-01T22:18:04.765 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:04.761+0000 7f32337fe700 1 -- 172.21.15.73:0/3491050723 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f323c014d90 con 0x7f324c0601e0 2023-12-01T22:18:04.765 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:04.761+0000 7f324bfff700 1 -- 172.21.15.73:0/3491050723 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f324c107080 con 0x7f324c0601e0 2023-12-01T22:18:04.765 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:04.762+0000 7f32337fe700 1 -- 172.21.15.73:0/3491050723 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f323c014420 con 0x7f324c0601e0 2023-12-01T22:18:04.765 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:04.762+0000 7f324bfff700 1 -- 172.21.15.73:0/3491050723 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f324c045ad0 con 0x7f324c0601e0 2023-12-01T22:18:04.765 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:04.762+0000 7f32337fe700 1 --2- 172.21.15.73:0/3491050723 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f323402ff60 0x7f3234032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:18:04.765 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:04.762+0000 7f32337fe700 1 -- 172.21.15.73:0/3491050723 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f323c0410e0 con 0x7f324c0601e0 2023-12-01T22:18:04.765 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:04.762+0000 7f324a7fc700 1 --2- 172.21.15.73:0/3491050723 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f323402ff60 0x7f3234032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:18:04.765 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:04.763+0000 7f324a7fc700 1 --2- 172.21.15.73:0/3491050723 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f323402ff60 0x7f3234032420 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f32400096a0 tx=0x7f3240006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:18:04.768 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:04.767+0000 7f32337fe700 1 -- 172.21.15.73:0/3491050723 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f323c0095f0 con 0x7f324c0601e0 2023-12-01T22:18:04.975 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:04.973+0000 7f324bfff700 1 -- 172.21.15.73:0/3491050723 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f324c107ed0 con 0x7f324c0601e0 2023-12-01T22:18:04.975 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:04.973+0000 7f32337fe700 1 -- 172.21.15.73:0/3491050723 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f323c01db70 con 0x7f324c0601e0 2023-12-01T22:18:04.975 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:18:04.975 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:18:04.977 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:04.975+0000 7f324bfff700 1 -- 172.21.15.73:0/3491050723 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f323402ff60 msgr2=0x7f3234032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:18:04.977 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:04.975+0000 7f324bfff700 1 --2- 172.21.15.73:0/3491050723 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f323402ff60 0x7f3234032420 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f32400096a0 tx=0x7f3240006b40).stop 2023-12-01T22:18:04.977 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:04.975+0000 7f324bfff700 1 -- 172.21.15.73:0/3491050723 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f324c0601e0 msgr2=0x7f324c106500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:18:04.977 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:04.975+0000 7f324bfff700 1 --2- 172.21.15.73:0/3491050723 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f324c0601e0 0x7f324c106500 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7f323c013510 tx=0x7f323c00b040).stop 2023-12-01T22:18:04.977 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:04.975+0000 7f324bfff700 1 -- 172.21.15.73:0/3491050723 shutdown_connections 2023-12-01T22:18:04.977 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:04.975+0000 7f324bfff700 1 --2- 172.21.15.73:0/3491050723 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f323402ff60 0x7f3234032420 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:18:04.977 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:04.975+0000 7f324bfff700 1 --2- 172.21.15.73:0/3491050723 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f324c0601e0 0x7f324c106500 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:18:04.977 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:04.975+0000 7f324bfff700 1 -- 172.21.15.73:0/3491050723 >> 172.21.15.73:0/3491050723 conn(0x7f324c0f09a0 msgr2=0x7f324c0f1420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:18:04.977 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:04.975+0000 7f324bfff700 1 -- 172.21.15.73:0/3491050723 shutdown_connections 2023-12-01T22:18:04.977 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:04.975+0000 7f324bfff700 1 -- 172.21.15.73:0/3491050723 wait complete. 2023-12-01T22:18:04.978 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:18:06.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:18:05 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:18:04.685618+0000 mgr.smithi029.loxrfu (mgr.14164) 212 : cluster [DBG] pgmap v200: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:18:06.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:18:05 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:18:04.974928+0000 mon.smithi029 (mon.0) 276 : audit [DBG] from='client.? 172.21.15.73:0/3491050723' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:18:06.342 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:18:06.342 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:18:07.549 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:07.547+0000 7f29f315f700 1 -- 172.21.15.73:0/2789993585 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29ec0f5410 msgr2=0x7f29ec0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:18:07.549 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:07.547+0000 7f29f315f700 1 --2- 172.21.15.73:0/2789993585 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29ec0f5410 0x7f29ec0f5830 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 rx=0x7f29e0009fb0 tx=0x7f29e0009b30).stop 2023-12-01T22:18:07.549 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:07.547+0000 7f29f315f700 1 -- 172.21.15.73:0/2789993585 shutdown_connections 2023-12-01T22:18:07.549 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:07.547+0000 7f29f315f700 1 --2- 172.21.15.73:0/2789993585 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29ec0f5410 0x7f29ec0f5830 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:18:07.549 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:07.547+0000 7f29f315f700 1 -- 172.21.15.73:0/2789993585 >> 172.21.15.73:0/2789993585 conn(0x7f29ec0f0b70 msgr2=0x7f29ec0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:18:07.549 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:07.547+0000 7f29f315f700 1 -- 172.21.15.73:0/2789993585 shutdown_connections 2023-12-01T22:18:07.549 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:07.547+0000 7f29f315f700 1 -- 172.21.15.73:0/2789993585 wait complete. 2023-12-01T22:18:07.551 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:07.548+0000 7f29f315f700 1 Processor -- start 2023-12-01T22:18:07.551 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:07.548+0000 7f29f315f700 1 -- start start 2023-12-01T22:18:07.551 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:07.548+0000 7f29f315f700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29ec0f5410 0x7f29ec105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:18:07.551 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:07.548+0000 7f29f315f700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29ec105b20 con 0x7f29ec0f5410 2023-12-01T22:18:07.551 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:07.548+0000 7f29f0efb700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29ec0f5410 0x7f29ec105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:18:07.551 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:07.548+0000 7f29f0efb700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29ec0f5410 0x7f29ec105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:39928/0 (socket says 172.21.15.73:39928) 2023-12-01T22:18:07.551 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:07.548+0000 7f29f0efb700 1 -- 172.21.15.73:0/3621979349 learned_addr learned my addr 172.21.15.73:0/3621979349 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:18:07.551 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:07.549+0000 7f29f0efb700 1 -- 172.21.15.73:0/3621979349 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29e0005040 con 0x7f29ec0f5410 2023-12-01T22:18:07.551 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:07.549+0000 7f29f0efb700 1 --2- 172.21.15.73:0/3621979349 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29ec0f5410 0x7f29ec105630 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7f29e0013510 tx=0x7f29e000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:18:07.551 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:07.549+0000 7f29e97fa700 1 -- 172.21.15.73:0/3621979349 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f29e0009320 con 0x7f29ec0f5410 2023-12-01T22:18:07.551 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:07.549+0000 7f29e97fa700 1 -- 172.21.15.73:0/3621979349 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f29e0013f00 con 0x7f29ec0f5410 2023-12-01T22:18:07.551 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:07.549+0000 7f29f315f700 1 -- 172.21.15.73:0/3621979349 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29ec105d20 con 0x7f29ec0f5410 2023-12-01T22:18:07.551 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:07.549+0000 7f29e97fa700 1 -- 172.21.15.73:0/3621979349 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f29e0014d90 con 0x7f29ec0f5410 2023-12-01T22:18:07.551 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:07.549+0000 7f29f315f700 1 -- 172.21.15.73:0/3621979349 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29ec1061b0 con 0x7f29ec0f5410 2023-12-01T22:18:07.552 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:07.550+0000 7f29e97fa700 1 -- 172.21.15.73:0/3621979349 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f29e00263a0 con 0x7f29ec0f5410 2023-12-01T22:18:07.552 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:07.550+0000 7f29e97fa700 1 --2- 172.21.15.73:0/3621979349 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f29dc02ffb0 0x7f29dc032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:18:07.552 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:07.550+0000 7f29e97fa700 1 -- 172.21.15.73:0/3621979349 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f29e00410c0 con 0x7f29ec0f5410 2023-12-01T22:18:07.553 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:07.550+0000 7f29f315f700 1 -- 172.21.15.73:0/3621979349 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29ec045ad0 con 0x7f29ec0f5410 2023-12-01T22:18:07.553 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:07.550+0000 7f29ebfff700 1 --2- 172.21.15.73:0/3621979349 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f29dc02ffb0 0x7f29dc032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:18:07.553 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:07.551+0000 7f29ebfff700 1 --2- 172.21.15.73:0/3621979349 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f29dc02ffb0 0x7f29dc032470 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f29d80096a0 tx=0x7f29d8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:18:07.556 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:07.555+0000 7f29e97fa700 1 -- 172.21.15.73:0/3621979349 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f29e0012920 con 0x7f29ec0f5410 2023-12-01T22:18:07.762 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:07.761+0000 7f29f315f700 1 -- 172.21.15.73:0/3621979349 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f29ec0faf50 con 0x7f29ec0f5410 2023-12-01T22:18:07.763 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:07.762+0000 7f29e97fa700 1 -- 172.21.15.73:0/3621979349 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f29e002b020 con 0x7f29ec0f5410 2023-12-01T22:18:07.763 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:18:07.763 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:18:07.765 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:07.763+0000 7f29f315f700 1 -- 172.21.15.73:0/3621979349 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f29dc02ffb0 msgr2=0x7f29dc032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:18:07.765 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:07.763+0000 7f29f315f700 1 --2- 172.21.15.73:0/3621979349 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f29dc02ffb0 0x7f29dc032470 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f29d80096a0 tx=0x7f29d8006b40).stop 2023-12-01T22:18:07.765 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:07.763+0000 7f29f315f700 1 -- 172.21.15.73:0/3621979349 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29ec0f5410 msgr2=0x7f29ec105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:18:07.765 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:07.763+0000 7f29f315f700 1 --2- 172.21.15.73:0/3621979349 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29ec0f5410 0x7f29ec105630 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7f29e0013510 tx=0x7f29e000b040).stop 2023-12-01T22:18:07.765 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:07.763+0000 7f29f315f700 1 -- 172.21.15.73:0/3621979349 shutdown_connections 2023-12-01T22:18:07.765 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:07.763+0000 7f29f315f700 1 --2- 172.21.15.73:0/3621979349 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f29dc02ffb0 0x7f29dc032470 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:18:07.765 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:07.763+0000 7f29f315f700 1 --2- 172.21.15.73:0/3621979349 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29ec0f5410 0x7f29ec105630 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:18:07.765 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:07.763+0000 7f29f315f700 1 -- 172.21.15.73:0/3621979349 >> 172.21.15.73:0/3621979349 conn(0x7f29ec0f0b70 msgr2=0x7f29ec0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:18:07.765 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:07.763+0000 7f29f315f700 1 -- 172.21.15.73:0/3621979349 shutdown_connections 2023-12-01T22:18:07.765 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:07.763+0000 7f29f315f700 1 -- 172.21.15.73:0/3621979349 wait complete. 2023-12-01T22:18:07.766 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:18:08.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:18:07 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:18:06.686077+0000 mgr.smithi029.loxrfu (mgr.14164) 213 : cluster [DBG] pgmap v201: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:18:09.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:18:08 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:18:07.762944+0000 mon.smithi029 (mon.0) 277 : audit [DBG] from='client.? 172.21.15.73:0/3621979349' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:18:09.146 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:18:09.147 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:18:10.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:18:09 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:18:08.686466+0000 mgr.smithi029.loxrfu (mgr.14164) 214 : cluster [DBG] pgmap v202: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:18:10.362 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:10.359+0000 7f42022d4700 1 -- 172.21.15.73:0/1682238650 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41fc0f51a0 msgr2=0x7f41fc0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:18:10.362 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:10.359+0000 7f42022d4700 1 --2- 172.21.15.73:0/1682238650 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41fc0f51a0 0x7f41fc0f55c0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 rx=0x7f41ec009fb0 tx=0x7f41ec009b30).stop 2023-12-01T22:18:10.362 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:10.360+0000 7f42022d4700 1 -- 172.21.15.73:0/1682238650 shutdown_connections 2023-12-01T22:18:10.362 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:10.360+0000 7f42022d4700 1 --2- 172.21.15.73:0/1682238650 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41fc0f51a0 0x7f41fc0f55c0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:18:10.362 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:10.360+0000 7f42022d4700 1 -- 172.21.15.73:0/1682238650 >> 172.21.15.73:0/1682238650 conn(0x7f41fc0f09a0 msgr2=0x7f41fc0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:18:10.363 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:10.360+0000 7f42022d4700 1 -- 172.21.15.73:0/1682238650 shutdown_connections 2023-12-01T22:18:10.363 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:10.360+0000 7f42022d4700 1 -- 172.21.15.73:0/1682238650 wait complete. 2023-12-01T22:18:10.363 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:10.360+0000 7f42022d4700 1 Processor -- start 2023-12-01T22:18:10.363 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:10.360+0000 7f42022d4700 1 -- start start 2023-12-01T22:18:10.363 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:10.361+0000 7f42022d4700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41fc0f51a0 0x7f41fc109170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:18:10.363 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:10.361+0000 7f42022d4700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41fc060280 con 0x7f41fc0f51a0 2023-12-01T22:18:10.365 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:10.361+0000 7f42012d2700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41fc0f51a0 0x7f41fc109170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:18:10.365 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:10.361+0000 7f42012d2700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41fc0f51a0 0x7f41fc109170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:39958/0 (socket says 172.21.15.73:39958) 2023-12-01T22:18:10.365 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:10.361+0000 7f42012d2700 1 -- 172.21.15.73:0/328424293 learned_addr learned my addr 172.21.15.73:0/328424293 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:18:10.365 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:10.361+0000 7f42012d2700 1 -- 172.21.15.73:0/328424293 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41ec005040 con 0x7f41fc0f51a0 2023-12-01T22:18:10.365 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:10.361+0000 7f42012d2700 1 --2- 172.21.15.73:0/328424293 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41fc0f51a0 0x7f41fc109170 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7f41ec013510 tx=0x7f41ec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:18:10.365 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:10.362+0000 7f41f1ffb700 1 -- 172.21.15.73:0/328424293 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f41ec009320 con 0x7f41fc0f51a0 2023-12-01T22:18:10.365 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:10.362+0000 7f42022d4700 1 -- 172.21.15.73:0/328424293 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41fc060480 con 0x7f41fc0f51a0 2023-12-01T22:18:10.365 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:10.362+0000 7f42022d4700 1 -- 172.21.15.73:0/328424293 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41fc1098a0 con 0x7f41fc0f51a0 2023-12-01T22:18:10.365 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:10.362+0000 7f41f1ffb700 1 -- 172.21.15.73:0/328424293 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f41ec013f00 con 0x7f41fc0f51a0 2023-12-01T22:18:10.365 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:10.362+0000 7f41f1ffb700 1 -- 172.21.15.73:0/328424293 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f41ec014d90 con 0x7f41fc0f51a0 2023-12-01T22:18:10.365 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:10.363+0000 7f41f1ffb700 1 -- 172.21.15.73:0/328424293 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f41ec0263a0 con 0x7f41fc0f51a0 2023-12-01T22:18:10.366 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:10.363+0000 7f42022d4700 1 -- 172.21.15.73:0/328424293 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41fc045ad0 con 0x7f41fc0f51a0 2023-12-01T22:18:10.366 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:10.363+0000 7f41f1ffb700 1 --2- 172.21.15.73:0/328424293 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f41e802ff60 0x7f41e8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:18:10.366 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:10.363+0000 7f41f1ffb700 1 -- 172.21.15.73:0/328424293 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f41ec041340 con 0x7f41fc0f51a0 2023-12-01T22:18:10.366 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:10.363+0000 7f4200ad1700 1 --2- 172.21.15.73:0/328424293 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f41e802ff60 0x7f41e8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:18:10.366 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:10.363+0000 7f4200ad1700 1 --2- 172.21.15.73:0/328424293 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f41e802ff60 0x7f41e8032420 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f41f80096a0 tx=0x7f41f8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:18:10.369 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:10.368+0000 7f41f1ffb700 1 -- 172.21.15.73:0/328424293 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f41ec012330 con 0x7f41fc0f51a0 2023-12-01T22:18:10.574 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:10.572+0000 7f42022d4700 1 -- 172.21.15.73:0/328424293 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f41fc0fabe0 con 0x7f41fc0f51a0 2023-12-01T22:18:10.574 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:10.573+0000 7f41f1ffb700 1 -- 172.21.15.73:0/328424293 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f41ec01cc70 con 0x7f41fc0f51a0 2023-12-01T22:18:10.575 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:18:10.575 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:18:10.576 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:10.574+0000 7f42022d4700 1 -- 172.21.15.73:0/328424293 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f41e802ff60 msgr2=0x7f41e8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:18:10.576 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:10.574+0000 7f42022d4700 1 --2- 172.21.15.73:0/328424293 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f41e802ff60 0x7f41e8032420 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f41f80096a0 tx=0x7f41f8006b40).stop 2023-12-01T22:18:10.576 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:10.574+0000 7f42022d4700 1 -- 172.21.15.73:0/328424293 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41fc0f51a0 msgr2=0x7f41fc109170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:18:10.576 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:10.574+0000 7f42022d4700 1 --2- 172.21.15.73:0/328424293 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41fc0f51a0 0x7f41fc109170 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7f41ec013510 tx=0x7f41ec00b040).stop 2023-12-01T22:18:10.576 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:10.574+0000 7f42022d4700 1 -- 172.21.15.73:0/328424293 shutdown_connections 2023-12-01T22:18:10.577 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:10.574+0000 7f42022d4700 1 --2- 172.21.15.73:0/328424293 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f41e802ff60 0x7f41e8032420 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:18:10.577 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:10.574+0000 7f42022d4700 1 --2- 172.21.15.73:0/328424293 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41fc0f51a0 0x7f41fc109170 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:18:10.577 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:10.574+0000 7f42022d4700 1 -- 172.21.15.73:0/328424293 >> 172.21.15.73:0/328424293 conn(0x7f41fc0f09a0 msgr2=0x7f41fc0f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:18:10.577 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:10.574+0000 7f42022d4700 1 -- 172.21.15.73:0/328424293 shutdown_connections 2023-12-01T22:18:10.577 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:10.574+0000 7f42022d4700 1 -- 172.21.15.73:0/328424293 wait complete. 2023-12-01T22:18:10.577 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:18:11.776 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:18:11 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:18:10.574199+0000 mon.smithi029 (mon.0) 278 : audit [DBG] from='client.? 172.21.15.73:0/328424293' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:18:11.776 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:18:11 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:18:10.686743+0000 mgr.smithi029.loxrfu (mgr.14164) 215 : cluster [DBG] pgmap v203: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:18:11.959 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:18:11.960 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:18:13.196 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:13.194+0000 7f61822dc700 1 -- 172.21.15.73:0/1138577764 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f617c0f5410 msgr2=0x7f617c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:18:13.197 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:13.194+0000 7f61822dc700 1 --2- 172.21.15.73:0/1138577764 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f617c0f5410 0x7f617c0f5830 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 rx=0x7f616c009fb0 tx=0x7f616c009b30).stop 2023-12-01T22:18:13.197 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:13.194+0000 7f61822dc700 1 -- 172.21.15.73:0/1138577764 shutdown_connections 2023-12-01T22:18:13.197 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:13.194+0000 7f61822dc700 1 --2- 172.21.15.73:0/1138577764 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f617c0f5410 0x7f617c0f5830 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:18:13.197 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:13.195+0000 7f61822dc700 1 -- 172.21.15.73:0/1138577764 >> 172.21.15.73:0/1138577764 conn(0x7f617c0f0b70 msgr2=0x7f617c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:18:13.197 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:13.195+0000 7f61822dc700 1 -- 172.21.15.73:0/1138577764 shutdown_connections 2023-12-01T22:18:13.197 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:13.195+0000 7f61822dc700 1 -- 172.21.15.73:0/1138577764 wait complete. 2023-12-01T22:18:13.197 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:13.195+0000 7f61822dc700 1 Processor -- start 2023-12-01T22:18:13.198 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:13.195+0000 7f61822dc700 1 -- start start 2023-12-01T22:18:13.198 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:13.195+0000 7f61822dc700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f617c0f5410 0x7f617c106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:18:13.198 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:13.195+0000 7f61822dc700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f617c106c10 con 0x7f617c0f5410 2023-12-01T22:18:13.198 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:13.196+0000 7f617bfff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f617c0f5410 0x7f617c106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:18:13.198 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:13.196+0000 7f617bfff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f617c0f5410 0x7f617c106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:39986/0 (socket says 172.21.15.73:39986) 2023-12-01T22:18:13.199 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:13.196+0000 7f617bfff700 1 -- 172.21.15.73:0/1503474479 learned_addr learned my addr 172.21.15.73:0/1503474479 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:18:13.199 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:13.196+0000 7f617bfff700 1 -- 172.21.15.73:0/1503474479 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f616c005040 con 0x7f617c0f5410 2023-12-01T22:18:13.199 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:13.196+0000 7f617bfff700 1 --2- 172.21.15.73:0/1503474479 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f617c0f5410 0x7f617c106720 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7f616c014000 tx=0x7f616c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:18:13.199 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:13.196+0000 7f6178ff9700 1 -- 172.21.15.73:0/1503474479 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f616c0095f0 con 0x7f617c0f5410 2023-12-01T22:18:13.199 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:13.196+0000 7f61822dc700 1 -- 172.21.15.73:0/1503474479 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f617c106e10 con 0x7f617c0f5410 2023-12-01T22:18:13.199 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:13.196+0000 7f6178ff9700 1 -- 172.21.15.73:0/1503474479 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f616c007e70 con 0x7f617c0f5410 2023-12-01T22:18:13.199 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:13.196+0000 7f6178ff9700 1 -- 172.21.15.73:0/1503474479 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f616c014ea0 con 0x7f617c0f5410 2023-12-01T22:18:13.199 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:13.196+0000 7f61822dc700 1 -- 172.21.15.73:0/1503474479 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f617c1072a0 con 0x7f617c0f5410 2023-12-01T22:18:13.200 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:13.197+0000 7f6178ff9700 1 -- 172.21.15.73:0/1503474479 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f616c025070 con 0x7f617c0f5410 2023-12-01T22:18:13.200 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:13.197+0000 7f61822dc700 1 -- 172.21.15.73:0/1503474479 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f617c045ad0 con 0x7f617c0f5410 2023-12-01T22:18:13.200 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:13.197+0000 7f6178ff9700 1 --2- 172.21.15.73:0/1503474479 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f616402ff60 0x7f6164032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:18:13.200 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:13.197+0000 7f6178ff9700 1 -- 172.21.15.73:0/1503474479 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f616c040930 con 0x7f617c0f5410 2023-12-01T22:18:13.200 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:13.198+0000 7f617b7fe700 1 --2- 172.21.15.73:0/1503474479 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f616402ff60 0x7f6164032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:18:13.200 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:13.198+0000 7f617b7fe700 1 --2- 172.21.15.73:0/1503474479 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f616402ff60 0x7f6164032420 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f61700096a0 tx=0x7f6170006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:18:13.204 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:13.202+0000 7f6178ff9700 1 -- 172.21.15.73:0/1503474479 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f616c00f4e0 con 0x7f617c0f5410 2023-12-01T22:18:13.409 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:13.407+0000 7f61822dc700 1 -- 172.21.15.73:0/1503474479 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f617c108150 con 0x7f617c0f5410 2023-12-01T22:18:13.410 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:13.408+0000 7f6178ff9700 1 -- 172.21.15.73:0/1503474479 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f616c00f300 con 0x7f617c0f5410 2023-12-01T22:18:13.410 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:18:13.410 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:18:13.411 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:13.409+0000 7f61822dc700 1 -- 172.21.15.73:0/1503474479 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f616402ff60 msgr2=0x7f6164032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:18:13.411 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:13.409+0000 7f61822dc700 1 --2- 172.21.15.73:0/1503474479 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f616402ff60 0x7f6164032420 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f61700096a0 tx=0x7f6170006b40).stop 2023-12-01T22:18:13.411 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:13.409+0000 7f61822dc700 1 -- 172.21.15.73:0/1503474479 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f617c0f5410 msgr2=0x7f617c106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:18:13.412 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:13.409+0000 7f61822dc700 1 --2- 172.21.15.73:0/1503474479 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f617c0f5410 0x7f617c106720 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7f616c014000 tx=0x7f616c00b040).stop 2023-12-01T22:18:13.412 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:13.410+0000 7f61822dc700 1 -- 172.21.15.73:0/1503474479 shutdown_connections 2023-12-01T22:18:13.412 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:13.410+0000 7f61822dc700 1 --2- 172.21.15.73:0/1503474479 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f616402ff60 0x7f6164032420 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:18:13.412 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:13.410+0000 7f61822dc700 1 --2- 172.21.15.73:0/1503474479 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f617c0f5410 0x7f617c106720 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:18:13.412 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:13.410+0000 7f61822dc700 1 -- 172.21.15.73:0/1503474479 >> 172.21.15.73:0/1503474479 conn(0x7f617c0f0b70 msgr2=0x7f617c05fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:18:13.412 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:13.410+0000 7f61822dc700 1 -- 172.21.15.73:0/1503474479 shutdown_connections 2023-12-01T22:18:13.412 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:13.410+0000 7f61822dc700 1 -- 172.21.15.73:0/1503474479 wait complete. 2023-12-01T22:18:13.412 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:18:14.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:18:13 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:18:12.687179+0000 mgr.smithi029.loxrfu (mgr.14164) 216 : cluster [DBG] pgmap v204: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:18:14.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:18:13 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:18:12.783127+0000 mon.smithi029 (mon.0) 279 : audit [INF] from='mgr.14164 172.21.15.29:0/3195927111' entity='mgr.smithi029.loxrfu' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi029.loxrfu/mirror_snapshot_schedule"}]: dispatch 2023-12-01T22:18:14.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:18:13 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:18:12.786316+0000 mon.smithi029 (mon.0) 280 : audit [INF] from='mgr.14164 172.21.15.29:0/3195927111' entity='mgr.smithi029.loxrfu' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi029.loxrfu/trash_purge_schedule"}]: dispatch 2023-12-01T22:18:14.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:18:13 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:18:13.409319+0000 mon.smithi029 (mon.0) 281 : audit [DBG] from='client.? 172.21.15.73:0/1503474479' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:18:14.818 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:18:14.819 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:18:16.009 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:16.007+0000 7f78ecadd700 1 -- 172.21.15.73:0/46933630 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78e80f51a0 msgr2=0x7f78e80f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:18:16.009 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:16.007+0000 7f78ecadd700 1 --2- 172.21.15.73:0/46933630 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78e80f51a0 0x7f78e80f55c0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 rx=0x7f78d8009fb0 tx=0x7f78d8009b30).stop 2023-12-01T22:18:16.009 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:16.007+0000 7f78ecadd700 1 -- 172.21.15.73:0/46933630 shutdown_connections 2023-12-01T22:18:16.009 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:16.007+0000 7f78ecadd700 1 --2- 172.21.15.73:0/46933630 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78e80f51a0 0x7f78e80f55c0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:18:16.009 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:16.007+0000 7f78ecadd700 1 -- 172.21.15.73:0/46933630 >> 172.21.15.73:0/46933630 conn(0x7f78e80f09a0 msgr2=0x7f78e80f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:18:16.009 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:16.007+0000 7f78ecadd700 1 -- 172.21.15.73:0/46933630 shutdown_connections 2023-12-01T22:18:16.009 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:16.007+0000 7f78ecadd700 1 -- 172.21.15.73:0/46933630 wait complete. 2023-12-01T22:18:16.011 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:16.008+0000 7f78ecadd700 1 Processor -- start 2023-12-01T22:18:16.011 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:16.008+0000 7f78ecadd700 1 -- start start 2023-12-01T22:18:16.011 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:16.008+0000 7f78ecadd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78e80f51a0 0x7f78e81053e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:18:16.011 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:16.008+0000 7f78ecadd700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78e81058d0 con 0x7f78e80f51a0 2023-12-01T22:18:16.011 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:16.008+0000 7f78e77fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78e80f51a0 0x7f78e81053e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:18:16.011 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:16.008+0000 7f78e77fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78e80f51a0 0x7f78e81053e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:39430/0 (socket says 172.21.15.73:39430) 2023-12-01T22:18:16.011 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:16.008+0000 7f78e77fe700 1 -- 172.21.15.73:0/1707385418 learned_addr learned my addr 172.21.15.73:0/1707385418 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:18:16.012 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:16.009+0000 7f78e77fe700 1 -- 172.21.15.73:0/1707385418 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78d8005040 con 0x7f78e80f51a0 2023-12-01T22:18:16.012 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:16.009+0000 7f78e77fe700 1 --2- 172.21.15.73:0/1707385418 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78e80f51a0 0x7f78e81053e0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7f78d8013510 tx=0x7f78d800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:18:16.012 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:16.009+0000 7f78cffff700 1 -- 172.21.15.73:0/1707385418 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f78d8009320 con 0x7f78e80f51a0 2023-12-01T22:18:16.012 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:16.009+0000 7f78ecadd700 1 -- 172.21.15.73:0/1707385418 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78e8105ad0 con 0x7f78e80f51a0 2023-12-01T22:18:16.012 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:16.009+0000 7f78cffff700 1 -- 172.21.15.73:0/1707385418 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f78d8013f00 con 0x7f78e80f51a0 2023-12-01T22:18:16.012 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:16.009+0000 7f78ecadd700 1 -- 172.21.15.73:0/1707385418 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78e8105f60 con 0x7f78e80f51a0 2023-12-01T22:18:16.012 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:16.009+0000 7f78cffff700 1 -- 172.21.15.73:0/1707385418 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f78d8014dc0 con 0x7f78e80f51a0 2023-12-01T22:18:16.013 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:16.010+0000 7f78cffff700 1 -- 172.21.15.73:0/1707385418 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f78d80095f0 con 0x7f78e80f51a0 2023-12-01T22:18:16.013 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:16.010+0000 7f78ecadd700 1 -- 172.21.15.73:0/1707385418 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78e8045ad0 con 0x7f78e80f51a0 2023-12-01T22:18:16.013 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:16.010+0000 7f78cffff700 1 --2- 172.21.15.73:0/1707385418 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f78d002ff60 0x7f78d0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:18:16.013 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:16.010+0000 7f78cffff700 1 -- 172.21.15.73:0/1707385418 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f78d8042e10 con 0x7f78e80f51a0 2023-12-01T22:18:16.013 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:16.010+0000 7f78e6ffd700 1 --2- 172.21.15.73:0/1707385418 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f78d002ff60 0x7f78d0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:18:16.013 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:16.011+0000 7f78e6ffd700 1 --2- 172.21.15.73:0/1707385418 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f78d002ff60 0x7f78d0032420 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f78dc0096a0 tx=0x7f78dc006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:18:16.016 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:16.015+0000 7f78cffff700 1 -- 172.21.15.73:0/1707385418 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f78d8014420 con 0x7f78e80f51a0 2023-12-01T22:18:16.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:18:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:18:14.687488+0000 mgr.smithi029.loxrfu (mgr.14164) 217 : cluster [DBG] pgmap v205: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:18:16.221 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:16.219+0000 7f78ecadd700 1 -- 172.21.15.73:0/1707385418 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f78e8106db0 con 0x7f78e80f51a0 2023-12-01T22:18:16.221 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:16.220+0000 7f78cffff700 1 -- 172.21.15.73:0/1707385418 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f78d801f980 con 0x7f78e80f51a0 2023-12-01T22:18:16.222 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:18:16.222 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:18:16.223 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:16.221+0000 7f78ecadd700 1 -- 172.21.15.73:0/1707385418 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f78d002ff60 msgr2=0x7f78d0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:18:16.223 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:16.221+0000 7f78ecadd700 1 --2- 172.21.15.73:0/1707385418 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f78d002ff60 0x7f78d0032420 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f78dc0096a0 tx=0x7f78dc006b40).stop 2023-12-01T22:18:16.223 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:16.221+0000 7f78ecadd700 1 -- 172.21.15.73:0/1707385418 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78e80f51a0 msgr2=0x7f78e81053e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:18:16.223 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:16.221+0000 7f78ecadd700 1 --2- 172.21.15.73:0/1707385418 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78e80f51a0 0x7f78e81053e0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7f78d8013510 tx=0x7f78d800b040).stop 2023-12-01T22:18:16.224 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:16.221+0000 7f78ecadd700 1 -- 172.21.15.73:0/1707385418 shutdown_connections 2023-12-01T22:18:16.224 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:16.221+0000 7f78ecadd700 1 --2- 172.21.15.73:0/1707385418 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f78d002ff60 0x7f78d0032420 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:18:16.224 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:16.221+0000 7f78ecadd700 1 --2- 172.21.15.73:0/1707385418 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78e80f51a0 0x7f78e81053e0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:18:16.224 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:16.221+0000 7f78ecadd700 1 -- 172.21.15.73:0/1707385418 >> 172.21.15.73:0/1707385418 conn(0x7f78e80f09a0 msgr2=0x7f78e80f13e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:18:16.224 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:16.222+0000 7f78ecadd700 1 -- 172.21.15.73:0/1707385418 shutdown_connections 2023-12-01T22:18:16.224 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:16.222+0000 7f78ecadd700 1 -- 172.21.15.73:0/1707385418 wait complete. 2023-12-01T22:18:16.224 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:18:17.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:18:16 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:18:16.221217+0000 mon.smithi029 (mon.0) 282 : audit [DBG] from='client.? 172.21.15.73:0/1707385418' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:18:17.608 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:18:17.609 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:18:18.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:18:17 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:18:16.687795+0000 mgr.smithi029.loxrfu (mgr.14164) 218 : cluster [DBG] pgmap v206: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:18:18.847 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:18.843+0000 7fc99e0f2700 1 -- 172.21.15.73:0/1923651871 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc9980f51a0 msgr2=0x7fc9980f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:18:18.847 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:18.843+0000 7fc99e0f2700 1 --2- 172.21.15.73:0/1923651871 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc9980f51a0 0x7fc9980f55c0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 rx=0x7fc988009fb0 tx=0x7fc988009b30).stop 2023-12-01T22:18:18.847 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:18.844+0000 7fc99e0f2700 1 -- 172.21.15.73:0/1923651871 shutdown_connections 2023-12-01T22:18:18.847 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:18.844+0000 7fc99e0f2700 1 --2- 172.21.15.73:0/1923651871 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc9980f51a0 0x7fc9980f55c0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:18:18.847 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:18.844+0000 7fc99e0f2700 1 -- 172.21.15.73:0/1923651871 >> 172.21.15.73:0/1923651871 conn(0x7fc9980f09a0 msgr2=0x7fc9980f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:18:18.847 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:18.844+0000 7fc99e0f2700 1 -- 172.21.15.73:0/1923651871 shutdown_connections 2023-12-01T22:18:18.847 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:18.844+0000 7fc99e0f2700 1 -- 172.21.15.73:0/1923651871 wait complete. 2023-12-01T22:18:18.847 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:18.844+0000 7fc99e0f2700 1 Processor -- start 2023-12-01T22:18:18.848 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:18.844+0000 7fc99e0f2700 1 -- start start 2023-12-01T22:18:18.848 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:18.844+0000 7fc99e0f2700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc9980f51a0 0x7fc9981053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:18:18.848 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:18.844+0000 7fc99e0f2700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9981058a0 con 0x7fc9980f51a0 2023-12-01T22:18:18.848 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:18.844+0000 7fc99d0f0700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc9980f51a0 0x7fc9981053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:18:18.848 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:18.844+0000 7fc99d0f0700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc9980f51a0 0x7fc9981053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:39456/0 (socket says 172.21.15.73:39456) 2023-12-01T22:18:18.848 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:18.845+0000 7fc99d0f0700 1 -- 172.21.15.73:0/524848358 learned_addr learned my addr 172.21.15.73:0/524848358 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:18:18.848 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:18.845+0000 7fc99d0f0700 1 -- 172.21.15.73:0/524848358 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc988005040 con 0x7fc9980f51a0 2023-12-01T22:18:18.848 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:18.845+0000 7fc99d0f0700 1 --2- 172.21.15.73:0/524848358 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc9980f51a0 0x7fc9981053b0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7fc988013510 tx=0x7fc98800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:18:18.848 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:18.845+0000 7fc98dffb700 1 -- 172.21.15.73:0/524848358 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc988009320 con 0x7fc9980f51a0 2023-12-01T22:18:18.848 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:18.845+0000 7fc99e0f2700 1 -- 172.21.15.73:0/524848358 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc998105aa0 con 0x7fc9980f51a0 2023-12-01T22:18:18.848 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:18.845+0000 7fc98dffb700 1 -- 172.21.15.73:0/524848358 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc988013f00 con 0x7fc9980f51a0 2023-12-01T22:18:18.848 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:18.845+0000 7fc98dffb700 1 -- 172.21.15.73:0/524848358 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc988014dc0 con 0x7fc9980f51a0 2023-12-01T22:18:18.848 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:18.845+0000 7fc99e0f2700 1 -- 172.21.15.73:0/524848358 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc998105f30 con 0x7fc9980f51a0 2023-12-01T22:18:18.849 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:18.846+0000 7fc98dffb700 1 -- 172.21.15.73:0/524848358 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc98801d3d0 con 0x7fc9980f51a0 2023-12-01T22:18:18.849 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:18.846+0000 7fc98dffb700 1 --2- 172.21.15.73:0/524848358 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fc98402ff60 0x7fc984032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:18:18.849 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:18.846+0000 7fc98dffb700 1 -- 172.21.15.73:0/524848358 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc9880413c0 con 0x7fc9980f51a0 2023-12-01T22:18:18.850 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:18.846+0000 7fc99e0f2700 1 -- 172.21.15.73:0/524848358 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc998045ad0 con 0x7fc9980f51a0 2023-12-01T22:18:18.850 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:18.846+0000 7fc99c8ef700 1 --2- 172.21.15.73:0/524848358 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fc98402ff60 0x7fc984032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:18:18.850 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:18.847+0000 7fc99c8ef700 1 --2- 172.21.15.73:0/524848358 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fc98402ff60 0x7fc984032420 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7fc9940096a0 tx=0x7fc994006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:18:18.853 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:18.851+0000 7fc98dffb700 1 -- 172.21.15.73:0/524848358 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc988012920 con 0x7fc9980f51a0 2023-12-01T22:18:19.058 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:19.056+0000 7fc99e0f2700 1 -- 172.21.15.73:0/524848358 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc9980facd0 con 0x7fc9980f51a0 2023-12-01T22:18:19.058 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:19.057+0000 7fc98dffb700 1 -- 172.21.15.73:0/524848358 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc98802b030 con 0x7fc9980f51a0 2023-12-01T22:18:19.058 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:18:19.059 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:18:19.060 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:19.058+0000 7fc99e0f2700 1 -- 172.21.15.73:0/524848358 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fc98402ff60 msgr2=0x7fc984032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:18:19.060 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:19.058+0000 7fc99e0f2700 1 --2- 172.21.15.73:0/524848358 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fc98402ff60 0x7fc984032420 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7fc9940096a0 tx=0x7fc994006b40).stop 2023-12-01T22:18:19.060 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:19.058+0000 7fc99e0f2700 1 -- 172.21.15.73:0/524848358 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc9980f51a0 msgr2=0x7fc9981053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:18:19.060 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:19.058+0000 7fc99e0f2700 1 --2- 172.21.15.73:0/524848358 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc9980f51a0 0x7fc9981053b0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7fc988013510 tx=0x7fc98800b040).stop 2023-12-01T22:18:19.060 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:19.058+0000 7fc99e0f2700 1 -- 172.21.15.73:0/524848358 shutdown_connections 2023-12-01T22:18:19.060 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:19.058+0000 7fc99e0f2700 1 --2- 172.21.15.73:0/524848358 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fc98402ff60 0x7fc984032420 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:18:19.060 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:19.058+0000 7fc99e0f2700 1 --2- 172.21.15.73:0/524848358 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc9980f51a0 0x7fc9981053b0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:18:19.061 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:19.058+0000 7fc99e0f2700 1 -- 172.21.15.73:0/524848358 >> 172.21.15.73:0/524848358 conn(0x7fc9980f09a0 msgr2=0x7fc9980f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:18:19.061 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:19.058+0000 7fc99e0f2700 1 -- 172.21.15.73:0/524848358 shutdown_connections 2023-12-01T22:18:19.061 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:19.058+0000 7fc99e0f2700 1 -- 172.21.15.73:0/524848358 wait complete. 2023-12-01T22:18:19.061 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:18:20.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:18:19 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:18:18.688196+0000 mgr.smithi029.loxrfu (mgr.14164) 219 : cluster [DBG] pgmap v207: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:18:20.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:18:19 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:18:19.058096+0000 mon.smithi029 (mon.0) 283 : audit [DBG] from='client.? 172.21.15.73:0/524848358' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:18:20.463 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:18:20.464 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:18:21.627 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:21.625+0000 7f8715e14700 1 -- 172.21.15.73:0/4078276200 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8710060280 msgr2=0x7f87100f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:18:21.628 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:21.625+0000 7f8715e14700 1 --2- 172.21.15.73:0/4078276200 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8710060280 0x7f87100f90f0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 rx=0x7f8700009fb0 tx=0x7f8700009b30).stop 2023-12-01T22:18:21.628 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:21.625+0000 7f8715e14700 1 -- 172.21.15.73:0/4078276200 shutdown_connections 2023-12-01T22:18:21.628 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:21.625+0000 7f8715e14700 1 --2- 172.21.15.73:0/4078276200 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8710060280 0x7f87100f90f0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:18:21.628 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:21.625+0000 7f8715e14700 1 -- 172.21.15.73:0/4078276200 >> 172.21.15.73:0/4078276200 conn(0x7f87100f0b70 msgr2=0x7f87100f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:18:21.628 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:21.625+0000 7f8715e14700 1 -- 172.21.15.73:0/4078276200 shutdown_connections 2023-12-01T22:18:21.628 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:21.625+0000 7f8715e14700 1 -- 172.21.15.73:0/4078276200 wait complete. 2023-12-01T22:18:21.628 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:21.625+0000 7f8715e14700 1 Processor -- start 2023-12-01T22:18:21.628 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:21.626+0000 7f8715e14700 1 -- start start 2023-12-01T22:18:21.628 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:21.626+0000 7f8715e14700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8710060280 0x7f87101066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:18:21.628 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:21.626+0000 7f8715e14700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8710106bd0 con 0x7f8710060280 2023-12-01T22:18:21.630 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:21.626+0000 7f870f7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8710060280 0x7f87101066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:18:21.630 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:21.626+0000 7f870f7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8710060280 0x7f87101066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:39480/0 (socket says 172.21.15.73:39480) 2023-12-01T22:18:21.630 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:21.626+0000 7f870f7fe700 1 -- 172.21.15.73:0/2220652510 learned_addr learned my addr 172.21.15.73:0/2220652510 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:18:21.630 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:21.626+0000 7f870f7fe700 1 -- 172.21.15.73:0/2220652510 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8700005040 con 0x7f8710060280 2023-12-01T22:18:21.630 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:21.627+0000 7f870f7fe700 1 --2- 172.21.15.73:0/2220652510 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8710060280 0x7f87101066e0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7f8700013510 tx=0x7f870000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:18:21.630 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:21.627+0000 7f8707fff700 1 -- 172.21.15.73:0/2220652510 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8700009320 con 0x7f8710060280 2023-12-01T22:18:21.630 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:21.627+0000 7f8715e14700 1 -- 172.21.15.73:0/2220652510 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8710106dd0 con 0x7f8710060280 2023-12-01T22:18:21.630 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:21.627+0000 7f8707fff700 1 -- 172.21.15.73:0/2220652510 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8700013f00 con 0x7f8710060280 2023-12-01T22:18:21.631 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:21.627+0000 7f8707fff700 1 -- 172.21.15.73:0/2220652510 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8700014dc0 con 0x7f8710060280 2023-12-01T22:18:21.631 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:21.627+0000 7f8715e14700 1 -- 172.21.15.73:0/2220652510 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8710107260 con 0x7f8710060280 2023-12-01T22:18:21.631 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:21.628+0000 7f8707fff700 1 -- 172.21.15.73:0/2220652510 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f87000095f0 con 0x7f8710060280 2023-12-01T22:18:21.631 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:21.628+0000 7f8715e14700 1 -- 172.21.15.73:0/2220652510 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8710045ad0 con 0x7f8710060280 2023-12-01T22:18:21.631 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:21.628+0000 7f8707fff700 1 --2- 172.21.15.73:0/2220652510 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f86fc02ff60 0x7f86fc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:18:21.631 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:21.628+0000 7f8707fff700 1 -- 172.21.15.73:0/2220652510 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8700041000 con 0x7f8710060280 2023-12-01T22:18:21.631 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:21.628+0000 7f870effd700 1 --2- 172.21.15.73:0/2220652510 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f86fc02ff60 0x7f86fc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:18:21.631 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:21.628+0000 7f870effd700 1 --2- 172.21.15.73:0/2220652510 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f86fc02ff60 0x7f86fc032420 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f86f80096a0 tx=0x7f86f8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:18:21.634 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:21.633+0000 7f8707fff700 1 -- 172.21.15.73:0/2220652510 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8700014420 con 0x7f8710060280 2023-12-01T22:18:21.776 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:18:21 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:18:20.688499+0000 mgr.smithi029.loxrfu (mgr.14164) 220 : cluster [DBG] pgmap v208: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:18:21.841 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:21.839+0000 7f8715e14700 1 -- 172.21.15.73:0/2220652510 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8710108110 con 0x7f8710060280 2023-12-01T22:18:21.841 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:21.841+0000 7f8707fff700 1 -- 172.21.15.73:0/2220652510 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8700010230 con 0x7f8710060280 2023-12-01T22:18:21.842 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:18:21.842 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:18:21.843 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:21.841+0000 7f8715e14700 1 -- 172.21.15.73:0/2220652510 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f86fc02ff60 msgr2=0x7f86fc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:18:21.843 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:21.841+0000 7f8715e14700 1 --2- 172.21.15.73:0/2220652510 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f86fc02ff60 0x7f86fc032420 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f86f80096a0 tx=0x7f86f8006b40).stop 2023-12-01T22:18:21.843 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:21.841+0000 7f8715e14700 1 -- 172.21.15.73:0/2220652510 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8710060280 msgr2=0x7f87101066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:18:21.843 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:21.841+0000 7f8715e14700 1 --2- 172.21.15.73:0/2220652510 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8710060280 0x7f87101066e0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7f8700013510 tx=0x7f870000b040).stop 2023-12-01T22:18:21.844 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:21.841+0000 7f8715e14700 1 -- 172.21.15.73:0/2220652510 shutdown_connections 2023-12-01T22:18:21.844 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:21.841+0000 7f8715e14700 1 --2- 172.21.15.73:0/2220652510 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f86fc02ff60 0x7f86fc032420 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:18:21.844 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:21.841+0000 7f8715e14700 1 --2- 172.21.15.73:0/2220652510 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8710060280 0x7f87101066e0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:18:21.844 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:21.841+0000 7f8715e14700 1 -- 172.21.15.73:0/2220652510 >> 172.21.15.73:0/2220652510 conn(0x7f87100f0b70 msgr2=0x7f87100f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:18:21.844 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:21.842+0000 7f8715e14700 1 -- 172.21.15.73:0/2220652510 shutdown_connections 2023-12-01T22:18:21.844 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:21.842+0000 7f8715e14700 1 -- 172.21.15.73:0/2220652510 wait complete. 2023-12-01T22:18:21.844 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:18:22.776 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:18:22 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:18:21.841310+0000 mon.smithi029 (mon.0) 284 : audit [DBG] from='client.? 172.21.15.73:0/2220652510' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:18:23.229 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:18:23.229 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:18:23.776 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:18:23 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:18:22.688872+0000 mgr.smithi029.loxrfu (mgr.14164) 221 : cluster [DBG] pgmap v209: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:18:24.433 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:24.431+0000 7f56ba00c700 1 -- 172.21.15.73:0/3948334642 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56b40f5410 msgr2=0x7f56b40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:18:24.433 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:24.431+0000 7f56ba00c700 1 --2- 172.21.15.73:0/3948334642 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56b40f5410 0x7f56b40f5830 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 rx=0x7f56a4009fb0 tx=0x7f56a4009b30).stop 2023-12-01T22:18:24.433 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:24.431+0000 7f56ba00c700 1 -- 172.21.15.73:0/3948334642 shutdown_connections 2023-12-01T22:18:24.433 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:24.431+0000 7f56ba00c700 1 --2- 172.21.15.73:0/3948334642 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56b40f5410 0x7f56b40f5830 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:18:24.433 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:24.431+0000 7f56ba00c700 1 -- 172.21.15.73:0/3948334642 >> 172.21.15.73:0/3948334642 conn(0x7f56b40f0b70 msgr2=0x7f56b40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:18:24.434 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:24.431+0000 7f56ba00c700 1 -- 172.21.15.73:0/3948334642 shutdown_connections 2023-12-01T22:18:24.434 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:24.431+0000 7f56ba00c700 1 -- 172.21.15.73:0/3948334642 wait complete. 2023-12-01T22:18:24.434 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:24.432+0000 7f56ba00c700 1 Processor -- start 2023-12-01T22:18:24.434 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:24.432+0000 7f56ba00c700 1 -- start start 2023-12-01T22:18:24.434 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:24.432+0000 7f56ba00c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56b40f5410 0x7f56b4105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:18:24.434 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:24.432+0000 7f56ba00c700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56b4105b20 con 0x7f56b40f5410 2023-12-01T22:18:24.435 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:24.433+0000 7f56b37fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56b40f5410 0x7f56b4105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:18:24.435 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:24.433+0000 7f56b37fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56b40f5410 0x7f56b4105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:34968/0 (socket says 172.21.15.73:34968) 2023-12-01T22:18:24.435 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:24.433+0000 7f56b37fe700 1 -- 172.21.15.73:0/504566979 learned_addr learned my addr 172.21.15.73:0/504566979 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:18:24.436 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:24.434+0000 7f56b37fe700 1 -- 172.21.15.73:0/504566979 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56a4005040 con 0x7f56b40f5410 2023-12-01T22:18:24.436 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:24.434+0000 7f56b37fe700 1 --2- 172.21.15.73:0/504566979 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56b40f5410 0x7f56b4105630 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7f56a4013510 tx=0x7f56a400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:18:24.436 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:24.434+0000 7f569bfff700 1 -- 172.21.15.73:0/504566979 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f56a4009320 con 0x7f56b40f5410 2023-12-01T22:18:24.436 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:24.434+0000 7f56ba00c700 1 -- 172.21.15.73:0/504566979 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56b4105d20 con 0x7f56b40f5410 2023-12-01T22:18:24.436 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:24.434+0000 7f569bfff700 1 -- 172.21.15.73:0/504566979 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f56a4013f00 con 0x7f56b40f5410 2023-12-01T22:18:24.436 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:24.434+0000 7f569bfff700 1 -- 172.21.15.73:0/504566979 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f56a4014d90 con 0x7f56b40f5410 2023-12-01T22:18:24.436 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:24.435+0000 7f56ba00c700 1 -- 172.21.15.73:0/504566979 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56b41061b0 con 0x7f56b40f5410 2023-12-01T22:18:24.437 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:24.435+0000 7f569bfff700 1 -- 172.21.15.73:0/504566979 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f56a40263a0 con 0x7f56b40f5410 2023-12-01T22:18:24.437 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:24.435+0000 7f56ba00c700 1 -- 172.21.15.73:0/504566979 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56b4045ad0 con 0x7f56b40f5410 2023-12-01T22:18:24.437 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:24.435+0000 7f569bfff700 1 --2- 172.21.15.73:0/504566979 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f569c02ff60 0x7f569c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:18:24.437 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:24.436+0000 7f569bfff700 1 -- 172.21.15.73:0/504566979 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f56a40410a0 con 0x7f56b40f5410 2023-12-01T22:18:24.437 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:24.436+0000 7f56b2ffd700 1 --2- 172.21.15.73:0/504566979 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f569c02ff60 0x7f569c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:18:24.438 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:24.436+0000 7f56b2ffd700 1 --2- 172.21.15.73:0/504566979 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f569c02ff60 0x7f569c032420 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f56a80096a0 tx=0x7f56a8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:18:24.442 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:24.440+0000 7f569bfff700 1 -- 172.21.15.73:0/504566979 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f56a40126f0 con 0x7f56b40f5410 2023-12-01T22:18:24.645 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:24.644+0000 7f56ba00c700 1 -- 172.21.15.73:0/504566979 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f56b40fae60 con 0x7f56b40f5410 2023-12-01T22:18:24.646 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:24.645+0000 7f569bfff700 1 -- 172.21.15.73:0/504566979 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f56a4012330 con 0x7f56b40f5410 2023-12-01T22:18:24.646 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:18:24.647 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:18:24.648 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:24.646+0000 7f56ba00c700 1 -- 172.21.15.73:0/504566979 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f569c02ff60 msgr2=0x7f569c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:18:24.648 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:24.646+0000 7f56ba00c700 1 --2- 172.21.15.73:0/504566979 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f569c02ff60 0x7f569c032420 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f56a80096a0 tx=0x7f56a8006b40).stop 2023-12-01T22:18:24.648 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:24.646+0000 7f56ba00c700 1 -- 172.21.15.73:0/504566979 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56b40f5410 msgr2=0x7f56b4105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:18:24.648 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:24.646+0000 7f56ba00c700 1 --2- 172.21.15.73:0/504566979 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56b40f5410 0x7f56b4105630 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7f56a4013510 tx=0x7f56a400b040).stop 2023-12-01T22:18:24.648 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:24.646+0000 7f56ba00c700 1 -- 172.21.15.73:0/504566979 shutdown_connections 2023-12-01T22:18:24.648 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:24.646+0000 7f56ba00c700 1 --2- 172.21.15.73:0/504566979 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f569c02ff60 0x7f569c032420 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:18:24.648 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:24.646+0000 7f56ba00c700 1 --2- 172.21.15.73:0/504566979 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56b40f5410 0x7f56b4105630 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:18:24.648 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:24.646+0000 7f56ba00c700 1 -- 172.21.15.73:0/504566979 >> 172.21.15.73:0/504566979 conn(0x7f56b40f0b70 msgr2=0x7f56b40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:18:24.649 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:24.646+0000 7f56ba00c700 1 -- 172.21.15.73:0/504566979 shutdown_connections 2023-12-01T22:18:24.649 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:24.646+0000 7f56ba00c700 1 -- 172.21.15.73:0/504566979 wait complete. 2023-12-01T22:18:24.649 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:18:25.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:18:24 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:18:24.645966+0000 mon.smithi029 (mon.0) 285 : audit [DBG] from='client.? 172.21.15.73:0/504566979' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:18:26.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:18:25 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:18:24.689250+0000 mgr.smithi029.loxrfu (mgr.14164) 222 : cluster [DBG] pgmap v210: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:18:26.033 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:18:26.033 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:18:27.251 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:27.248+0000 7f84403b3700 1 -- 172.21.15.73:0/2338715951 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84380f5410 msgr2=0x7f84380f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:18:27.251 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:27.248+0000 7f84403b3700 1 --2- 172.21.15.73:0/2338715951 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84380f5410 0x7f84380f5830 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 rx=0x7f8434009fb0 tx=0x7f8434009b30).stop 2023-12-01T22:18:27.251 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:27.248+0000 7f84403b3700 1 -- 172.21.15.73:0/2338715951 shutdown_connections 2023-12-01T22:18:27.251 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:27.248+0000 7f84403b3700 1 --2- 172.21.15.73:0/2338715951 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84380f5410 0x7f84380f5830 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:18:27.251 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:27.248+0000 7f84403b3700 1 -- 172.21.15.73:0/2338715951 >> 172.21.15.73:0/2338715951 conn(0x7f84380f0b70 msgr2=0x7f84380f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:18:27.251 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:27.248+0000 7f84403b3700 1 -- 172.21.15.73:0/2338715951 shutdown_connections 2023-12-01T22:18:27.252 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:27.248+0000 7f84403b3700 1 -- 172.21.15.73:0/2338715951 wait complete. 2023-12-01T22:18:27.252 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:27.249+0000 7f84403b3700 1 Processor -- start 2023-12-01T22:18:27.252 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:27.249+0000 7f84403b3700 1 -- start start 2023-12-01T22:18:27.252 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:27.249+0000 7f84403b3700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84380f5410 0x7f8438106710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:18:27.252 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:27.249+0000 7f84403b3700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8438106c00 con 0x7f84380f5410 2023-12-01T22:18:27.252 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:27.249+0000 7f843e14f700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84380f5410 0x7f8438106710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:18:27.252 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:27.249+0000 7f843e14f700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84380f5410 0x7f8438106710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:34994/0 (socket says 172.21.15.73:34994) 2023-12-01T22:18:27.252 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:27.249+0000 7f843e14f700 1 -- 172.21.15.73:0/1392964214 learned_addr learned my addr 172.21.15.73:0/1392964214 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:18:27.252 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:27.250+0000 7f843e14f700 1 -- 172.21.15.73:0/1392964214 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8434005040 con 0x7f84380f5410 2023-12-01T22:18:27.252 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:27.250+0000 7f843e14f700 1 --2- 172.21.15.73:0/1392964214 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84380f5410 0x7f8438106710 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7f8434013510 tx=0x7f843400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:18:27.253 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:27.250+0000 7f842effd700 1 -- 172.21.15.73:0/1392964214 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8434009320 con 0x7f84380f5410 2023-12-01T22:18:27.253 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:27.250+0000 7f84403b3700 1 -- 172.21.15.73:0/1392964214 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8438106e00 con 0x7f84380f5410 2023-12-01T22:18:27.253 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:27.250+0000 7f842effd700 1 -- 172.21.15.73:0/1392964214 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8434013f00 con 0x7f84380f5410 2023-12-01T22:18:27.254 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:27.250+0000 7f842effd700 1 -- 172.21.15.73:0/1392964214 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8434014d90 con 0x7f84380f5410 2023-12-01T22:18:27.254 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:27.250+0000 7f84403b3700 1 -- 172.21.15.73:0/1392964214 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8438107290 con 0x7f84380f5410 2023-12-01T22:18:27.254 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:27.251+0000 7f84403b3700 1 -- 172.21.15.73:0/1392964214 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8438045ad0 con 0x7f84380f5410 2023-12-01T22:18:27.254 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:27.251+0000 7f842effd700 1 -- 172.21.15.73:0/1392964214 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f84340095f0 con 0x7f84380f5410 2023-12-01T22:18:27.254 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:27.251+0000 7f842effd700 1 --2- 172.21.15.73:0/1392964214 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f842402ff60 0x7f8424032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:18:27.254 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:27.251+0000 7f842effd700 1 -- 172.21.15.73:0/1392964214 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f84340410a0 con 0x7f84380f5410 2023-12-01T22:18:27.254 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:27.251+0000 7f843d94e700 1 --2- 172.21.15.73:0/1392964214 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f842402ff60 0x7f8424032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:18:27.254 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:27.251+0000 7f843d94e700 1 --2- 172.21.15.73:0/1392964214 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f842402ff60 0x7f8424032420 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f84280096a0 tx=0x7f8428006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:18:27.257 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:27.256+0000 7f842effd700 1 -- 172.21.15.73:0/1392964214 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f843401d7f0 con 0x7f84380f5410 2023-12-01T22:18:27.462 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:27.461+0000 7f84403b3700 1 -- 172.21.15.73:0/1392964214 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f84380fae60 con 0x7f84380f5410 2023-12-01T22:18:27.463 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:27.461+0000 7f842effd700 1 -- 172.21.15.73:0/1392964214 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8434010230 con 0x7f84380f5410 2023-12-01T22:18:27.463 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:18:27.463 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:18:27.465 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:27.463+0000 7f84403b3700 1 -- 172.21.15.73:0/1392964214 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f842402ff60 msgr2=0x7f8424032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:18:27.465 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:27.463+0000 7f84403b3700 1 --2- 172.21.15.73:0/1392964214 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f842402ff60 0x7f8424032420 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f84280096a0 tx=0x7f8428006b40).stop 2023-12-01T22:18:27.465 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:27.463+0000 7f84403b3700 1 -- 172.21.15.73:0/1392964214 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84380f5410 msgr2=0x7f8438106710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:18:27.465 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:27.463+0000 7f84403b3700 1 --2- 172.21.15.73:0/1392964214 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84380f5410 0x7f8438106710 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7f8434013510 tx=0x7f843400b040).stop 2023-12-01T22:18:27.465 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:27.463+0000 7f84403b3700 1 -- 172.21.15.73:0/1392964214 shutdown_connections 2023-12-01T22:18:27.465 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:27.463+0000 7f84403b3700 1 --2- 172.21.15.73:0/1392964214 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f842402ff60 0x7f8424032420 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:18:27.465 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:27.463+0000 7f84403b3700 1 --2- 172.21.15.73:0/1392964214 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84380f5410 0x7f8438106710 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:18:27.465 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:27.463+0000 7f84403b3700 1 -- 172.21.15.73:0/1392964214 >> 172.21.15.73:0/1392964214 conn(0x7f84380f0b70 msgr2=0x7f843805fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:18:27.465 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:27.463+0000 7f84403b3700 1 -- 172.21.15.73:0/1392964214 shutdown_connections 2023-12-01T22:18:27.465 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:27.463+0000 7f84403b3700 1 -- 172.21.15.73:0/1392964214 wait complete. 2023-12-01T22:18:27.466 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:18:28.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:18:27 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:18:26.689617+0000 mgr.smithi029.loxrfu (mgr.14164) 223 : cluster [DBG] pgmap v211: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:18:28.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:18:27 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:18:27.462886+0000 mon.smithi029 (mon.0) 286 : audit [DBG] from='client.? 172.21.15.73:0/1392964214' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:18:28.851 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:18:28.851 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:18:30.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:18:29 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:18:28.689921+0000 mgr.smithi029.loxrfu (mgr.14164) 224 : cluster [DBG] pgmap v212: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:18:30.178 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:30.171+0000 7fdf6c3ba700 1 -- 172.21.15.73:0/2178158724 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf640f5410 msgr2=0x7fdf640f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:18:30.178 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:30.171+0000 7fdf6c3ba700 1 --2- 172.21.15.73:0/2178158724 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf640f5410 0x7fdf640f5830 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 rx=0x7fdf54009fb0 tx=0x7fdf54009b30).stop 2023-12-01T22:18:30.178 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:30.171+0000 7fdf6c3ba700 1 -- 172.21.15.73:0/2178158724 shutdown_connections 2023-12-01T22:18:30.178 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:30.171+0000 7fdf6c3ba700 1 --2- 172.21.15.73:0/2178158724 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf640f5410 0x7fdf640f5830 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:18:30.179 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:30.171+0000 7fdf6c3ba700 1 -- 172.21.15.73:0/2178158724 >> 172.21.15.73:0/2178158724 conn(0x7fdf640f0b70 msgr2=0x7fdf640f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:18:30.179 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:30.171+0000 7fdf6c3ba700 1 -- 172.21.15.73:0/2178158724 shutdown_connections 2023-12-01T22:18:30.179 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:30.171+0000 7fdf6c3ba700 1 -- 172.21.15.73:0/2178158724 wait complete. 2023-12-01T22:18:30.179 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:30.172+0000 7fdf6c3ba700 1 Processor -- start 2023-12-01T22:18:30.179 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:30.172+0000 7fdf6c3ba700 1 -- start start 2023-12-01T22:18:30.179 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:30.172+0000 7fdf6c3ba700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf640f5410 0x7fdf64109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:18:30.179 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:30.172+0000 7fdf6c3ba700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf64060220 con 0x7fdf640f5410 2023-12-01T22:18:30.180 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:30.172+0000 7fdf6a156700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf640f5410 0x7fdf64109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:18:30.180 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:30.172+0000 7fdf6a156700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf640f5410 0x7fdf64109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:35018/0 (socket says 172.21.15.73:35018) 2023-12-01T22:18:30.180 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:30.172+0000 7fdf6a156700 1 -- 172.21.15.73:0/3997471969 learned_addr learned my addr 172.21.15.73:0/3997471969 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:18:30.180 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:30.172+0000 7fdf6a156700 1 -- 172.21.15.73:0/3997471969 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf54005040 con 0x7fdf640f5410 2023-12-01T22:18:30.180 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:30.172+0000 7fdf6a156700 1 --2- 172.21.15.73:0/3997471969 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf640f5410 0x7fdf64109450 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7fdf54013510 tx=0x7fdf5400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:18:30.180 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:30.173+0000 7fdf5affd700 1 -- 172.21.15.73:0/3997471969 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdf54009320 con 0x7fdf640f5410 2023-12-01T22:18:30.180 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:30.173+0000 7fdf6c3ba700 1 -- 172.21.15.73:0/3997471969 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf64060420 con 0x7fdf640f5410 2023-12-01T22:18:30.181 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:30.173+0000 7fdf5affd700 1 -- 172.21.15.73:0/3997471969 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdf54013f00 con 0x7fdf640f5410 2023-12-01T22:18:30.181 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:30.173+0000 7fdf5affd700 1 -- 172.21.15.73:0/3997471969 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdf54014d90 con 0x7fdf640f5410 2023-12-01T22:18:30.181 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:30.173+0000 7fdf6c3ba700 1 -- 172.21.15.73:0/3997471969 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf64109b00 con 0x7fdf640f5410 2023-12-01T22:18:30.181 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:30.173+0000 7fdf6c3ba700 1 -- 172.21.15.73:0/3997471969 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf64059100 con 0x7fdf640f5410 2023-12-01T22:18:30.181 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:30.173+0000 7fdf5affd700 1 -- 172.21.15.73:0/3997471969 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdf540095f0 con 0x7fdf640f5410 2023-12-01T22:18:30.181 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:30.174+0000 7fdf5affd700 1 --2- 172.21.15.73:0/3997471969 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fdf5002ff60 0x7fdf50032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:18:30.181 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:30.174+0000 7fdf5affd700 1 -- 172.21.15.73:0/3997471969 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdf54040fd0 con 0x7fdf640f5410 2023-12-01T22:18:30.182 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:30.174+0000 7fdf69955700 1 --2- 172.21.15.73:0/3997471969 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fdf5002ff60 0x7fdf50032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:18:30.182 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:30.174+0000 7fdf69955700 1 --2- 172.21.15.73:0/3997471969 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fdf5002ff60 0x7fdf50032420 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7fdf600096a0 tx=0x7fdf60006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:18:30.182 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:30.179+0000 7fdf5affd700 1 -- 172.21.15.73:0/3997471969 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdf54014420 con 0x7fdf640f5410 2023-12-01T22:18:30.386 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:30.384+0000 7fdf6c3ba700 1 -- 172.21.15.73:0/3997471969 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdf64045ad0 con 0x7fdf640f5410 2023-12-01T22:18:30.387 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:30.385+0000 7fdf5affd700 1 -- 172.21.15.73:0/3997471969 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fdf5401d7f0 con 0x7fdf640f5410 2023-12-01T22:18:30.387 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:18:30.387 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:18:30.389 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:30.386+0000 7fdf6c3ba700 1 -- 172.21.15.73:0/3997471969 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fdf5002ff60 msgr2=0x7fdf50032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:18:30.389 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:30.386+0000 7fdf6c3ba700 1 --2- 172.21.15.73:0/3997471969 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fdf5002ff60 0x7fdf50032420 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7fdf600096a0 tx=0x7fdf60006b40).stop 2023-12-01T22:18:30.389 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:30.386+0000 7fdf6c3ba700 1 -- 172.21.15.73:0/3997471969 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf640f5410 msgr2=0x7fdf64109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:18:30.389 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:30.386+0000 7fdf6c3ba700 1 --2- 172.21.15.73:0/3997471969 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf640f5410 0x7fdf64109450 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7fdf54013510 tx=0x7fdf5400b040).stop 2023-12-01T22:18:30.390 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:30.386+0000 7fdf6c3ba700 1 -- 172.21.15.73:0/3997471969 shutdown_connections 2023-12-01T22:18:30.390 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:30.386+0000 7fdf6c3ba700 1 --2- 172.21.15.73:0/3997471969 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fdf5002ff60 0x7fdf50032420 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:18:30.390 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:30.386+0000 7fdf6c3ba700 1 --2- 172.21.15.73:0/3997471969 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf640f5410 0x7fdf64109450 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:18:30.390 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:30.386+0000 7fdf6c3ba700 1 -- 172.21.15.73:0/3997471969 >> 172.21.15.73:0/3997471969 conn(0x7fdf640f0b70 msgr2=0x7fdf640f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:18:30.390 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:30.386+0000 7fdf6c3ba700 1 -- 172.21.15.73:0/3997471969 shutdown_connections 2023-12-01T22:18:30.390 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:30.386+0000 7fdf6c3ba700 1 -- 172.21.15.73:0/3997471969 wait complete. 2023-12-01T22:18:30.391 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:18:31.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:18:30 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:18:30.386181+0000 mon.smithi029 (mon.0) 287 : audit [DBG] from='client.? 172.21.15.73:0/3997471969' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:18:31.787 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:18:31.787 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:18:32.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:18:31 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:18:30.690289+0000 mgr.smithi029.loxrfu (mgr.14164) 225 : cluster [DBG] pgmap v213: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:18:32.984 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:32.983+0000 7f2ccb449700 1 -- 172.21.15.73:0/2566956591 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2cc40f5410 msgr2=0x7f2cc40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:18:32.984 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:32.983+0000 7f2ccb449700 1 --2- 172.21.15.73:0/2566956591 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2cc40f5410 0x7f2cc40f5830 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 rx=0x7f2cb4009fb0 tx=0x7f2cb4009b30).stop 2023-12-01T22:18:32.984 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:32.983+0000 7f2ccb449700 1 -- 172.21.15.73:0/2566956591 shutdown_connections 2023-12-01T22:18:32.984 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:32.983+0000 7f2ccb449700 1 --2- 172.21.15.73:0/2566956591 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2cc40f5410 0x7f2cc40f5830 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:18:32.984 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:32.983+0000 7f2ccb449700 1 -- 172.21.15.73:0/2566956591 >> 172.21.15.73:0/2566956591 conn(0x7f2cc40f0b70 msgr2=0x7f2cc40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:18:32.985 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:32.983+0000 7f2ccb449700 1 -- 172.21.15.73:0/2566956591 shutdown_connections 2023-12-01T22:18:32.985 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:32.983+0000 7f2ccb449700 1 -- 172.21.15.73:0/2566956591 wait complete. 2023-12-01T22:18:32.985 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:32.983+0000 7f2ccb449700 1 Processor -- start 2023-12-01T22:18:32.985 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:32.983+0000 7f2ccb449700 1 -- start start 2023-12-01T22:18:32.985 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:32.983+0000 7f2ccb449700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2cc40f5410 0x7f2cc4106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:18:32.985 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:32.983+0000 7f2ccb449700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cc4106c20 con 0x7f2cc40f5410 2023-12-01T22:18:32.985 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:32.984+0000 7f2cc91e5700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2cc40f5410 0x7f2cc4106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:18:32.985 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:32.984+0000 7f2cc91e5700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2cc40f5410 0x7f2cc4106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:35046/0 (socket says 172.21.15.73:35046) 2023-12-01T22:18:32.985 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:32.984+0000 7f2cc91e5700 1 -- 172.21.15.73:0/3059615774 learned_addr learned my addr 172.21.15.73:0/3059615774 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:18:32.986 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:32.984+0000 7f2cc91e5700 1 -- 172.21.15.73:0/3059615774 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cb4005040 con 0x7f2cc40f5410 2023-12-01T22:18:32.986 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:32.984+0000 7f2cc91e5700 1 --2- 172.21.15.73:0/3059615774 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2cc40f5410 0x7f2cc4106730 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7f2cb4013510 tx=0x7f2cb400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:18:32.986 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:32.984+0000 7f2cb9ffb700 1 -- 172.21.15.73:0/3059615774 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2cb4009320 con 0x7f2cc40f5410 2023-12-01T22:18:32.987 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:32.984+0000 7f2ccb449700 1 -- 172.21.15.73:0/3059615774 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cc4106e20 con 0x7f2cc40f5410 2023-12-01T22:18:32.987 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:32.984+0000 7f2cb9ffb700 1 -- 172.21.15.73:0/3059615774 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2cb4013f00 con 0x7f2cc40f5410 2023-12-01T22:18:32.987 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:32.984+0000 7f2cb9ffb700 1 -- 172.21.15.73:0/3059615774 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2cb4014d90 con 0x7f2cc40f5410 2023-12-01T22:18:32.987 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:32.984+0000 7f2ccb449700 1 -- 172.21.15.73:0/3059615774 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cc41072b0 con 0x7f2cc40f5410 2023-12-01T22:18:32.987 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:32.985+0000 7f2cb9ffb700 1 -- 172.21.15.73:0/3059615774 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2cb40095f0 con 0x7f2cc40f5410 2023-12-01T22:18:32.987 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:32.985+0000 7f2ccb449700 1 -- 172.21.15.73:0/3059615774 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cc4045ad0 con 0x7f2cc40f5410 2023-12-01T22:18:32.987 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:32.986+0000 7f2cb9ffb700 1 --2- 172.21.15.73:0/3059615774 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f2cb002ff60 0x7f2cb0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:18:32.987 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:32.986+0000 7f2cb9ffb700 1 -- 172.21.15.73:0/3059615774 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2cb4040fc0 con 0x7f2cc40f5410 2023-12-01T22:18:32.987 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:32.986+0000 7f2cc89e4700 1 --2- 172.21.15.73:0/3059615774 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f2cb002ff60 0x7f2cb0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:18:32.987 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:32.986+0000 7f2cc89e4700 1 --2- 172.21.15.73:0/3059615774 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f2cb002ff60 0x7f2cb0032420 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f2cc00096a0 tx=0x7f2cc0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:18:32.991 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:32.990+0000 7f2cb9ffb700 1 -- 172.21.15.73:0/3059615774 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2cb4014420 con 0x7f2cc40f5410 2023-12-01T22:18:33.195 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:33.194+0000 7f2ccb449700 1 -- 172.21.15.73:0/3059615774 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2cc40fbf90 con 0x7f2cc40f5410 2023-12-01T22:18:33.196 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:33.195+0000 7f2cb9ffb700 1 -- 172.21.15.73:0/3059615774 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2cb4010230 con 0x7f2cc40f5410 2023-12-01T22:18:33.196 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:18:33.196 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:18:33.197 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:33.197+0000 7f2ccb449700 1 -- 172.21.15.73:0/3059615774 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f2cb002ff60 msgr2=0x7f2cb0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:18:33.197 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:33.197+0000 7f2ccb449700 1 --2- 172.21.15.73:0/3059615774 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f2cb002ff60 0x7f2cb0032420 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f2cc00096a0 tx=0x7f2cc0006b40).stop 2023-12-01T22:18:33.198 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:33.197+0000 7f2ccb449700 1 -- 172.21.15.73:0/3059615774 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2cc40f5410 msgr2=0x7f2cc4106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:18:33.198 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:33.197+0000 7f2ccb449700 1 --2- 172.21.15.73:0/3059615774 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2cc40f5410 0x7f2cc4106730 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7f2cb4013510 tx=0x7f2cb400b040).stop 2023-12-01T22:18:33.198 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:33.197+0000 7f2ccb449700 1 -- 172.21.15.73:0/3059615774 shutdown_connections 2023-12-01T22:18:33.198 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:33.197+0000 7f2ccb449700 1 --2- 172.21.15.73:0/3059615774 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f2cb002ff60 0x7f2cb0032420 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:18:33.198 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:33.197+0000 7f2ccb449700 1 --2- 172.21.15.73:0/3059615774 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2cc40f5410 0x7f2cc4106730 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:18:33.198 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:33.197+0000 7f2ccb449700 1 -- 172.21.15.73:0/3059615774 >> 172.21.15.73:0/3059615774 conn(0x7f2cc40f0b70 msgr2=0x7f2cc40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:18:33.198 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:33.197+0000 7f2ccb449700 1 -- 172.21.15.73:0/3059615774 shutdown_connections 2023-12-01T22:18:33.198 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:33.197+0000 7f2ccb449700 1 -- 172.21.15.73:0/3059615774 wait complete. 2023-12-01T22:18:33.198 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:18:34.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:18:33 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:18:32.690606+0000 mgr.smithi029.loxrfu (mgr.14164) 226 : cluster [DBG] pgmap v214: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:18:34.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:18:33 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:18:33.195563+0000 mon.smithi029 (mon.0) 288 : audit [DBG] from='client.? 172.21.15.73:0/3059615774' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:18:34.593 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:18:34.593 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:18:35.681 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:35.679+0000 7fb55cb4c700 1 -- 172.21.15.73:0/841735920 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5580f6510 msgr2=0x7fb5580f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:18:35.682 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:35.679+0000 7fb55cb4c700 1 --2- 172.21.15.73:0/841735920 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5580f6510 0x7fb5580f6930 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 rx=0x7fb54c009fb0 tx=0x7fb54c009b30).stop 2023-12-01T22:18:35.682 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:35.679+0000 7fb55cb4c700 1 -- 172.21.15.73:0/841735920 shutdown_connections 2023-12-01T22:18:35.682 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:35.679+0000 7fb55cb4c700 1 --2- 172.21.15.73:0/841735920 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5580f6510 0x7fb5580f6930 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:18:35.682 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:35.679+0000 7fb55cb4c700 1 -- 172.21.15.73:0/841735920 >> 172.21.15.73:0/841735920 conn(0x7fb5580f1c70 msgr2=0x7fb5580f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:18:35.682 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:35.679+0000 7fb55cb4c700 1 -- 172.21.15.73:0/841735920 shutdown_connections 2023-12-01T22:18:35.682 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:35.679+0000 7fb55cb4c700 1 -- 172.21.15.73:0/841735920 wait complete. 2023-12-01T22:18:35.682 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:35.680+0000 7fb55cb4c700 1 Processor -- start 2023-12-01T22:18:35.682 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:35.680+0000 7fb55cb4c700 1 -- start start 2023-12-01T22:18:35.682 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:35.680+0000 7fb55cb4c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5580f6510 0x7fb558106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:18:35.682 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:35.680+0000 7fb55cb4c700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb558106c20 con 0x7fb5580f6510 2023-12-01T22:18:35.682 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:35.680+0000 7fb55659c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5580f6510 0x7fb558106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:18:35.682 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:35.680+0000 7fb55659c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5580f6510 0x7fb558106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:49798/0 (socket says 172.21.15.73:49798) 2023-12-01T22:18:35.682 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:35.680+0000 7fb55659c700 1 -- 172.21.15.73:0/1124255092 learned_addr learned my addr 172.21.15.73:0/1124255092 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:18:35.682 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:35.680+0000 7fb55659c700 1 -- 172.21.15.73:0/1124255092 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb54c005040 con 0x7fb5580f6510 2023-12-01T22:18:35.683 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:35.680+0000 7fb55659c700 1 --2- 172.21.15.73:0/1124255092 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5580f6510 0x7fb558106730 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7fb54c013510 tx=0x7fb54c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:18:35.683 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:35.681+0000 7fb546ffd700 1 -- 172.21.15.73:0/1124255092 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb54c009320 con 0x7fb5580f6510 2023-12-01T22:18:35.683 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:35.681+0000 7fb55cb4c700 1 -- 172.21.15.73:0/1124255092 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb558106e20 con 0x7fb5580f6510 2023-12-01T22:18:35.683 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:35.681+0000 7fb546ffd700 1 -- 172.21.15.73:0/1124255092 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb54c013f00 con 0x7fb5580f6510 2023-12-01T22:18:35.683 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:35.681+0000 7fb546ffd700 1 -- 172.21.15.73:0/1124255092 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb54c014d90 con 0x7fb5580f6510 2023-12-01T22:18:35.683 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:35.681+0000 7fb55cb4c700 1 -- 172.21.15.73:0/1124255092 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5581072b0 con 0x7fb5580f6510 2023-12-01T22:18:35.684 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:35.681+0000 7fb546ffd700 1 -- 172.21.15.73:0/1124255092 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb54c0095f0 con 0x7fb5580f6510 2023-12-01T22:18:35.684 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:35.681+0000 7fb55cb4c700 1 -- 172.21.15.73:0/1124255092 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb558045ad0 con 0x7fb5580f6510 2023-12-01T22:18:35.684 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:35.681+0000 7fb546ffd700 1 --2- 172.21.15.73:0/1124255092 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fb54002ff60 0x7fb540032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:18:35.684 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:35.682+0000 7fb546ffd700 1 -- 172.21.15.73:0/1124255092 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb54c041000 con 0x7fb5580f6510 2023-12-01T22:18:35.684 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:35.682+0000 7fb555d9b700 1 --2- 172.21.15.73:0/1124255092 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fb54002ff60 0x7fb540032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:18:35.684 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:35.682+0000 7fb555d9b700 1 --2- 172.21.15.73:0/1124255092 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fb54002ff60 0x7fb540032420 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7fb5480096a0 tx=0x7fb548006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:18:35.687 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:35.686+0000 7fb546ffd700 1 -- 172.21.15.73:0/1124255092 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb54c014420 con 0x7fb5580f6510 2023-12-01T22:18:35.895 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:35.895+0000 7fb55cb4c700 1 -- 172.21.15.73:0/1124255092 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb5580fbf60 con 0x7fb5580f6510 2023-12-01T22:18:35.896 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:35.895+0000 7fb546ffd700 1 -- 172.21.15.73:0/1124255092 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb54c010230 con 0x7fb5580f6510 2023-12-01T22:18:35.896 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:18:35.896 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:18:35.897 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:35.897+0000 7fb55cb4c700 1 -- 172.21.15.73:0/1124255092 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fb54002ff60 msgr2=0x7fb540032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:18:35.898 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:35.897+0000 7fb55cb4c700 1 --2- 172.21.15.73:0/1124255092 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fb54002ff60 0x7fb540032420 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7fb5480096a0 tx=0x7fb548006b40).stop 2023-12-01T22:18:35.898 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:35.897+0000 7fb55cb4c700 1 -- 172.21.15.73:0/1124255092 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5580f6510 msgr2=0x7fb558106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:18:35.898 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:35.897+0000 7fb55cb4c700 1 --2- 172.21.15.73:0/1124255092 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5580f6510 0x7fb558106730 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7fb54c013510 tx=0x7fb54c00b040).stop 2023-12-01T22:18:35.898 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:35.897+0000 7fb55cb4c700 1 -- 172.21.15.73:0/1124255092 shutdown_connections 2023-12-01T22:18:35.898 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:35.897+0000 7fb55cb4c700 1 --2- 172.21.15.73:0/1124255092 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fb54002ff60 0x7fb540032420 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:18:35.898 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:35.897+0000 7fb55cb4c700 1 --2- 172.21.15.73:0/1124255092 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5580f6510 0x7fb558106730 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:18:35.898 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:35.897+0000 7fb55cb4c700 1 -- 172.21.15.73:0/1124255092 >> 172.21.15.73:0/1124255092 conn(0x7fb5580f1c70 msgr2=0x7fb5580f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:18:35.898 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:35.897+0000 7fb55cb4c700 1 -- 172.21.15.73:0/1124255092 shutdown_connections 2023-12-01T22:18:35.898 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:35.897+0000 7fb55cb4c700 1 -- 172.21.15.73:0/1124255092 wait complete. 2023-12-01T22:18:35.899 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:18:36.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:18:35 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:18:34.690974+0000 mgr.smithi029.loxrfu (mgr.14164) 227 : cluster [DBG] pgmap v215: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:18:37.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:18:36 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:18:35.895633+0000 mon.smithi029 (mon.0) 289 : audit [DBG] from='client.? 172.21.15.73:0/1124255092' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:18:37.273 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:18:37.273 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:18:38.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:18:37 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:18:36.691447+0000 mgr.smithi029.loxrfu (mgr.14164) 228 : cluster [DBG] pgmap v216: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:18:38.490 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:38.488+0000 7f4ecd39d700 1 -- 172.21.15.73:0/2149734836 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4ec80f6510 msgr2=0x7f4ec80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:18:38.491 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:38.488+0000 7f4ecd39d700 1 --2- 172.21.15.73:0/2149734836 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4ec80f6510 0x7f4ec80f6930 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 rx=0x7f4ebc009fb0 tx=0x7f4ebc009b30).stop 2023-12-01T22:18:38.491 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:38.488+0000 7f4ecd39d700 1 -- 172.21.15.73:0/2149734836 shutdown_connections 2023-12-01T22:18:38.491 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:38.488+0000 7f4ecd39d700 1 --2- 172.21.15.73:0/2149734836 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4ec80f6510 0x7f4ec80f6930 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:18:38.491 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:38.488+0000 7f4ecd39d700 1 -- 172.21.15.73:0/2149734836 >> 172.21.15.73:0/2149734836 conn(0x7f4ec80f1c70 msgr2=0x7f4ec80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:18:38.491 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:38.488+0000 7f4ecd39d700 1 -- 172.21.15.73:0/2149734836 shutdown_connections 2023-12-01T22:18:38.491 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:38.488+0000 7f4ecd39d700 1 -- 172.21.15.73:0/2149734836 wait complete. 2023-12-01T22:18:38.491 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:38.489+0000 7f4ecd39d700 1 Processor -- start 2023-12-01T22:18:38.491 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:38.489+0000 7f4ecd39d700 1 -- start start 2023-12-01T22:18:38.491 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:38.489+0000 7f4ecd39d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4ec80f6510 0x7f4ec806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:18:38.491 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:38.489+0000 7f4ecd39d700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ec8069e20 con 0x7f4ec80f6510 2023-12-01T22:18:38.491 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:38.489+0000 7f4ec6ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4ec80f6510 0x7f4ec806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:18:38.491 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:38.489+0000 7f4ec6ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4ec80f6510 0x7f4ec806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:49806/0 (socket says 172.21.15.73:49806) 2023-12-01T22:18:38.492 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:38.489+0000 7f4ec6ffd700 1 -- 172.21.15.73:0/1616870733 learned_addr learned my addr 172.21.15.73:0/1616870733 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:18:38.492 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:38.490+0000 7f4ec6ffd700 1 -- 172.21.15.73:0/1616870733 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ebc005040 con 0x7f4ec80f6510 2023-12-01T22:18:38.492 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:38.490+0000 7f4ec6ffd700 1 --2- 172.21.15.73:0/1616870733 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4ec80f6510 0x7f4ec806b9a0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7f4ebc013510 tx=0x7f4ebc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:18:38.492 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:38.490+0000 7f4eaf7fe700 1 -- 172.21.15.73:0/1616870733 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4ebc009320 con 0x7f4ec80f6510 2023-12-01T22:18:38.492 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:38.490+0000 7f4ecd39d700 1 -- 172.21.15.73:0/1616870733 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ec8069fc0 con 0x7f4ec80f6510 2023-12-01T22:18:38.492 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:38.490+0000 7f4eaf7fe700 1 -- 172.21.15.73:0/1616870733 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4ebc013f00 con 0x7f4ec80f6510 2023-12-01T22:18:38.492 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:38.490+0000 7f4eaf7fe700 1 -- 172.21.15.73:0/1616870733 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4ebc014d90 con 0x7f4ec80f6510 2023-12-01T22:18:38.492 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:38.490+0000 7f4ecd39d700 1 -- 172.21.15.73:0/1616870733 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ec806a450 con 0x7f4ec80f6510 2023-12-01T22:18:38.493 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:38.491+0000 7f4eaf7fe700 1 -- 172.21.15.73:0/1616870733 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4ebc0095f0 con 0x7f4ec80f6510 2023-12-01T22:18:38.493 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:38.491+0000 7f4ecd39d700 1 -- 172.21.15.73:0/1616870733 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ec8059100 con 0x7f4ec80f6510 2023-12-01T22:18:38.493 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:38.491+0000 7f4eaf7fe700 1 --2- 172.21.15.73:0/1616870733 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f4eb002ff60 0x7f4eb0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:18:38.493 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:38.491+0000 7f4eaf7fe700 1 -- 172.21.15.73:0/1616870733 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4ebc041000 con 0x7f4ec80f6510 2023-12-01T22:18:38.493 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:38.491+0000 7f4ec67fc700 1 --2- 172.21.15.73:0/1616870733 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f4eb002ff60 0x7f4eb0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:18:38.493 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:38.491+0000 7f4ec67fc700 1 --2- 172.21.15.73:0/1616870733 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f4eb002ff60 0x7f4eb0032420 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f4eb80096a0 tx=0x7f4eb8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:18:38.496 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:38.496+0000 7f4eaf7fe700 1 -- 172.21.15.73:0/1616870733 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4ebc014420 con 0x7f4ec80f6510 2023-12-01T22:18:38.703 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:38.702+0000 7f4ecd39d700 1 -- 172.21.15.73:0/1616870733 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4ec80fbf60 con 0x7f4ec80f6510 2023-12-01T22:18:38.703 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:38.703+0000 7f4eaf7fe700 1 -- 172.21.15.73:0/1616870733 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4ebc010230 con 0x7f4ec80f6510 2023-12-01T22:18:38.703 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:18:38.703 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:18:38.705 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:38.704+0000 7f4ecd39d700 1 -- 172.21.15.73:0/1616870733 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f4eb002ff60 msgr2=0x7f4eb0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:18:38.705 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:38.704+0000 7f4ecd39d700 1 --2- 172.21.15.73:0/1616870733 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f4eb002ff60 0x7f4eb0032420 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f4eb80096a0 tx=0x7f4eb8006b40).stop 2023-12-01T22:18:38.705 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:38.704+0000 7f4ecd39d700 1 -- 172.21.15.73:0/1616870733 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4ec80f6510 msgr2=0x7f4ec806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:18:38.705 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:38.704+0000 7f4ecd39d700 1 --2- 172.21.15.73:0/1616870733 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4ec80f6510 0x7f4ec806b9a0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7f4ebc013510 tx=0x7f4ebc00b040).stop 2023-12-01T22:18:38.705 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:38.704+0000 7f4ecd39d700 1 -- 172.21.15.73:0/1616870733 shutdown_connections 2023-12-01T22:18:38.705 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:38.704+0000 7f4ecd39d700 1 --2- 172.21.15.73:0/1616870733 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f4eb002ff60 0x7f4eb0032420 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:18:38.705 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:38.704+0000 7f4ecd39d700 1 --2- 172.21.15.73:0/1616870733 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4ec80f6510 0x7f4ec806b9a0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:18:38.706 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:38.704+0000 7f4ecd39d700 1 -- 172.21.15.73:0/1616870733 >> 172.21.15.73:0/1616870733 conn(0x7f4ec80f1c70 msgr2=0x7f4ec80f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:18:38.706 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:38.704+0000 7f4ecd39d700 1 -- 172.21.15.73:0/1616870733 shutdown_connections 2023-12-01T22:18:38.706 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:38.704+0000 7f4ecd39d700 1 -- 172.21.15.73:0/1616870733 wait complete. 2023-12-01T22:18:38.706 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:18:39.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:18:38 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:18:38.703092+0000 mon.smithi029 (mon.0) 290 : audit [DBG] from='client.? 172.21.15.73:0/1616870733' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:18:40.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:18:39 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:18:38.691804+0000 mgr.smithi029.loxrfu (mgr.14164) 229 : cluster [DBG] pgmap v217: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:18:40.090 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:18:40.090 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:18:41.348 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:41.346+0000 7fc135e08700 1 -- 172.21.15.73:0/1541519032 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc1300f6510 msgr2=0x7fc1300f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:18:41.348 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:41.346+0000 7fc135e08700 1 --2- 172.21.15.73:0/1541519032 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc1300f6510 0x7fc1300f6930 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 rx=0x7fc120009fb0 tx=0x7fc120009b30).stop 2023-12-01T22:18:41.348 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:41.346+0000 7fc135e08700 1 -- 172.21.15.73:0/1541519032 shutdown_connections 2023-12-01T22:18:41.348 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:41.346+0000 7fc135e08700 1 --2- 172.21.15.73:0/1541519032 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc1300f6510 0x7fc1300f6930 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:18:41.348 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:41.346+0000 7fc135e08700 1 -- 172.21.15.73:0/1541519032 >> 172.21.15.73:0/1541519032 conn(0x7fc1300f1c70 msgr2=0x7fc1300f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:18:41.348 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:41.346+0000 7fc135e08700 1 -- 172.21.15.73:0/1541519032 shutdown_connections 2023-12-01T22:18:41.348 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:41.346+0000 7fc135e08700 1 -- 172.21.15.73:0/1541519032 wait complete. 2023-12-01T22:18:41.348 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:41.347+0000 7fc135e08700 1 Processor -- start 2023-12-01T22:18:41.348 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:41.347+0000 7fc135e08700 1 -- start start 2023-12-01T22:18:41.348 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:41.347+0000 7fc135e08700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc1300f6510 0x7fc13006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:18:41.349 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:41.347+0000 7fc135e08700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc130069e20 con 0x7fc1300f6510 2023-12-01T22:18:41.349 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:41.347+0000 7fc12f7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc1300f6510 0x7fc13006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:18:41.349 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:41.347+0000 7fc12f7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc1300f6510 0x7fc13006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:49834/0 (socket says 172.21.15.73:49834) 2023-12-01T22:18:41.349 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:41.347+0000 7fc12f7fe700 1 -- 172.21.15.73:0/570503142 learned_addr learned my addr 172.21.15.73:0/570503142 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:18:41.350 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:41.348+0000 7fc12f7fe700 1 -- 172.21.15.73:0/570503142 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc120005040 con 0x7fc1300f6510 2023-12-01T22:18:41.350 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:41.348+0000 7fc12f7fe700 1 --2- 172.21.15.73:0/570503142 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc1300f6510 0x7fc13006b9a0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7fc120013510 tx=0x7fc12000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:18:41.350 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:41.348+0000 7fc12cff9700 1 -- 172.21.15.73:0/570503142 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc120009320 con 0x7fc1300f6510 2023-12-01T22:18:41.350 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:41.348+0000 7fc135e08700 1 -- 172.21.15.73:0/570503142 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc130069fc0 con 0x7fc1300f6510 2023-12-01T22:18:41.351 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:41.348+0000 7fc12cff9700 1 -- 172.21.15.73:0/570503142 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc120013f00 con 0x7fc1300f6510 2023-12-01T22:18:41.351 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:41.348+0000 7fc12cff9700 1 -- 172.21.15.73:0/570503142 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc120014d90 con 0x7fc1300f6510 2023-12-01T22:18:41.351 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:41.348+0000 7fc135e08700 1 -- 172.21.15.73:0/570503142 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc13006a450 con 0x7fc1300f6510 2023-12-01T22:18:41.351 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:41.349+0000 7fc12cff9700 1 -- 172.21.15.73:0/570503142 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc1200263a0 con 0x7fc1300f6510 2023-12-01T22:18:41.351 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:41.349+0000 7fc12cff9700 1 --2- 172.21.15.73:0/570503142 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fc11c02ff60 0x7fc11c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:18:41.351 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:41.349+0000 7fc12cff9700 1 -- 172.21.15.73:0/570503142 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc120041040 con 0x7fc1300f6510 2023-12-01T22:18:41.351 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:41.349+0000 7fc135e08700 1 -- 172.21.15.73:0/570503142 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc130059100 con 0x7fc1300f6510 2023-12-01T22:18:41.351 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:41.349+0000 7fc127fff700 1 --2- 172.21.15.73:0/570503142 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fc11c02ff60 0x7fc11c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:18:41.351 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:41.350+0000 7fc127fff700 1 --2- 172.21.15.73:0/570503142 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fc11c02ff60 0x7fc11c032420 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7fc1180096a0 tx=0x7fc118006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:18:41.354 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:41.354+0000 7fc12cff9700 1 -- 172.21.15.73:0/570503142 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc120012920 con 0x7fc1300f6510 2023-12-01T22:18:41.558 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:41.558+0000 7fc135e08700 1 -- 172.21.15.73:0/570503142 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc130045ad0 con 0x7fc1300f6510 2023-12-01T22:18:41.559 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:41.558+0000 7fc12cff9700 1 -- 172.21.15.73:0/570503142 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc12002b020 con 0x7fc1300f6510 2023-12-01T22:18:41.559 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:18:41.559 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:18:41.561 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:41.560+0000 7fc135e08700 1 -- 172.21.15.73:0/570503142 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fc11c02ff60 msgr2=0x7fc11c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:18:41.561 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:41.560+0000 7fc135e08700 1 --2- 172.21.15.73:0/570503142 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fc11c02ff60 0x7fc11c032420 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7fc1180096a0 tx=0x7fc118006b40).stop 2023-12-01T22:18:41.561 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:41.560+0000 7fc135e08700 1 -- 172.21.15.73:0/570503142 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc1300f6510 msgr2=0x7fc13006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:18:41.561 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:41.560+0000 7fc135e08700 1 --2- 172.21.15.73:0/570503142 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc1300f6510 0x7fc13006b9a0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7fc120013510 tx=0x7fc12000b040).stop 2023-12-01T22:18:41.561 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:41.560+0000 7fc135e08700 1 -- 172.21.15.73:0/570503142 shutdown_connections 2023-12-01T22:18:41.561 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:41.560+0000 7fc135e08700 1 --2- 172.21.15.73:0/570503142 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fc11c02ff60 0x7fc11c032420 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:18:41.561 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:41.560+0000 7fc135e08700 1 --2- 172.21.15.73:0/570503142 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc1300f6510 0x7fc13006b9a0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:18:41.561 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:41.560+0000 7fc135e08700 1 -- 172.21.15.73:0/570503142 >> 172.21.15.73:0/570503142 conn(0x7fc1300f1c70 msgr2=0x7fc1300f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:18:41.561 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:41.560+0000 7fc135e08700 1 -- 172.21.15.73:0/570503142 shutdown_connections 2023-12-01T22:18:41.561 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:41.560+0000 7fc135e08700 1 -- 172.21.15.73:0/570503142 wait complete. 2023-12-01T22:18:41.561 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:18:41.776 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:18:41 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:18:40.692142+0000 mgr.smithi029.loxrfu (mgr.14164) 230 : cluster [DBG] pgmap v218: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:18:42.776 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:18:42 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:18:41.558569+0000 mon.smithi029 (mon.0) 291 : audit [DBG] from='client.? 172.21.15.73:0/570503142' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:18:42.953 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:18:42.953 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:18:43.776 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:18:43 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:18:42.692496+0000 mgr.smithi029.loxrfu (mgr.14164) 231 : cluster [DBG] pgmap v219: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:18:44.192 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:44.189+0000 7f642bfff700 1 -- 172.21.15.73:0/625113218 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f642c0f51a0 msgr2=0x7f642c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:18:44.192 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:44.190+0000 7f642bfff700 1 --2- 172.21.15.73:0/625113218 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f642c0f51a0 0x7f642c0f55c0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 rx=0x7f641c009fb0 tx=0x7f641c009b30).stop 2023-12-01T22:18:44.192 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:44.190+0000 7f642bfff700 1 -- 172.21.15.73:0/625113218 shutdown_connections 2023-12-01T22:18:44.192 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:44.190+0000 7f642bfff700 1 --2- 172.21.15.73:0/625113218 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f642c0f51a0 0x7f642c0f55c0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:18:44.193 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:44.190+0000 7f642bfff700 1 -- 172.21.15.73:0/625113218 >> 172.21.15.73:0/625113218 conn(0x7f642c0f09a0 msgr2=0x7f642c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:18:44.193 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:44.190+0000 7f642bfff700 1 -- 172.21.15.73:0/625113218 shutdown_connections 2023-12-01T22:18:44.193 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:44.190+0000 7f642bfff700 1 -- 172.21.15.73:0/625113218 wait complete. 2023-12-01T22:18:44.193 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:44.190+0000 7f642bfff700 1 Processor -- start 2023-12-01T22:18:44.193 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:44.190+0000 7f642bfff700 1 -- start start 2023-12-01T22:18:44.193 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:44.191+0000 7f642bfff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f642c0f51a0 0x7f642c1053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:18:44.193 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:44.191+0000 7f642bfff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f642c1058a0 con 0x7f642c0f51a0 2023-12-01T22:18:44.194 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:44.191+0000 7f642affd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f642c0f51a0 0x7f642c1053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:18:44.194 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:44.191+0000 7f642affd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f642c0f51a0 0x7f642c1053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:49858/0 (socket says 172.21.15.73:49858) 2023-12-01T22:18:44.194 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:44.191+0000 7f642affd700 1 -- 172.21.15.73:0/3591440620 learned_addr learned my addr 172.21.15.73:0/3591440620 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:18:44.194 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:44.191+0000 7f642affd700 1 -- 172.21.15.73:0/3591440620 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f641c005040 con 0x7f642c0f51a0 2023-12-01T22:18:44.195 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:44.192+0000 7f642affd700 1 --2- 172.21.15.73:0/3591440620 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f642c0f51a0 0x7f642c1053b0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7f641c013510 tx=0x7f641c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:18:44.195 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:44.192+0000 7f64137fe700 1 -- 172.21.15.73:0/3591440620 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f641c009320 con 0x7f642c0f51a0 2023-12-01T22:18:44.195 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:44.192+0000 7f642bfff700 1 -- 172.21.15.73:0/3591440620 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f642c105aa0 con 0x7f642c0f51a0 2023-12-01T22:18:44.195 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:44.192+0000 7f64137fe700 1 -- 172.21.15.73:0/3591440620 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f641c013f00 con 0x7f642c0f51a0 2023-12-01T22:18:44.195 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:44.192+0000 7f64137fe700 1 -- 172.21.15.73:0/3591440620 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f641c014dc0 con 0x7f642c0f51a0 2023-12-01T22:18:44.195 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:44.192+0000 7f642bfff700 1 -- 172.21.15.73:0/3591440620 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f642c105f30 con 0x7f642c0f51a0 2023-12-01T22:18:44.195 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:44.193+0000 7f64137fe700 1 -- 172.21.15.73:0/3591440620 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f641c0095f0 con 0x7f642c0f51a0 2023-12-01T22:18:44.195 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:44.193+0000 7f642bfff700 1 -- 172.21.15.73:0/3591440620 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f642c045ad0 con 0x7f642c0f51a0 2023-12-01T22:18:44.195 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:44.193+0000 7f64137fe700 1 --2- 172.21.15.73:0/3591440620 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f641402ff60 0x7f6414032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:18:44.195 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:44.193+0000 7f64137fe700 1 -- 172.21.15.73:0/3591440620 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f641c041020 con 0x7f642c0f51a0 2023-12-01T22:18:44.196 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:44.193+0000 7f642a7fc700 1 --2- 172.21.15.73:0/3591440620 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f641402ff60 0x7f6414032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:18:44.196 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:44.194+0000 7f642a7fc700 1 --2- 172.21.15.73:0/3591440620 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f641402ff60 0x7f6414032420 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f64200096a0 tx=0x7f6420006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:18:44.200 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:44.198+0000 7f64137fe700 1 -- 172.21.15.73:0/3591440620 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f641c014420 con 0x7f642c0f51a0 2023-12-01T22:18:44.404 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:44.403+0000 7f642bfff700 1 -- 172.21.15.73:0/3591440620 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f642c106d80 con 0x7f642c0f51a0 2023-12-01T22:18:44.405 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:44.403+0000 7f64137fe700 1 -- 172.21.15.73:0/3591440620 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f641c01c920 con 0x7f642c0f51a0 2023-12-01T22:18:44.405 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:18:44.405 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:18:44.406 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:44.404+0000 7f642bfff700 1 -- 172.21.15.73:0/3591440620 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f641402ff60 msgr2=0x7f6414032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:18:44.407 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:44.405+0000 7f642bfff700 1 --2- 172.21.15.73:0/3591440620 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f641402ff60 0x7f6414032420 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f64200096a0 tx=0x7f6420006b40).stop 2023-12-01T22:18:44.407 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:44.405+0000 7f642bfff700 1 -- 172.21.15.73:0/3591440620 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f642c0f51a0 msgr2=0x7f642c1053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:18:44.407 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:44.405+0000 7f642bfff700 1 --2- 172.21.15.73:0/3591440620 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f642c0f51a0 0x7f642c1053b0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7f641c013510 tx=0x7f641c00b040).stop 2023-12-01T22:18:44.407 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:44.405+0000 7f642bfff700 1 -- 172.21.15.73:0/3591440620 shutdown_connections 2023-12-01T22:18:44.407 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:44.405+0000 7f642bfff700 1 --2- 172.21.15.73:0/3591440620 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f641402ff60 0x7f6414032420 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:18:44.407 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:44.405+0000 7f642bfff700 1 --2- 172.21.15.73:0/3591440620 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f642c0f51a0 0x7f642c1053b0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:18:44.407 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:44.405+0000 7f642bfff700 1 -- 172.21.15.73:0/3591440620 >> 172.21.15.73:0/3591440620 conn(0x7f642c0f09a0 msgr2=0x7f642c0f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:18:44.407 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:44.405+0000 7f642bfff700 1 -- 172.21.15.73:0/3591440620 shutdown_connections 2023-12-01T22:18:44.407 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:44.405+0000 7f642bfff700 1 -- 172.21.15.73:0/3591440620 wait complete. 2023-12-01T22:18:44.407 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:18:44.777 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:18:44 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cephadm 2023-12-01T22:18:43.068403+0000 mgr.smithi029.loxrfu (mgr.14164) 232 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon node-exporter.smithi029 ... 2023-12-01T22:18:44.777 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:18:44 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: Verifying port 9100 ... 2023-12-01T22:18:44.777 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:18:44 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: Non-zero exit code 1 from systemctl start ceph-5558ab6c-9096-11ee-95a2-87774f69a715@node-exporter.smithi029 2023-12-01T22:18:44.777 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:18:44 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: systemctl: stderr Job for ceph-5558ab6c-9096-11ee-95a2-87774f69a715@node-exporter.smithi029.service failed because a timeout was exceeded. 2023-12-01T22:18:44.777 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:18:44 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: systemctl: stderr See "systemctl status ceph-5558ab6c-9096-11ee-95a2-87774f69a715@node-exporter.smithi029.service" and "journalctl -xe" for details. 2023-12-01T22:18:44.777 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:18:44 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: Traceback (most recent call last): 2023-12-01T22:18:44.777 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:18:44 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: File "/var/lib/ceph/5558ab6c-9096-11ee-95a2-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2023-12-01T22:18:44.778 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:18:44 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: main() 2023-12-01T22:18:44.778 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:18:44 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: File "/var/lib/ceph/5558ab6c-9096-11ee-95a2-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2023-12-01T22:18:44.778 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:18:44 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: r = ctx.func(ctx) 2023-12-01T22:18:44.778 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:18:44 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: File "/var/lib/ceph/5558ab6c-9096-11ee-95a2-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2023-12-01T22:18:44.778 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:18:44 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: return func(ctx) 2023-12-01T22:18:44.778 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:18:44 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: File "/var/lib/ceph/5558ab6c-9096-11ee-95a2-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2023-12-01T22:18:44.778 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:18:44 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: ports=daemon_ports) 2023-12-01T22:18:44.778 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:18:44 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: File "/var/lib/ceph/5558ab6c-9096-11ee-95a2-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2608, in deploy_daemon 2023-12-01T22:18:44.778 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:18:44 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: c, osd_fsid=osd_fsid, ports=ports) 2023-12-01T22:18:44.778 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:18:44 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: File "/var/lib/ceph/5558ab6c-9096-11ee-95a2-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2806, in deploy_daemon_units 2023-12-01T22:18:44.778 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:18:44 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: call_throws(ctx, ['systemctl', 'start', unit_name]) 2023-12-01T22:18:44.778 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:18:44 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: File "/var/lib/ceph/5558ab6c-9096-11ee-95a2-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2023-12-01T22:18:44.778 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:18:44 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2023-12-01T22:18:44.779 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:18:44 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: RuntimeError: Failed command: systemctl start ceph-5558ab6c-9096-11ee-95a2-87774f69a715@node-exporter.smithi029 2023-12-01T22:18:44.779 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:18:44 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: Traceback (most recent call last): 2023-12-01T22:18:44.779 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:18:44 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2023-12-01T22:18:44.779 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:18:44 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: yield (conn, connr) 2023-12-01T22:18:44.779 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:18:44 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2023-12-01T22:18:44.779 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:18:44 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: code, '\n'.join(err))) 2023-12-01T22:18:44.779 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:18:44 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon node-exporter.smithi029 ... 2023-12-01T22:18:44.779 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:18:44 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: Verifying port 9100 ... 2023-12-01T22:18:44.779 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:18:44 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: Non-zero exit code 1 from systemctl start ceph-5558ab6c-9096-11ee-95a2-87774f69a715@node-exporter.smithi029 2023-12-01T22:18:44.779 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:18:44 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: systemctl: stderr Job for ceph-5558ab6c-9096-11ee-95a2-87774f69a715@node-exporter.smithi029.service failed because a timeout was exceeded. 2023-12-01T22:18:44.779 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:18:44 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: systemctl: stderr See "systemctl status ceph-5558ab6c-9096-11ee-95a2-87774f69a715@node-exporter.smithi029.service" and "journalctl -xe" for details. 2023-12-01T22:18:44.779 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:18:44 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: Traceback (most recent call last): 2023-12-01T22:18:44.779 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:18:44 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: File "/var/lib/ceph/5558ab6c-9096-11ee-95a2-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2023-12-01T22:18:44.779 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:18:44 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: main() 2023-12-01T22:18:44.780 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:18:44 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: File "/var/lib/ceph/5558ab6c-9096-11ee-95a2-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2023-12-01T22:18:44.780 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:18:44 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: r = ctx.func(ctx) 2023-12-01T22:18:44.780 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:18:44 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: File "/var/lib/ceph/5558ab6c-9096-11ee-95a2-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2023-12-01T22:18:44.780 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:18:44 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: return func(ctx) 2023-12-01T22:18:44.780 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:18:44 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: File "/var/lib/ceph/5558ab6c-9096-11ee-95a2-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2023-12-01T22:18:44.780 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:18:44 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: ports=daemon_ports) 2023-12-01T22:18:44.780 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:18:44 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: File "/var/lib/ceph/5558ab6c-9096-11ee-95a2-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2608, in deploy_daemon 2023-12-01T22:18:44.780 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:18:44 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: c, osd_fsid=osd_fsid, ports=ports) 2023-12-01T22:18:44.780 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:18:44 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: File "/var/lib/ceph/5558ab6c-9096-11ee-95a2-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2806, in deploy_daemon_units 2023-12-01T22:18:44.780 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:18:44 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: call_throws(ctx, ['systemctl', 'start', unit_name]) 2023-12-01T22:18:44.780 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:18:44 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: File "/var/lib/ceph/5558ab6c-9096-11ee-95a2-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2023-12-01T22:18:44.780 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:18:44 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2023-12-01T22:18:44.780 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:18:44 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: RuntimeError: Failed command: systemctl start ceph-5558ab6c-9096-11ee-95a2-87774f69a715@node-exporter.smithi029 2023-12-01T22:18:44.781 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:18:44 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cephadm 2023-12-01T22:18:43.074323+0000 mgr.smithi029.loxrfu (mgr.14164) 233 : cephadm [INF] Deploying daemon prometheus.smithi029 on smithi029 2023-12-01T22:18:44.781 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:18:44 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:18:44.404572+0000 mon.smithi029 (mon.0) 292 : audit [DBG] from='client.? 172.21.15.73:0/3591440620' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:18:45.776 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:18:45 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:18:44.692892+0000 mgr.smithi029.loxrfu (mgr.14164) 234 : cluster [DBG] pgmap v220: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:18:45.828 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:18:45.828 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:18:47.045 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:47.043+0000 7fe2f0a30700 1 -- 172.21.15.73:0/176715801 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2ec0601e0 msgr2=0x7fe2ec0f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:18:47.045 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:47.043+0000 7fe2f0a30700 1 --2- 172.21.15.73:0/176715801 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2ec0601e0 0x7fe2ec0f8f20 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 rx=0x7fe2dc009fb0 tx=0x7fe2dc009b30).stop 2023-12-01T22:18:47.045 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:47.043+0000 7fe2f0a30700 1 -- 172.21.15.73:0/176715801 shutdown_connections 2023-12-01T22:18:47.045 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:47.043+0000 7fe2f0a30700 1 --2- 172.21.15.73:0/176715801 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2ec0601e0 0x7fe2ec0f8f20 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:18:47.045 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:47.043+0000 7fe2f0a30700 1 -- 172.21.15.73:0/176715801 >> 172.21.15.73:0/176715801 conn(0x7fe2ec0f09a0 msgr2=0x7fe2ec0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:18:47.045 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:47.043+0000 7fe2f0a30700 1 -- 172.21.15.73:0/176715801 shutdown_connections 2023-12-01T22:18:47.045 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:47.043+0000 7fe2f0a30700 1 -- 172.21.15.73:0/176715801 wait complete. 2023-12-01T22:18:47.046 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:47.043+0000 7fe2f0a30700 1 Processor -- start 2023-12-01T22:18:47.046 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:47.043+0000 7fe2f0a30700 1 -- start start 2023-12-01T22:18:47.046 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:47.044+0000 7fe2f0a30700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2ec0601e0 0x7fe2ec0f69a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:18:47.046 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:47.044+0000 7fe2f0a30700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2ec0f4e20 con 0x7fe2ec0601e0 2023-12-01T22:18:47.047 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:47.044+0000 7fe2ead9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2ec0601e0 0x7fe2ec0f69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:18:47.047 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:47.044+0000 7fe2ead9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2ec0601e0 0x7fe2ec0f69a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:38614/0 (socket says 172.21.15.73:38614) 2023-12-01T22:18:47.048 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:47.044+0000 7fe2ead9d700 1 -- 172.21.15.73:0/2567327773 learned_addr learned my addr 172.21.15.73:0/2567327773 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:18:47.048 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:47.044+0000 7fe2ead9d700 1 -- 172.21.15.73:0/2567327773 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2dc005040 con 0x7fe2ec0601e0 2023-12-01T22:18:47.048 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:47.044+0000 7fe2ead9d700 1 --2- 172.21.15.73:0/2567327773 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2ec0601e0 0x7fe2ec0f69a0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7fe2dc013510 tx=0x7fe2dc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:18:47.048 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:47.044+0000 7fe2d37fe700 1 -- 172.21.15.73:0/2567327773 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe2dc009320 con 0x7fe2ec0601e0 2023-12-01T22:18:47.048 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:47.044+0000 7fe2f0a30700 1 -- 172.21.15.73:0/2567327773 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2ec0f4fc0 con 0x7fe2ec0601e0 2023-12-01T22:18:47.048 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:47.044+0000 7fe2d37fe700 1 -- 172.21.15.73:0/2567327773 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe2dc013f00 con 0x7fe2ec0601e0 2023-12-01T22:18:47.048 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:47.044+0000 7fe2d37fe700 1 -- 172.21.15.73:0/2567327773 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe2dc014d90 con 0x7fe2ec0601e0 2023-12-01T22:18:47.048 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:47.044+0000 7fe2f0a30700 1 -- 172.21.15.73:0/2567327773 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2ec0f5450 con 0x7fe2ec0601e0 2023-12-01T22:18:47.048 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:47.045+0000 7fe2d37fe700 1 -- 172.21.15.73:0/2567327773 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe2dc0095f0 con 0x7fe2ec0601e0 2023-12-01T22:18:47.048 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:47.045+0000 7fe2f0a30700 1 -- 172.21.15.73:0/2567327773 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2ec059100 con 0x7fe2ec0601e0 2023-12-01T22:18:47.048 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:47.045+0000 7fe2d37fe700 1 --2- 172.21.15.73:0/2567327773 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fe2d402ffb0 0x7fe2d4032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:18:47.048 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:47.045+0000 7fe2d37fe700 1 -- 172.21.15.73:0/2567327773 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe2dc040fe0 con 0x7fe2ec0601e0 2023-12-01T22:18:47.048 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:47.046+0000 7fe2ea59c700 1 --2- 172.21.15.73:0/2567327773 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fe2d402ffb0 0x7fe2d4032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:18:47.048 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:47.046+0000 7fe2ea59c700 1 --2- 172.21.15.73:0/2567327773 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fe2d402ffb0 0x7fe2d4032470 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7fe2e00096a0 tx=0x7fe2e0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:18:47.052 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:47.050+0000 7fe2d37fe700 1 -- 172.21.15.73:0/2567327773 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe2dc06d2b0 con 0x7fe2ec0601e0 2023-12-01T22:18:47.255 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:47.253+0000 7fe2f0a30700 1 -- 172.21.15.73:0/2567327773 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe2ec0fbcd0 con 0x7fe2ec0601e0 2023-12-01T22:18:47.256 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:47.254+0000 7fe2d37fe700 1 -- 172.21.15.73:0/2567327773 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe2dc014420 con 0x7fe2ec0601e0 2023-12-01T22:18:47.256 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:18:47.256 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:18:47.257 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:47.255+0000 7fe2f0a30700 1 -- 172.21.15.73:0/2567327773 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fe2d402ffb0 msgr2=0x7fe2d4032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:18:47.257 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:47.256+0000 7fe2f0a30700 1 --2- 172.21.15.73:0/2567327773 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fe2d402ffb0 0x7fe2d4032470 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7fe2e00096a0 tx=0x7fe2e0006b40).stop 2023-12-01T22:18:47.258 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:47.256+0000 7fe2f0a30700 1 -- 172.21.15.73:0/2567327773 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2ec0601e0 msgr2=0x7fe2ec0f69a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:18:47.258 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:47.256+0000 7fe2f0a30700 1 --2- 172.21.15.73:0/2567327773 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2ec0601e0 0x7fe2ec0f69a0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7fe2dc013510 tx=0x7fe2dc00b040).stop 2023-12-01T22:18:47.258 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:47.256+0000 7fe2f0a30700 1 -- 172.21.15.73:0/2567327773 shutdown_connections 2023-12-01T22:18:47.258 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:47.256+0000 7fe2f0a30700 1 --2- 172.21.15.73:0/2567327773 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fe2d402ffb0 0x7fe2d4032470 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:18:47.258 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:47.256+0000 7fe2f0a30700 1 --2- 172.21.15.73:0/2567327773 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2ec0601e0 0x7fe2ec0f69a0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:18:47.258 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:47.256+0000 7fe2f0a30700 1 -- 172.21.15.73:0/2567327773 >> 172.21.15.73:0/2567327773 conn(0x7fe2ec0f09a0 msgr2=0x7fe2ec0f1420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:18:47.258 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:47.256+0000 7fe2f0a30700 1 -- 172.21.15.73:0/2567327773 shutdown_connections 2023-12-01T22:18:47.258 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:47.256+0000 7fe2f0a30700 1 -- 172.21.15.73:0/2567327773 wait complete. 2023-12-01T22:18:47.258 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:18:48.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:18:47 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:18:46.693331+0000 mgr.smithi029.loxrfu (mgr.14164) 235 : cluster [DBG] pgmap v221: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:18:48.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:18:47 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:18:47.255268+0000 mon.smithi029 (mon.0) 293 : audit [DBG] from='client.? 172.21.15.73:0/2567327773' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:18:48.647 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:18:48.647 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:18:49.873 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:49.871+0000 7ffa17228700 1 -- 172.21.15.73:0/196876927 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa10060220 msgr2=0x7ffa100f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:18:49.874 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:49.871+0000 7ffa17228700 1 --2- 172.21.15.73:0/196876927 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa10060220 0x7ffa100f9150 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 rx=0x7ffa04009fb0 tx=0x7ffa04009b30).stop 2023-12-01T22:18:49.874 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:49.871+0000 7ffa17228700 1 -- 172.21.15.73:0/196876927 shutdown_connections 2023-12-01T22:18:49.874 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:49.871+0000 7ffa17228700 1 --2- 172.21.15.73:0/196876927 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa10060220 0x7ffa100f9150 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:18:49.874 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:49.871+0000 7ffa17228700 1 -- 172.21.15.73:0/196876927 >> 172.21.15.73:0/196876927 conn(0x7ffa100f0b70 msgr2=0x7ffa100f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:18:49.874 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:49.871+0000 7ffa17228700 1 -- 172.21.15.73:0/196876927 shutdown_connections 2023-12-01T22:18:49.874 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:49.871+0000 7ffa17228700 1 -- 172.21.15.73:0/196876927 wait complete. 2023-12-01T22:18:49.874 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:49.872+0000 7ffa17228700 1 Processor -- start 2023-12-01T22:18:49.874 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:49.872+0000 7ffa17228700 1 -- start start 2023-12-01T22:18:49.874 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:49.872+0000 7ffa17228700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa10060220 0x7ffa101066a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:18:49.874 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:49.872+0000 7ffa17228700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa10106b90 con 0x7ffa10060220 2023-12-01T22:18:49.876 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:49.872+0000 7ffa14fc4700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa10060220 0x7ffa101066a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:18:49.876 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:49.872+0000 7ffa14fc4700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa10060220 0x7ffa101066a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:38632/0 (socket says 172.21.15.73:38632) 2023-12-01T22:18:49.876 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:49.872+0000 7ffa14fc4700 1 -- 172.21.15.73:0/2797204436 learned_addr learned my addr 172.21.15.73:0/2797204436 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:18:49.876 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:49.873+0000 7ffa14fc4700 1 -- 172.21.15.73:0/2797204436 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa04005040 con 0x7ffa10060220 2023-12-01T22:18:49.876 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:49.873+0000 7ffa14fc4700 1 --2- 172.21.15.73:0/2797204436 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa10060220 0x7ffa101066a0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7ffa04013510 tx=0x7ffa0400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:18:49.876 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:49.873+0000 7ffa0d7fa700 1 -- 172.21.15.73:0/2797204436 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffa04009320 con 0x7ffa10060220 2023-12-01T22:18:49.876 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:49.873+0000 7ffa17228700 1 -- 172.21.15.73:0/2797204436 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa10106d90 con 0x7ffa10060220 2023-12-01T22:18:49.876 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:49.873+0000 7ffa0d7fa700 1 -- 172.21.15.73:0/2797204436 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ffa04013f00 con 0x7ffa10060220 2023-12-01T22:18:49.876 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:49.873+0000 7ffa0d7fa700 1 -- 172.21.15.73:0/2797204436 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffa04014dc0 con 0x7ffa10060220 2023-12-01T22:18:49.876 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:49.873+0000 7ffa17228700 1 -- 172.21.15.73:0/2797204436 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa10107220 con 0x7ffa10060220 2023-12-01T22:18:49.876 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:49.874+0000 7ffa0d7fa700 1 -- 172.21.15.73:0/2797204436 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ffa04014420 con 0x7ffa10060220 2023-12-01T22:18:49.877 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:49.874+0000 7ffa17228700 1 -- 172.21.15.73:0/2797204436 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa10045ad0 con 0x7ffa10060220 2023-12-01T22:18:49.877 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:49.874+0000 7ffa0d7fa700 1 --2- 172.21.15.73:0/2797204436 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7ffa0002ff60 0x7ffa00032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:18:49.877 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:49.874+0000 7ffa0d7fa700 1 -- 172.21.15.73:0/2797204436 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ffa040410f0 con 0x7ffa10060220 2023-12-01T22:18:49.877 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:49.874+0000 7ffa0ffff700 1 --2- 172.21.15.73:0/2797204436 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7ffa0002ff60 0x7ffa00032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:18:49.877 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:49.875+0000 7ffa0ffff700 1 --2- 172.21.15.73:0/2797204436 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7ffa0002ff60 0x7ffa00032420 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7ff9fc0096a0 tx=0x7ff9fc006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:18:49.880 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:49.879+0000 7ffa0d7fa700 1 -- 172.21.15.73:0/2797204436 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ffa040095f0 con 0x7ffa10060220 2023-12-01T22:18:50.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:18:49 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:18:48.693723+0000 mgr.smithi029.loxrfu (mgr.14164) 236 : cluster [DBG] pgmap v222: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:18:50.087 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:50.085+0000 7ffa17228700 1 -- 172.21.15.73:0/2797204436 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ffa101080d0 con 0x7ffa10060220 2023-12-01T22:18:50.088 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:50.086+0000 7ffa0d7fa700 1 -- 172.21.15.73:0/2797204436 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ffa0401cb10 con 0x7ffa10060220 2023-12-01T22:18:50.088 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:18:50.088 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:18:50.089 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:50.087+0000 7ffa17228700 1 -- 172.21.15.73:0/2797204436 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7ffa0002ff60 msgr2=0x7ffa00032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:18:50.089 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:50.087+0000 7ffa17228700 1 --2- 172.21.15.73:0/2797204436 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7ffa0002ff60 0x7ffa00032420 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7ff9fc0096a0 tx=0x7ff9fc006b40).stop 2023-12-01T22:18:50.089 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:50.087+0000 7ffa17228700 1 -- 172.21.15.73:0/2797204436 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa10060220 msgr2=0x7ffa101066a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:18:50.089 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:50.087+0000 7ffa17228700 1 --2- 172.21.15.73:0/2797204436 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa10060220 0x7ffa101066a0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7ffa04013510 tx=0x7ffa0400b040).stop 2023-12-01T22:18:50.089 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:50.088+0000 7ffa17228700 1 -- 172.21.15.73:0/2797204436 shutdown_connections 2023-12-01T22:18:50.090 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:50.088+0000 7ffa17228700 1 --2- 172.21.15.73:0/2797204436 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7ffa0002ff60 0x7ffa00032420 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:18:50.090 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:50.088+0000 7ffa17228700 1 --2- 172.21.15.73:0/2797204436 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa10060220 0x7ffa101066a0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:18:50.090 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:50.088+0000 7ffa17228700 1 -- 172.21.15.73:0/2797204436 >> 172.21.15.73:0/2797204436 conn(0x7ffa100f0b70 msgr2=0x7ffa100f16a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:18:50.090 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:50.088+0000 7ffa17228700 1 -- 172.21.15.73:0/2797204436 shutdown_connections 2023-12-01T22:18:50.090 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:50.088+0000 7ffa17228700 1 -- 172.21.15.73:0/2797204436 wait complete. 2023-12-01T22:18:50.090 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:18:51.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:18:50 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:18:50.087496+0000 mon.smithi029 (mon.0) 294 : audit [DBG] from='client.? 172.21.15.73:0/2797204436' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:18:51.481 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:18:51.481 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:18:52.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:18:51 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:18:50.694034+0000 mgr.smithi029.loxrfu (mgr.14164) 237 : cluster [DBG] pgmap v223: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:18:52.739 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:52.737+0000 7f245f4f8700 1 -- 172.21.15.73:0/1780420976 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f24580f62a0 msgr2=0x7f24580f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:18:52.740 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:52.737+0000 7f245f4f8700 1 --2- 172.21.15.73:0/1780420976 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f24580f62a0 0x7f24580f66c0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 rx=0x7f2448009fb0 tx=0x7f2448009b30).stop 2023-12-01T22:18:52.740 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:52.737+0000 7f245f4f8700 1 -- 172.21.15.73:0/1780420976 shutdown_connections 2023-12-01T22:18:52.740 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:52.737+0000 7f245f4f8700 1 --2- 172.21.15.73:0/1780420976 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f24580f62a0 0x7f24580f66c0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:18:52.740 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:52.737+0000 7f245f4f8700 1 -- 172.21.15.73:0/1780420976 >> 172.21.15.73:0/1780420976 conn(0x7f24580f1aa0 msgr2=0x7f24580f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:18:52.740 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:52.737+0000 7f245f4f8700 1 -- 172.21.15.73:0/1780420976 shutdown_connections 2023-12-01T22:18:52.740 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:52.737+0000 7f245f4f8700 1 -- 172.21.15.73:0/1780420976 wait complete. 2023-12-01T22:18:52.740 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:52.737+0000 7f245f4f8700 1 Processor -- start 2023-12-01T22:18:52.740 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:52.737+0000 7f245f4f8700 1 -- start start 2023-12-01T22:18:52.740 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:52.737+0000 7f245f4f8700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f24580f62a0 0x7f24581064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:18:52.740 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:52.738+0000 7f245f4f8700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24581069a0 con 0x7f24580f62a0 2023-12-01T22:18:52.740 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:52.738+0000 7f245e4f6700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f24580f62a0 0x7f24581064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:18:52.740 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:52.738+0000 7f245e4f6700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f24580f62a0 0x7f24581064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:38642/0 (socket says 172.21.15.73:38642) 2023-12-01T22:18:52.740 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:52.738+0000 7f245e4f6700 1 -- 172.21.15.73:0/1363224798 learned_addr learned my addr 172.21.15.73:0/1363224798 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:18:52.740 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:52.738+0000 7f245e4f6700 1 -- 172.21.15.73:0/1363224798 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2448005040 con 0x7f24580f62a0 2023-12-01T22:18:52.742 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:52.738+0000 7f245e4f6700 1 --2- 172.21.15.73:0/1363224798 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f24580f62a0 0x7f24581064b0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7f2448013510 tx=0x7f244800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:18:52.742 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:52.738+0000 7f244effd700 1 -- 172.21.15.73:0/1363224798 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2448009320 con 0x7f24580f62a0 2023-12-01T22:18:52.742 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:52.738+0000 7f245f4f8700 1 -- 172.21.15.73:0/1363224798 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2458106ba0 con 0x7f24580f62a0 2023-12-01T22:18:52.742 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:52.738+0000 7f244effd700 1 -- 172.21.15.73:0/1363224798 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2448013f00 con 0x7f24580f62a0 2023-12-01T22:18:52.742 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:52.738+0000 7f244effd700 1 -- 172.21.15.73:0/1363224798 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2448014dc0 con 0x7f24580f62a0 2023-12-01T22:18:52.742 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:52.738+0000 7f245f4f8700 1 -- 172.21.15.73:0/1363224798 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2458107030 con 0x7f24580f62a0 2023-12-01T22:18:52.742 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:52.739+0000 7f244effd700 1 -- 172.21.15.73:0/1363224798 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f24480095f0 con 0x7f24580f62a0 2023-12-01T22:18:52.742 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:52.739+0000 7f245f4f8700 1 -- 172.21.15.73:0/1363224798 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2458045ad0 con 0x7f24580f62a0 2023-12-01T22:18:52.742 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:52.739+0000 7f244effd700 1 --2- 172.21.15.73:0/1363224798 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f244402ff60 0x7f2444032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:18:52.742 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:52.739+0000 7f244effd700 1 -- 172.21.15.73:0/1363224798 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2448041030 con 0x7f24580f62a0 2023-12-01T22:18:52.742 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:52.740+0000 7f245dcf5700 1 --2- 172.21.15.73:0/1363224798 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f244402ff60 0x7f2444032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:18:52.742 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:52.740+0000 7f245dcf5700 1 --2- 172.21.15.73:0/1363224798 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f244402ff60 0x7f2444032420 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f24540096a0 tx=0x7f2454006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:18:52.746 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:52.744+0000 7f244effd700 1 -- 172.21.15.73:0/1363224798 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2448014420 con 0x7f24580f62a0 2023-12-01T22:18:52.951 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:52.949+0000 7f245f4f8700 1 -- 172.21.15.73:0/1363224798 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2458107ee0 con 0x7f24580f62a0 2023-12-01T22:18:52.952 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:52.950+0000 7f244effd700 1 -- 172.21.15.73:0/1363224798 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2448010230 con 0x7f24580f62a0 2023-12-01T22:18:52.952 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:18:52.952 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:18:52.953 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:52.951+0000 7f245f4f8700 1 -- 172.21.15.73:0/1363224798 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f244402ff60 msgr2=0x7f2444032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:18:52.953 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:52.951+0000 7f245f4f8700 1 --2- 172.21.15.73:0/1363224798 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f244402ff60 0x7f2444032420 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f24540096a0 tx=0x7f2454006b40).stop 2023-12-01T22:18:52.953 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:52.951+0000 7f245f4f8700 1 -- 172.21.15.73:0/1363224798 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f24580f62a0 msgr2=0x7f24581064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:18:52.954 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:52.951+0000 7f245f4f8700 1 --2- 172.21.15.73:0/1363224798 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f24580f62a0 0x7f24581064b0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7f2448013510 tx=0x7f244800b040).stop 2023-12-01T22:18:52.954 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:52.951+0000 7f245f4f8700 1 -- 172.21.15.73:0/1363224798 shutdown_connections 2023-12-01T22:18:52.954 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:52.951+0000 7f245f4f8700 1 --2- 172.21.15.73:0/1363224798 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f244402ff60 0x7f2444032420 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:18:52.954 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:52.951+0000 7f245f4f8700 1 --2- 172.21.15.73:0/1363224798 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f24580f62a0 0x7f24581064b0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:18:52.954 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:52.951+0000 7f245f4f8700 1 -- 172.21.15.73:0/1363224798 >> 172.21.15.73:0/1363224798 conn(0x7f24580f1aa0 msgr2=0x7f24580f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:18:52.954 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:52.952+0000 7f245f4f8700 1 -- 172.21.15.73:0/1363224798 shutdown_connections 2023-12-01T22:18:52.954 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:52.952+0000 7f245f4f8700 1 -- 172.21.15.73:0/1363224798 wait complete. 2023-12-01T22:18:52.954 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:18:54.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:18:53 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:18:52.694352+0000 mgr.smithi029.loxrfu (mgr.14164) 238 : cluster [DBG] pgmap v224: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:18:54.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:18:53 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:18:52.951432+0000 mon.smithi029 (mon.0) 295 : audit [DBG] from='client.? 172.21.15.73:0/1363224798' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:18:54.352 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:18:54.352 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:18:55.504 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:55.501+0000 7f49996e8700 1 -- 172.21.15.73:0/1062922732 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f49940f46a0 msgr2=0x7f49940f4ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:18:55.504 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:55.501+0000 7f49996e8700 1 --2- 172.21.15.73:0/1062922732 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f49940f46a0 0x7f49940f4ac0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 rx=0x7f4984009fb0 tx=0x7f4984009b30).stop 2023-12-01T22:18:55.504 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:55.501+0000 7f49996e8700 1 -- 172.21.15.73:0/1062922732 shutdown_connections 2023-12-01T22:18:55.504 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:55.501+0000 7f49996e8700 1 --2- 172.21.15.73:0/1062922732 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f49940f46a0 0x7f49940f4ac0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:18:55.504 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:55.501+0000 7f49996e8700 1 -- 172.21.15.73:0/1062922732 >> 172.21.15.73:0/1062922732 conn(0x7f49940f0120 msgr2=0x7f49940f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:18:55.504 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:55.501+0000 7f49996e8700 1 -- 172.21.15.73:0/1062922732 shutdown_connections 2023-12-01T22:18:55.504 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:55.501+0000 7f49996e8700 1 -- 172.21.15.73:0/1062922732 wait complete. 2023-12-01T22:18:55.505 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:55.502+0000 7f49996e8700 1 Processor -- start 2023-12-01T22:18:55.505 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:55.502+0000 7f49996e8700 1 -- start start 2023-12-01T22:18:55.505 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:55.502+0000 7f49996e8700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4994106820 0x7f4994106c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:18:55.505 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:55.502+0000 7f49996e8700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4994107130 con 0x7f4994106820 2023-12-01T22:18:55.505 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:55.502+0000 7f4993fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4994106820 0x7f4994106c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:18:55.505 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:55.502+0000 7f4993fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4994106820 0x7f4994106c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:36530/0 (socket says 172.21.15.73:36530) 2023-12-01T22:18:55.505 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:55.502+0000 7f4993fff700 1 -- 172.21.15.73:0/3542909745 learned_addr learned my addr 172.21.15.73:0/3542909745 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:18:55.506 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:55.503+0000 7f4993fff700 1 -- 172.21.15.73:0/3542909745 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4984005040 con 0x7f4994106820 2023-12-01T22:18:55.506 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:55.503+0000 7f4993fff700 1 --2- 172.21.15.73:0/3542909745 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4994106820 0x7f4994106c40 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7f4984009f80 tx=0x7f4984009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:18:55.507 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:55.503+0000 7f4990ff9700 1 -- 172.21.15.73:0/3542909745 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f49840136c0 con 0x7f4994106820 2023-12-01T22:18:55.507 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:55.503+0000 7f49996e8700 1 -- 172.21.15.73:0/3542909745 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4994107330 con 0x7f4994106820 2023-12-01T22:18:55.507 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:55.503+0000 7f4990ff9700 1 -- 172.21.15.73:0/3542909745 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f49840095f0 con 0x7f4994106820 2023-12-01T22:18:55.507 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:55.503+0000 7f4990ff9700 1 -- 172.21.15.73:0/3542909745 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4984014420 con 0x7f4994106820 2023-12-01T22:18:55.507 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:55.503+0000 7f49996e8700 1 -- 172.21.15.73:0/3542909745 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4994107e80 con 0x7f4994106820 2023-12-01T22:18:55.507 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:55.504+0000 7f4990ff9700 1 -- 172.21.15.73:0/3542909745 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4984025060 con 0x7f4994106820 2023-12-01T22:18:55.507 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:55.504+0000 7f49996e8700 1 -- 172.21.15.73:0/3542909745 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4994059100 con 0x7f4994106820 2023-12-01T22:18:55.507 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:55.504+0000 7f4990ff9700 1 --2- 172.21.15.73:0/3542909745 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f497c02ff60 0x7f497c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:18:55.507 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:55.504+0000 7f4990ff9700 1 -- 172.21.15.73:0/3542909745 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4984040d80 con 0x7f4994106820 2023-12-01T22:18:55.507 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:55.504+0000 7f49937fe700 1 --2- 172.21.15.73:0/3542909745 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f497c02ff60 0x7f497c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:18:55.507 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:55.505+0000 7f49937fe700 1 --2- 172.21.15.73:0/3542909745 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f497c02ff60 0x7f497c032420 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f49880096a0 tx=0x7f4988006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:18:55.510 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:55.509+0000 7f4990ff9700 1 -- 172.21.15.73:0/3542909745 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f498401c950 con 0x7f4994106820 2023-12-01T22:18:55.716 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:55.714+0000 7f49996e8700 1 -- 172.21.15.73:0/3542909745 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f49940fc090 con 0x7f4994106820 2023-12-01T22:18:55.717 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:55.715+0000 7f4990ff9700 1 -- 172.21.15.73:0/3542909745 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4984042090 con 0x7f4994106820 2023-12-01T22:18:55.717 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:18:55.717 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:18:55.718 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:55.717+0000 7f49996e8700 1 -- 172.21.15.73:0/3542909745 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f497c02ff60 msgr2=0x7f497c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:18:55.719 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:55.717+0000 7f49996e8700 1 --2- 172.21.15.73:0/3542909745 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f497c02ff60 0x7f497c032420 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f49880096a0 tx=0x7f4988006b40).stop 2023-12-01T22:18:55.719 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:55.717+0000 7f49996e8700 1 -- 172.21.15.73:0/3542909745 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4994106820 msgr2=0x7f4994106c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:18:55.719 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:55.717+0000 7f49996e8700 1 --2- 172.21.15.73:0/3542909745 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4994106820 0x7f4994106c40 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7f4984009f80 tx=0x7f4984009b30).stop 2023-12-01T22:18:55.719 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:55.717+0000 7f49996e8700 1 -- 172.21.15.73:0/3542909745 shutdown_connections 2023-12-01T22:18:55.719 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:55.717+0000 7f49996e8700 1 --2- 172.21.15.73:0/3542909745 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f497c02ff60 0x7f497c032420 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:18:55.719 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:55.717+0000 7f49996e8700 1 --2- 172.21.15.73:0/3542909745 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4994106820 0x7f4994106c40 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:18:55.719 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:55.717+0000 7f49996e8700 1 -- 172.21.15.73:0/3542909745 >> 172.21.15.73:0/3542909745 conn(0x7f49940f0120 msgr2=0x7f49940f0c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:18:55.719 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:55.717+0000 7f49996e8700 1 -- 172.21.15.73:0/3542909745 shutdown_connections 2023-12-01T22:18:55.719 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:55.717+0000 7f49996e8700 1 -- 172.21.15.73:0/3542909745 wait complete. 2023-12-01T22:18:55.720 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:18:56.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:18:55 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:18:54.694712+0000 mgr.smithi029.loxrfu (mgr.14164) 239 : cluster [DBG] pgmap v225: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:18:56.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:18:55 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:18:55.716602+0000 mon.smithi029 (mon.0) 296 : audit [DBG] from='client.? 172.21.15.73:0/3542909745' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:18:57.120 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:18:57.120 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:18:58.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:18:57 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:18:56.695033+0000 mgr.smithi029.loxrfu (mgr.14164) 240 : cluster [DBG] pgmap v226: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:18:58.306 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:58.304+0000 7f0639b54700 1 -- 172.21.15.73:0/1786328951 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f06340f4800 msgr2=0x7f06340f4c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:18:58.306 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:58.304+0000 7f0639b54700 1 --2- 172.21.15.73:0/1786328951 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f06340f4800 0x7f06340f4c20 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 rx=0x7f0624009fb0 tx=0x7f0624009b30).stop 2023-12-01T22:18:58.306 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:58.304+0000 7f0639b54700 1 -- 172.21.15.73:0/1786328951 shutdown_connections 2023-12-01T22:18:58.306 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:58.304+0000 7f0639b54700 1 --2- 172.21.15.73:0/1786328951 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f06340f4800 0x7f06340f4c20 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:18:58.306 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:58.304+0000 7f0639b54700 1 -- 172.21.15.73:0/1786328951 >> 172.21.15.73:0/1786328951 conn(0x7f06340f0260 msgr2=0x7f06340f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:18:58.306 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:58.304+0000 7f0639b54700 1 -- 172.21.15.73:0/1786328951 shutdown_connections 2023-12-01T22:18:58.306 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:58.304+0000 7f0639b54700 1 -- 172.21.15.73:0/1786328951 wait complete. 2023-12-01T22:18:58.307 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:58.304+0000 7f0639b54700 1 Processor -- start 2023-12-01T22:18:58.307 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:58.304+0000 7f0639b54700 1 -- start start 2023-12-01T22:18:58.307 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:58.305+0000 7f0639b54700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f06340f4800 0x7f06341066b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:18:58.307 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:58.305+0000 7f0639b54700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0634106ba0 con 0x7f06340f4800 2023-12-01T22:18:58.308 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:58.305+0000 7f06337fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f06340f4800 0x7f06341066b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:18:58.308 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:58.305+0000 7f06337fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f06340f4800 0x7f06341066b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:36540/0 (socket says 172.21.15.73:36540) 2023-12-01T22:18:58.308 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:58.305+0000 7f06337fe700 1 -- 172.21.15.73:0/643933195 learned_addr learned my addr 172.21.15.73:0/643933195 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:18:58.308 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:58.305+0000 7f06337fe700 1 -- 172.21.15.73:0/643933195 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0624005040 con 0x7f06340f4800 2023-12-01T22:18:58.308 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:58.305+0000 7f06337fe700 1 --2- 172.21.15.73:0/643933195 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f06340f4800 0x7f06341066b0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7f0624013510 tx=0x7f062400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:18:58.308 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:58.306+0000 7f061bfff700 1 -- 172.21.15.73:0/643933195 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0624009320 con 0x7f06340f4800 2023-12-01T22:18:58.308 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:58.306+0000 7f0639b54700 1 -- 172.21.15.73:0/643933195 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0634106da0 con 0x7f06340f4800 2023-12-01T22:18:58.308 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:58.306+0000 7f061bfff700 1 -- 172.21.15.73:0/643933195 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0624013f00 con 0x7f06340f4800 2023-12-01T22:18:58.308 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:58.306+0000 7f061bfff700 1 -- 172.21.15.73:0/643933195 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0624014dc0 con 0x7f06340f4800 2023-12-01T22:18:58.308 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:58.306+0000 7f0639b54700 1 -- 172.21.15.73:0/643933195 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0634107230 con 0x7f06340f4800 2023-12-01T22:18:58.308 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:58.306+0000 7f061bfff700 1 -- 172.21.15.73:0/643933195 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f062401d3d0 con 0x7f06340f4800 2023-12-01T22:18:58.309 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:58.307+0000 7f061bfff700 1 --2- 172.21.15.73:0/643933195 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f061c02ff60 0x7f061c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:18:58.309 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:58.307+0000 7f061bfff700 1 -- 172.21.15.73:0/643933195 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0624041000 con 0x7f06340f4800 2023-12-01T22:18:58.309 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:58.307+0000 7f0639b54700 1 -- 172.21.15.73:0/643933195 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0634045ad0 con 0x7f06340f4800 2023-12-01T22:18:58.309 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:58.307+0000 7f0632ffd700 1 --2- 172.21.15.73:0/643933195 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f061c02ff60 0x7f061c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:18:58.309 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:58.307+0000 7f0632ffd700 1 --2- 172.21.15.73:0/643933195 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f061c02ff60 0x7f061c032420 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f06280096a0 tx=0x7f0628006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:18:58.313 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:58.312+0000 7f061bfff700 1 -- 172.21.15.73:0/643933195 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0624012920 con 0x7f06340f4800 2023-12-01T22:18:58.521 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:58.519+0000 7f0639b54700 1 -- 172.21.15.73:0/643933195 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0634108080 con 0x7f06340f4800 2023-12-01T22:18:58.521 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:58.520+0000 7f061bfff700 1 -- 172.21.15.73:0/643933195 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0624012330 con 0x7f06340f4800 2023-12-01T22:18:58.522 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:18:58.522 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:18:58.523 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:58.521+0000 7f0639b54700 1 -- 172.21.15.73:0/643933195 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f061c02ff60 msgr2=0x7f061c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:18:58.523 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:58.521+0000 7f0639b54700 1 --2- 172.21.15.73:0/643933195 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f061c02ff60 0x7f061c032420 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f06280096a0 tx=0x7f0628006b40).stop 2023-12-01T22:18:58.524 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:58.522+0000 7f0639b54700 1 -- 172.21.15.73:0/643933195 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f06340f4800 msgr2=0x7f06341066b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:18:58.524 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:58.522+0000 7f0639b54700 1 --2- 172.21.15.73:0/643933195 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f06340f4800 0x7f06341066b0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7f0624013510 tx=0x7f062400b040).stop 2023-12-01T22:18:58.524 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:58.522+0000 7f0639b54700 1 -- 172.21.15.73:0/643933195 shutdown_connections 2023-12-01T22:18:58.524 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:58.522+0000 7f0639b54700 1 --2- 172.21.15.73:0/643933195 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f061c02ff60 0x7f061c032420 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:18:58.524 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:58.522+0000 7f0639b54700 1 --2- 172.21.15.73:0/643933195 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f06340f4800 0x7f06341066b0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:18:58.524 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:58.522+0000 7f0639b54700 1 -- 172.21.15.73:0/643933195 >> 172.21.15.73:0/643933195 conn(0x7f06340f0260 msgr2=0x7f06340f0da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:18:58.524 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:58.522+0000 7f0639b54700 1 -- 172.21.15.73:0/643933195 shutdown_connections 2023-12-01T22:18:58.524 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:18:58.522+0000 7f0639b54700 1 -- 172.21.15.73:0/643933195 wait complete. 2023-12-01T22:18:58.525 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:18:59.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:18:58 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:18:58.521443+0000 mon.smithi029 (mon.0) 297 : audit [DBG] from='client.? 172.21.15.73:0/643933195' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:18:59.920 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:18:59.920 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:19:00.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:18:59 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:18:58.695345+0000 mgr.smithi029.loxrfu (mgr.14164) 241 : cluster [DBG] pgmap v227: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:19:01.068 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:01.066+0000 7f97cda01700 1 -- 172.21.15.73:0/4164954379 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97c80f62c0 msgr2=0x7f97c80f66e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:19:01.069 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:01.066+0000 7f97cda01700 1 --2- 172.21.15.73:0/4164954379 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97c80f62c0 0x7f97c80f66e0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 rx=0x7f97bc009fb0 tx=0x7f97bc009b30).stop 2023-12-01T22:19:01.069 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:01.066+0000 7f97cda01700 1 -- 172.21.15.73:0/4164954379 shutdown_connections 2023-12-01T22:19:01.069 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:01.066+0000 7f97cda01700 1 --2- 172.21.15.73:0/4164954379 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97c80f62c0 0x7f97c80f66e0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:19:01.069 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:01.066+0000 7f97cda01700 1 -- 172.21.15.73:0/4164954379 >> 172.21.15.73:0/4164954379 conn(0x7f97c80f1a20 msgr2=0x7f97c80f3e40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:19:01.069 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:01.066+0000 7f97cda01700 1 -- 172.21.15.73:0/4164954379 shutdown_connections 2023-12-01T22:19:01.069 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:01.066+0000 7f97cda01700 1 -- 172.21.15.73:0/4164954379 wait complete. 2023-12-01T22:19:01.069 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:01.067+0000 7f97cda01700 1 Processor -- start 2023-12-01T22:19:01.069 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:01.067+0000 7f97cda01700 1 -- start start 2023-12-01T22:19:01.069 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:01.067+0000 7f97cda01700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97c80f62c0 0x7f97c8106580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:19:01.069 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:01.067+0000 7f97cda01700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97c8106a70 con 0x7f97c80f62c0 2023-12-01T22:19:01.071 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:01.067+0000 7f97c6ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97c80f62c0 0x7f97c8106580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:19:01.071 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:01.067+0000 7f97c6ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97c80f62c0 0x7f97c8106580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:36556/0 (socket says 172.21.15.73:36556) 2023-12-01T22:19:01.071 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:01.067+0000 7f97c6ffd700 1 -- 172.21.15.73:0/3689755217 learned_addr learned my addr 172.21.15.73:0/3689755217 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:19:01.071 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:01.068+0000 7f97c6ffd700 1 -- 172.21.15.73:0/3689755217 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97bc005040 con 0x7f97c80f62c0 2023-12-01T22:19:01.071 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:01.068+0000 7f97c6ffd700 1 --2- 172.21.15.73:0/3689755217 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97c80f62c0 0x7f97c8106580 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7f97bc013510 tx=0x7f97bc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:19:01.071 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:01.068+0000 7f97af7fe700 1 -- 172.21.15.73:0/3689755217 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f97bc009320 con 0x7f97c80f62c0 2023-12-01T22:19:01.071 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:01.068+0000 7f97cda01700 1 -- 172.21.15.73:0/3689755217 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97c8106c70 con 0x7f97c80f62c0 2023-12-01T22:19:01.071 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:01.068+0000 7f97af7fe700 1 -- 172.21.15.73:0/3689755217 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f97bc013f00 con 0x7f97c80f62c0 2023-12-01T22:19:01.071 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:01.068+0000 7f97af7fe700 1 -- 172.21.15.73:0/3689755217 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f97bc014d90 con 0x7f97c80f62c0 2023-12-01T22:19:01.071 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:01.068+0000 7f97cda01700 1 -- 172.21.15.73:0/3689755217 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97c8107100 con 0x7f97c80f62c0 2023-12-01T22:19:01.071 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:01.069+0000 7f97cda01700 1 -- 172.21.15.73:0/3689755217 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97c8045ad0 con 0x7f97c80f62c0 2023-12-01T22:19:01.071 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:01.069+0000 7f97af7fe700 1 -- 172.21.15.73:0/3689755217 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f97bc0095f0 con 0x7f97c80f62c0 2023-12-01T22:19:01.071 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:01.069+0000 7f97af7fe700 1 --2- 172.21.15.73:0/3689755217 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f97b002ff60 0x7f97b0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:19:01.072 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:01.069+0000 7f97af7fe700 1 -- 172.21.15.73:0/3689755217 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f97bc041000 con 0x7f97c80f62c0 2023-12-01T22:19:01.072 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:01.069+0000 7f97c67fc700 1 --2- 172.21.15.73:0/3689755217 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f97b002ff60 0x7f97b0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:19:01.072 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:01.069+0000 7f97c67fc700 1 --2- 172.21.15.73:0/3689755217 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f97b002ff60 0x7f97b0032420 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f97b80096a0 tx=0x7f97b8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:19:01.075 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:01.074+0000 7f97af7fe700 1 -- 172.21.15.73:0/3689755217 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f97bc01d7f0 con 0x7f97c80f62c0 2023-12-01T22:19:01.280 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:01.279+0000 7f97cda01700 1 -- 172.21.15.73:0/3689755217 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f97c8107f00 con 0x7f97c80f62c0 2023-12-01T22:19:01.281 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:01.279+0000 7f97af7fe700 1 -- 172.21.15.73:0/3689755217 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f97bc010230 con 0x7f97c80f62c0 2023-12-01T22:19:01.281 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:19:01.281 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:19:01.282 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:01.281+0000 7f97cda01700 1 -- 172.21.15.73:0/3689755217 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f97b002ff60 msgr2=0x7f97b0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:19:01.283 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:01.281+0000 7f97cda01700 1 --2- 172.21.15.73:0/3689755217 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f97b002ff60 0x7f97b0032420 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f97b80096a0 tx=0x7f97b8006b40).stop 2023-12-01T22:19:01.283 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:01.281+0000 7f97cda01700 1 -- 172.21.15.73:0/3689755217 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97c80f62c0 msgr2=0x7f97c8106580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:19:01.283 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:01.281+0000 7f97cda01700 1 --2- 172.21.15.73:0/3689755217 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97c80f62c0 0x7f97c8106580 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7f97bc013510 tx=0x7f97bc00b040).stop 2023-12-01T22:19:01.283 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:01.281+0000 7f97cda01700 1 -- 172.21.15.73:0/3689755217 shutdown_connections 2023-12-01T22:19:01.283 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:01.281+0000 7f97cda01700 1 --2- 172.21.15.73:0/3689755217 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f97b002ff60 0x7f97b0032420 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:19:01.283 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:01.281+0000 7f97cda01700 1 --2- 172.21.15.73:0/3689755217 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97c80f62c0 0x7f97c8106580 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:19:01.283 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:01.281+0000 7f97cda01700 1 -- 172.21.15.73:0/3689755217 >> 172.21.15.73:0/3689755217 conn(0x7f97c80f1a20 msgr2=0x7f97c80f2580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:19:01.283 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:01.281+0000 7f97cda01700 1 -- 172.21.15.73:0/3689755217 shutdown_connections 2023-12-01T22:19:01.283 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:01.281+0000 7f97cda01700 1 -- 172.21.15.73:0/3689755217 wait complete. 2023-12-01T22:19:01.284 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:19:01.776 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:19:01 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:19:00.695706+0000 mgr.smithi029.loxrfu (mgr.14164) 242 : cluster [DBG] pgmap v228: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:19:01.776 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:19:01 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:19:01.280830+0000 mon.smithi029 (mon.0) 298 : audit [DBG] from='client.? 172.21.15.73:0/3689755217' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:19:02.668 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:19:02.668 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:19:03.859 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:03.857+0000 7f38594aa700 1 -- 172.21.15.73:0/3622535999 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38540f4740 msgr2=0x7f38540f4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:19:03.859 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:03.857+0000 7f38594aa700 1 --2- 172.21.15.73:0/3622535999 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38540f4740 0x7f38540f4b60 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 rx=0x7f3848009fb0 tx=0x7f3848009b30).stop 2023-12-01T22:19:03.859 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:03.857+0000 7f38594aa700 1 -- 172.21.15.73:0/3622535999 shutdown_connections 2023-12-01T22:19:03.859 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:03.857+0000 7f38594aa700 1 --2- 172.21.15.73:0/3622535999 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38540f4740 0x7f38540f4b60 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:19:03.859 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:03.857+0000 7f38594aa700 1 -- 172.21.15.73:0/3622535999 >> 172.21.15.73:0/3622535999 conn(0x7f38540f0260 msgr2=0x7f38540f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:19:03.859 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:03.857+0000 7f38594aa700 1 -- 172.21.15.73:0/3622535999 shutdown_connections 2023-12-01T22:19:03.860 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:03.857+0000 7f38594aa700 1 -- 172.21.15.73:0/3622535999 wait complete. 2023-12-01T22:19:03.861 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:03.858+0000 7f38594aa700 1 Processor -- start 2023-12-01T22:19:03.861 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:03.858+0000 7f38594aa700 1 -- start start 2023-12-01T22:19:03.861 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:03.858+0000 7f38594aa700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38540f4740 0x7f38541066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:19:03.861 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:03.858+0000 7f38594aa700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3854106bc0 con 0x7f38540f4740 2023-12-01T22:19:03.861 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:03.858+0000 7f3852ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38540f4740 0x7f38541066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:19:03.861 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:03.858+0000 7f3852ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38540f4740 0x7f38541066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:36582/0 (socket says 172.21.15.73:36582) 2023-12-01T22:19:03.861 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:03.858+0000 7f3852ffd700 1 -- 172.21.15.73:0/1283800966 learned_addr learned my addr 172.21.15.73:0/1283800966 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:19:03.862 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:03.859+0000 7f3852ffd700 1 -- 172.21.15.73:0/1283800966 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3848005040 con 0x7f38540f4740 2023-12-01T22:19:03.862 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:03.859+0000 7f3852ffd700 1 --2- 172.21.15.73:0/1283800966 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38540f4740 0x7f38541066d0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7f3848013510 tx=0x7f384800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:19:03.862 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:03.859+0000 7f383b7fe700 1 -- 172.21.15.73:0/1283800966 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3848009320 con 0x7f38540f4740 2023-12-01T22:19:03.862 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:03.859+0000 7f38594aa700 1 -- 172.21.15.73:0/1283800966 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3854106dc0 con 0x7f38540f4740 2023-12-01T22:19:03.862 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:03.859+0000 7f383b7fe700 1 -- 172.21.15.73:0/1283800966 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3848013f00 con 0x7f38540f4740 2023-12-01T22:19:03.862 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:03.859+0000 7f383b7fe700 1 -- 172.21.15.73:0/1283800966 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3848014d90 con 0x7f38540f4740 2023-12-01T22:19:03.862 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:03.859+0000 7f38594aa700 1 -- 172.21.15.73:0/1283800966 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3854107250 con 0x7f38540f4740 2023-12-01T22:19:03.863 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:03.860+0000 7f383b7fe700 1 -- 172.21.15.73:0/1283800966 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f38480263a0 con 0x7f38540f4740 2023-12-01T22:19:03.863 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:03.860+0000 7f383b7fe700 1 --2- 172.21.15.73:0/1283800966 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f383c02ffb0 0x7f383c032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:19:03.863 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:03.860+0000 7f383b7fe700 1 -- 172.21.15.73:0/1283800966 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f38480410c0 con 0x7f38540f4740 2023-12-01T22:19:03.863 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:03.860+0000 7f38594aa700 1 -- 172.21.15.73:0/1283800966 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3854045ad0 con 0x7f38540f4740 2023-12-01T22:19:03.863 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:03.860+0000 7f38527fc700 1 --2- 172.21.15.73:0/1283800966 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f383c02ffb0 0x7f383c032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:19:03.863 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:03.861+0000 7f38527fc700 1 --2- 172.21.15.73:0/1283800966 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f383c02ffb0 0x7f383c032470 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f38440096a0 tx=0x7f3844006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:19:03.866 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:03.865+0000 7f383b7fe700 1 -- 172.21.15.73:0/1283800966 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3848012920 con 0x7f38540f4740 2023-12-01T22:19:04.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:19:03 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:19:02.695990+0000 mgr.smithi029.loxrfu (mgr.14164) 243 : cluster [DBG] pgmap v229: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:19:04.072 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:04.071+0000 7f38594aa700 1 -- 172.21.15.73:0/1283800966 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f38540fbf40 con 0x7f38540f4740 2023-12-01T22:19:04.073 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:04.071+0000 7f383b7fe700 1 -- 172.21.15.73:0/1283800966 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f384802b020 con 0x7f38540f4740 2023-12-01T22:19:04.073 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:19:04.073 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:19:04.075 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:04.073+0000 7f38594aa700 1 -- 172.21.15.73:0/1283800966 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f383c02ffb0 msgr2=0x7f383c032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:19:04.075 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:04.073+0000 7f38594aa700 1 --2- 172.21.15.73:0/1283800966 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f383c02ffb0 0x7f383c032470 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f38440096a0 tx=0x7f3844006b40).stop 2023-12-01T22:19:04.075 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:04.073+0000 7f38594aa700 1 -- 172.21.15.73:0/1283800966 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38540f4740 msgr2=0x7f38541066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:19:04.075 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:04.073+0000 7f38594aa700 1 --2- 172.21.15.73:0/1283800966 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38540f4740 0x7f38541066d0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7f3848013510 tx=0x7f384800b040).stop 2023-12-01T22:19:04.075 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:04.073+0000 7f38594aa700 1 -- 172.21.15.73:0/1283800966 shutdown_connections 2023-12-01T22:19:04.075 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:04.073+0000 7f38594aa700 1 --2- 172.21.15.73:0/1283800966 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f383c02ffb0 0x7f383c032470 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:19:04.075 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:04.073+0000 7f38594aa700 1 --2- 172.21.15.73:0/1283800966 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38540f4740 0x7f38541066d0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:19:04.075 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:04.073+0000 7f38594aa700 1 -- 172.21.15.73:0/1283800966 >> 172.21.15.73:0/1283800966 conn(0x7f38540f0260 msgr2=0x7f38540f0d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:19:04.075 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:04.073+0000 7f38594aa700 1 -- 172.21.15.73:0/1283800966 shutdown_connections 2023-12-01T22:19:04.075 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:04.073+0000 7f38594aa700 1 -- 172.21.15.73:0/1283800966 wait complete. 2023-12-01T22:19:04.076 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:19:05.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:19:04 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:19:04.072911+0000 mon.smithi029 (mon.0) 299 : audit [DBG] from='client.? 172.21.15.73:0/1283800966' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:19:05.468 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:19:05.468 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:19:06.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:19:05 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:19:04.696304+0000 mgr.smithi029.loxrfu (mgr.14164) 244 : cluster [DBG] pgmap v230: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:19:06.699 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:06.695+0000 7fb15a557700 1 -- 172.21.15.73:0/3097607186 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb15406b960 msgr2=0x7fb154069e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:19:06.699 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:06.695+0000 7fb15a557700 1 --2- 172.21.15.73:0/3097607186 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb15406b960 0x7fb154069e60 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 rx=0x7fb144009fb0 tx=0x7fb144009b30).stop 2023-12-01T22:19:06.699 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:06.695+0000 7fb15a557700 1 -- 172.21.15.73:0/3097607186 shutdown_connections 2023-12-01T22:19:06.699 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:06.695+0000 7fb15a557700 1 --2- 172.21.15.73:0/3097607186 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb15406b960 0x7fb154069e60 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:19:06.699 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:06.695+0000 7fb15a557700 1 -- 172.21.15.73:0/3097607186 >> 172.21.15.73:0/3097607186 conn(0x7fb1540f1c70 msgr2=0x7fb1540f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:19:06.699 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:06.695+0000 7fb15a557700 1 -- 172.21.15.73:0/3097607186 shutdown_connections 2023-12-01T22:19:06.699 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:06.695+0000 7fb15a557700 1 -- 172.21.15.73:0/3097607186 wait complete. 2023-12-01T22:19:06.699 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:06.696+0000 7fb15a557700 1 Processor -- start 2023-12-01T22:19:06.699 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:06.696+0000 7fb15a557700 1 -- start start 2023-12-01T22:19:06.699 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:06.696+0000 7fb15a557700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb15406b960 0x7fb1541088a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:19:06.699 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:06.696+0000 7fb15a557700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb154108d90 con 0x7fb15406b960 2023-12-01T22:19:06.699 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:06.696+0000 7fb153fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb15406b960 0x7fb1541088a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:19:06.700 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:06.696+0000 7fb153fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb15406b960 0x7fb1541088a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:37730/0 (socket says 172.21.15.73:37730) 2023-12-01T22:19:06.700 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:06.697+0000 7fb153fff700 1 -- 172.21.15.73:0/3696882502 learned_addr learned my addr 172.21.15.73:0/3696882502 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:19:06.700 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:06.697+0000 7fb153fff700 1 -- 172.21.15.73:0/3696882502 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb144005040 con 0x7fb15406b960 2023-12-01T22:19:06.700 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:06.697+0000 7fb153fff700 1 --2- 172.21.15.73:0/3696882502 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb15406b960 0x7fb1541088a0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7fb144013510 tx=0x7fb14400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:19:06.700 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:06.697+0000 7fb150ff9700 1 -- 172.21.15.73:0/3696882502 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb144009320 con 0x7fb15406b960 2023-12-01T22:19:06.700 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:06.697+0000 7fb15a557700 1 -- 172.21.15.73:0/3696882502 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb154108f90 con 0x7fb15406b960 2023-12-01T22:19:06.700 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:06.697+0000 7fb150ff9700 1 -- 172.21.15.73:0/3696882502 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb144013f00 con 0x7fb15406b960 2023-12-01T22:19:06.700 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:06.697+0000 7fb150ff9700 1 -- 172.21.15.73:0/3696882502 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb144014dc0 con 0x7fb15406b960 2023-12-01T22:19:06.700 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:06.697+0000 7fb15a557700 1 -- 172.21.15.73:0/3696882502 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb154109420 con 0x7fb15406b960 2023-12-01T22:19:06.701 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:06.698+0000 7fb15a557700 1 -- 172.21.15.73:0/3696882502 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb154045ad0 con 0x7fb15406b960 2023-12-01T22:19:06.701 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:06.698+0000 7fb150ff9700 1 -- 172.21.15.73:0/3696882502 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb144014420 con 0x7fb15406b960 2023-12-01T22:19:06.701 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:06.698+0000 7fb150ff9700 1 --2- 172.21.15.73:0/3696882502 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fb14002ff60 0x7fb140032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:19:06.701 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:06.698+0000 7fb150ff9700 1 -- 172.21.15.73:0/3696882502 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb1440410f0 con 0x7fb15406b960 2023-12-01T22:19:06.701 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:06.698+0000 7fb1537fe700 1 --2- 172.21.15.73:0/3696882502 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fb14002ff60 0x7fb140032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:19:06.701 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:06.699+0000 7fb1537fe700 1 --2- 172.21.15.73:0/3696882502 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fb14002ff60 0x7fb140032420 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7fb13c0096a0 tx=0x7fb13c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:19:06.704 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:06.703+0000 7fb150ff9700 1 -- 172.21.15.73:0/3696882502 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb1440095f0 con 0x7fb15406b960 2023-12-01T22:19:06.908 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:06.906+0000 7fb15a557700 1 -- 172.21.15.73:0/3696882502 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb15410a2d0 con 0x7fb15406b960 2023-12-01T22:19:06.909 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:06.907+0000 7fb150ff9700 1 -- 172.21.15.73:0/3696882502 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb14401cb10 con 0x7fb15406b960 2023-12-01T22:19:06.909 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:19:06.909 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:19:06.911 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:06.909+0000 7fb15a557700 1 -- 172.21.15.73:0/3696882502 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fb14002ff60 msgr2=0x7fb140032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:19:06.911 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:06.909+0000 7fb15a557700 1 --2- 172.21.15.73:0/3696882502 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fb14002ff60 0x7fb140032420 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7fb13c0096a0 tx=0x7fb13c006b40).stop 2023-12-01T22:19:06.911 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:06.909+0000 7fb15a557700 1 -- 172.21.15.73:0/3696882502 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb15406b960 msgr2=0x7fb1541088a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:19:06.911 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:06.909+0000 7fb15a557700 1 --2- 172.21.15.73:0/3696882502 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb15406b960 0x7fb1541088a0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7fb144013510 tx=0x7fb14400b040).stop 2023-12-01T22:19:06.911 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:06.909+0000 7fb15a557700 1 -- 172.21.15.73:0/3696882502 shutdown_connections 2023-12-01T22:19:06.911 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:06.909+0000 7fb15a557700 1 --2- 172.21.15.73:0/3696882502 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fb14002ff60 0x7fb140032420 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:19:06.911 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:06.909+0000 7fb15a557700 1 --2- 172.21.15.73:0/3696882502 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb15406b960 0x7fb1541088a0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:19:06.911 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:06.909+0000 7fb15a557700 1 -- 172.21.15.73:0/3696882502 >> 172.21.15.73:0/3696882502 conn(0x7fb1540f1c70 msgr2=0x7fb1540f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:19:06.912 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:06.909+0000 7fb15a557700 1 -- 172.21.15.73:0/3696882502 shutdown_connections 2023-12-01T22:19:06.912 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:06.909+0000 7fb15a557700 1 -- 172.21.15.73:0/3696882502 wait complete. 2023-12-01T22:19:06.912 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:19:08.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:19:07 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:19:06.696593+0000 mgr.smithi029.loxrfu (mgr.14164) 245 : cluster [DBG] pgmap v231: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:19:08.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:19:07 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:19:06.908679+0000 mon.smithi029 (mon.0) 300 : audit [DBG] from='client.? 172.21.15.73:0/3696882502' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:19:08.292 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:19:08.292 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:19:09.484 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:09.482+0000 7f9ac5d00700 1 -- 172.21.15.73:0/3559257135 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ac00f62a0 msgr2=0x7f9ac00f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:19:09.485 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:09.482+0000 7f9ac5d00700 1 --2- 172.21.15.73:0/3559257135 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ac00f62a0 0x7f9ac00f66c0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 rx=0x7f9ab4009fb0 tx=0x7f9ab4009b30).stop 2023-12-01T22:19:09.485 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:09.482+0000 7f9ac5d00700 1 -- 172.21.15.73:0/3559257135 shutdown_connections 2023-12-01T22:19:09.485 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:09.482+0000 7f9ac5d00700 1 --2- 172.21.15.73:0/3559257135 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ac00f62a0 0x7f9ac00f66c0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:19:09.485 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:09.482+0000 7f9ac5d00700 1 -- 172.21.15.73:0/3559257135 >> 172.21.15.73:0/3559257135 conn(0x7f9ac00f1aa0 msgr2=0x7f9ac00f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:19:09.485 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:09.482+0000 7f9ac5d00700 1 -- 172.21.15.73:0/3559257135 shutdown_connections 2023-12-01T22:19:09.485 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:09.482+0000 7f9ac5d00700 1 -- 172.21.15.73:0/3559257135 wait complete. 2023-12-01T22:19:09.485 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:09.482+0000 7f9ac5d00700 1 Processor -- start 2023-12-01T22:19:09.485 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:09.482+0000 7f9ac5d00700 1 -- start start 2023-12-01T22:19:09.485 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:09.483+0000 7f9ac5d00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ac00f62a0 0x7f9ac01064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:19:09.485 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:09.483+0000 7f9ac5d00700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ac01069a0 con 0x7f9ac00f62a0 2023-12-01T22:19:09.485 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:09.483+0000 7f9ac4cfe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ac00f62a0 0x7f9ac01064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:19:09.485 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:09.483+0000 7f9ac4cfe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ac00f62a0 0x7f9ac01064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:37740/0 (socket says 172.21.15.73:37740) 2023-12-01T22:19:09.485 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:09.483+0000 7f9ac4cfe700 1 -- 172.21.15.73:0/2147118394 learned_addr learned my addr 172.21.15.73:0/2147118394 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:19:09.485 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:09.483+0000 7f9ac4cfe700 1 -- 172.21.15.73:0/2147118394 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ab4005040 con 0x7f9ac00f62a0 2023-12-01T22:19:09.487 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:09.483+0000 7f9ac4cfe700 1 --2- 172.21.15.73:0/2147118394 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ac00f62a0 0x7f9ac01064b0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7f9ab4013510 tx=0x7f9ab400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:19:09.487 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:09.483+0000 7f9abd7fa700 1 -- 172.21.15.73:0/2147118394 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9ab4009320 con 0x7f9ac00f62a0 2023-12-01T22:19:09.487 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:09.483+0000 7f9ac5d00700 1 -- 172.21.15.73:0/2147118394 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ac0106ba0 con 0x7f9ac00f62a0 2023-12-01T22:19:09.487 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:09.483+0000 7f9ac5d00700 1 -- 172.21.15.73:0/2147118394 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ac0107030 con 0x7f9ac00f62a0 2023-12-01T22:19:09.487 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:09.483+0000 7f9abd7fa700 1 -- 172.21.15.73:0/2147118394 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9ab4013f00 con 0x7f9ac00f62a0 2023-12-01T22:19:09.487 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:09.483+0000 7f9abd7fa700 1 -- 172.21.15.73:0/2147118394 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9ab4014d90 con 0x7f9ac00f62a0 2023-12-01T22:19:09.487 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:09.484+0000 7f9abd7fa700 1 -- 172.21.15.73:0/2147118394 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9ab40095f0 con 0x7f9ac00f62a0 2023-12-01T22:19:09.487 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:09.484+0000 7f9ac5d00700 1 -- 172.21.15.73:0/2147118394 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ac0045ad0 con 0x7f9ac00f62a0 2023-12-01T22:19:09.488 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:09.484+0000 7f9abd7fa700 1 --2- 172.21.15.73:0/2147118394 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f9ab002ff60 0x7f9ab0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:19:09.488 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:09.485+0000 7f9abd7fa700 1 -- 172.21.15.73:0/2147118394 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9ab4041020 con 0x7f9ac00f62a0 2023-12-01T22:19:09.488 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:09.485+0000 7f9abffff700 1 --2- 172.21.15.73:0/2147118394 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f9ab002ff60 0x7f9ab0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:19:09.488 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:09.485+0000 7f9abffff700 1 --2- 172.21.15.73:0/2147118394 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f9ab002ff60 0x7f9ab0032420 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f9aac0096a0 tx=0x7f9aac006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:19:09.491 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:09.489+0000 7f9abd7fa700 1 -- 172.21.15.73:0/2147118394 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9ab4014420 con 0x7f9ac00f62a0 2023-12-01T22:19:09.698 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:09.697+0000 7f9ac5d00700 1 -- 172.21.15.73:0/2147118394 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9ac0107ee0 con 0x7f9ac00f62a0 2023-12-01T22:19:09.699 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:09.697+0000 7f9abd7fa700 1 -- 172.21.15.73:0/2147118394 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9ab4012980 con 0x7f9ac00f62a0 2023-12-01T22:19:09.699 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:19:09.699 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:19:09.701 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:09.699+0000 7f9ac5d00700 1 -- 172.21.15.73:0/2147118394 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f9ab002ff60 msgr2=0x7f9ab0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:19:09.701 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:09.699+0000 7f9ac5d00700 1 --2- 172.21.15.73:0/2147118394 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f9ab002ff60 0x7f9ab0032420 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f9aac0096a0 tx=0x7f9aac006b40).stop 2023-12-01T22:19:09.701 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:09.699+0000 7f9ac5d00700 1 -- 172.21.15.73:0/2147118394 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ac00f62a0 msgr2=0x7f9ac01064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:19:09.701 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:09.699+0000 7f9ac5d00700 1 --2- 172.21.15.73:0/2147118394 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ac00f62a0 0x7f9ac01064b0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7f9ab4013510 tx=0x7f9ab400b040).stop 2023-12-01T22:19:09.701 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:09.699+0000 7f9ac5d00700 1 -- 172.21.15.73:0/2147118394 shutdown_connections 2023-12-01T22:19:09.701 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:09.699+0000 7f9ac5d00700 1 --2- 172.21.15.73:0/2147118394 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f9ab002ff60 0x7f9ab0032420 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:19:09.701 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:09.699+0000 7f9ac5d00700 1 --2- 172.21.15.73:0/2147118394 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ac00f62a0 0x7f9ac01064b0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:19:09.701 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:09.699+0000 7f9ac5d00700 1 -- 172.21.15.73:0/2147118394 >> 172.21.15.73:0/2147118394 conn(0x7f9ac00f1aa0 msgr2=0x7f9ac00f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:19:09.701 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:09.699+0000 7f9ac5d00700 1 -- 172.21.15.73:0/2147118394 shutdown_connections 2023-12-01T22:19:09.701 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:09.699+0000 7f9ac5d00700 1 -- 172.21.15.73:0/2147118394 wait complete. 2023-12-01T22:19:09.702 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:19:10.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:19:09 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:19:08.696942+0000 mgr.smithi029.loxrfu (mgr.14164) 246 : cluster [DBG] pgmap v232: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:19:10.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:19:09 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:19:09.698822+0000 mon.smithi029 (mon.0) 301 : audit [DBG] from='client.? 172.21.15.73:0/2147118394' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:19:11.051 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:19:11.051 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:19:11.776 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:19:11 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:19:10.697255+0000 mgr.smithi029.loxrfu (mgr.14164) 247 : cluster [DBG] pgmap v233: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:19:12.222 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:12.219+0000 7f85dffff700 1 -- 172.21.15.73:0/84177819 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85e00f62a0 msgr2=0x7f85e00f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:19:12.223 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:12.219+0000 7f85dffff700 1 --2- 172.21.15.73:0/84177819 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85e00f62a0 0x7f85e00f66c0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 rx=0x7f85d4009fb0 tx=0x7f85d4009b30).stop 2023-12-01T22:19:12.223 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:12.219+0000 7f85dffff700 1 -- 172.21.15.73:0/84177819 shutdown_connections 2023-12-01T22:19:12.223 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:12.219+0000 7f85dffff700 1 --2- 172.21.15.73:0/84177819 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85e00f62a0 0x7f85e00f66c0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:19:12.223 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:12.219+0000 7f85dffff700 1 -- 172.21.15.73:0/84177819 >> 172.21.15.73:0/84177819 conn(0x7f85e00f1aa0 msgr2=0x7f85e00f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:19:12.223 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:12.219+0000 7f85dffff700 1 -- 172.21.15.73:0/84177819 shutdown_connections 2023-12-01T22:19:12.223 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:12.219+0000 7f85dffff700 1 -- 172.21.15.73:0/84177819 wait complete. 2023-12-01T22:19:12.223 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:12.220+0000 7f85dffff700 1 Processor -- start 2023-12-01T22:19:12.223 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:12.220+0000 7f85dffff700 1 -- start start 2023-12-01T22:19:12.223 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:12.220+0000 7f85dffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85e00f62a0 0x7f85e01064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:19:12.223 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:12.220+0000 7f85dffff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85e01069a0 con 0x7f85e00f62a0 2023-12-01T22:19:12.223 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:12.220+0000 7f85deffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85e00f62a0 0x7f85e01064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:19:12.223 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:12.220+0000 7f85deffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85e00f62a0 0x7f85e01064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:37752/0 (socket says 172.21.15.73:37752) 2023-12-01T22:19:12.224 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:12.220+0000 7f85deffd700 1 -- 172.21.15.73:0/577981444 learned_addr learned my addr 172.21.15.73:0/577981444 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:19:12.224 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:12.221+0000 7f85deffd700 1 -- 172.21.15.73:0/577981444 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85d4005040 con 0x7f85e00f62a0 2023-12-01T22:19:12.224 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:12.221+0000 7f85deffd700 1 --2- 172.21.15.73:0/577981444 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85e00f62a0 0x7f85e01064b0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7f85d4013510 tx=0x7f85d400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:19:12.224 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:12.221+0000 7f85c77fe700 1 -- 172.21.15.73:0/577981444 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f85d4009320 con 0x7f85e00f62a0 2023-12-01T22:19:12.224 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:12.221+0000 7f85dffff700 1 -- 172.21.15.73:0/577981444 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85e0106ba0 con 0x7f85e00f62a0 2023-12-01T22:19:12.224 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:12.221+0000 7f85c77fe700 1 -- 172.21.15.73:0/577981444 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f85d4013f00 con 0x7f85e00f62a0 2023-12-01T22:19:12.224 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:12.221+0000 7f85c77fe700 1 -- 172.21.15.73:0/577981444 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f85d4014dc0 con 0x7f85e00f62a0 2023-12-01T22:19:12.224 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:12.221+0000 7f85dffff700 1 -- 172.21.15.73:0/577981444 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85e0107030 con 0x7f85e00f62a0 2023-12-01T22:19:12.225 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:12.222+0000 7f85dffff700 1 -- 172.21.15.73:0/577981444 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85e0045ad0 con 0x7f85e00f62a0 2023-12-01T22:19:12.225 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:12.222+0000 7f85c77fe700 1 -- 172.21.15.73:0/577981444 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f85d40095f0 con 0x7f85e00f62a0 2023-12-01T22:19:12.225 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:12.222+0000 7f85c77fe700 1 --2- 172.21.15.73:0/577981444 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f85c802ff60 0x7f85c8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:19:12.225 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:12.222+0000 7f85c77fe700 1 -- 172.21.15.73:0/577981444 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f85d40410e0 con 0x7f85e00f62a0 2023-12-01T22:19:12.225 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:12.222+0000 7f85de7fc700 1 --2- 172.21.15.73:0/577981444 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f85c802ff60 0x7f85c8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:19:12.225 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:12.223+0000 7f85de7fc700 1 --2- 172.21.15.73:0/577981444 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f85c802ff60 0x7f85c8032420 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f85d00096a0 tx=0x7f85d0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:19:12.228 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:12.227+0000 7f85c77fe700 1 -- 172.21.15.73:0/577981444 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f85d4014420 con 0x7f85e00f62a0 2023-12-01T22:19:12.433 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:12.432+0000 7f85dffff700 1 -- 172.21.15.73:0/577981444 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f85e0107e80 con 0x7f85e00f62a0 2023-12-01T22:19:12.434 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:12.432+0000 7f85c77fe700 1 -- 172.21.15.73:0/577981444 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f85d401c920 con 0x7f85e00f62a0 2023-12-01T22:19:12.434 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:19:12.434 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:19:12.436 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:12.434+0000 7f85dffff700 1 -- 172.21.15.73:0/577981444 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f85c802ff60 msgr2=0x7f85c8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:19:12.436 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:12.434+0000 7f85dffff700 1 --2- 172.21.15.73:0/577981444 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f85c802ff60 0x7f85c8032420 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f85d00096a0 tx=0x7f85d0006b40).stop 2023-12-01T22:19:12.436 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:12.434+0000 7f85dffff700 1 -- 172.21.15.73:0/577981444 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85e00f62a0 msgr2=0x7f85e01064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:19:12.436 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:12.434+0000 7f85dffff700 1 --2- 172.21.15.73:0/577981444 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85e00f62a0 0x7f85e01064b0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7f85d4013510 tx=0x7f85d400b040).stop 2023-12-01T22:19:12.436 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:12.434+0000 7f85dffff700 1 -- 172.21.15.73:0/577981444 shutdown_connections 2023-12-01T22:19:12.436 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:12.434+0000 7f85dffff700 1 --2- 172.21.15.73:0/577981444 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f85c802ff60 0x7f85c8032420 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:19:12.436 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:12.434+0000 7f85dffff700 1 --2- 172.21.15.73:0/577981444 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85e00f62a0 0x7f85e01064b0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:19:12.436 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:12.434+0000 7f85dffff700 1 -- 172.21.15.73:0/577981444 >> 172.21.15.73:0/577981444 conn(0x7f85e00f1aa0 msgr2=0x7f85e00f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:19:12.436 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:12.434+0000 7f85dffff700 1 -- 172.21.15.73:0/577981444 shutdown_connections 2023-12-01T22:19:12.436 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:12.434+0000 7f85dffff700 1 -- 172.21.15.73:0/577981444 wait complete. 2023-12-01T22:19:12.437 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:19:12.776 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:19:12 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:19:12.433718+0000 mon.smithi029 (mon.0) 302 : audit [DBG] from='client.? 172.21.15.73:0/577981444' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:19:13.777 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:19:13 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:19:12.697644+0000 mgr.smithi029.loxrfu (mgr.14164) 248 : cluster [DBG] pgmap v234: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:19:13.777 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:19:13 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:19:12.784185+0000 mon.smithi029 (mon.0) 303 : audit [INF] from='mgr.14164 172.21.15.29:0/3195927111' entity='mgr.smithi029.loxrfu' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi029.loxrfu/mirror_snapshot_schedule"}]: dispatch 2023-12-01T22:19:13.777 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:19:13 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:19:12.787279+0000 mon.smithi029 (mon.0) 304 : audit [INF] from='mgr.14164 172.21.15.29:0/3195927111' entity='mgr.smithi029.loxrfu' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi029.loxrfu/trash_purge_schedule"}]: dispatch 2023-12-01T22:19:13.819 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:19:13.819 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:19:14.975 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:14.972+0000 7f20cb0cf700 1 -- 172.21.15.73:0/3144632360 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f20c40f51a0 msgr2=0x7f20c40f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:19:14.975 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:14.972+0000 7f20cb0cf700 1 --2- 172.21.15.73:0/3144632360 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f20c40f51a0 0x7f20c40f55c0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 rx=0x7f20b4009fb0 tx=0x7f20b4009b30).stop 2023-12-01T22:19:14.975 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:14.973+0000 7f20cb0cf700 1 -- 172.21.15.73:0/3144632360 shutdown_connections 2023-12-01T22:19:14.975 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:14.973+0000 7f20cb0cf700 1 --2- 172.21.15.73:0/3144632360 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f20c40f51a0 0x7f20c40f55c0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:19:14.975 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:14.973+0000 7f20cb0cf700 1 -- 172.21.15.73:0/3144632360 >> 172.21.15.73:0/3144632360 conn(0x7f20c40f09a0 msgr2=0x7f20c40f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:19:14.975 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:14.973+0000 7f20cb0cf700 1 -- 172.21.15.73:0/3144632360 shutdown_connections 2023-12-01T22:19:14.975 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:14.973+0000 7f20cb0cf700 1 -- 172.21.15.73:0/3144632360 wait complete. 2023-12-01T22:19:14.975 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:14.973+0000 7f20cb0cf700 1 Processor -- start 2023-12-01T22:19:14.975 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:14.973+0000 7f20cb0cf700 1 -- start start 2023-12-01T22:19:14.976 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:14.973+0000 7f20cb0cf700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f20c40f51a0 0x7f20c41091d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:19:14.976 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:14.973+0000 7f20cb0cf700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20c4060220 con 0x7f20c40f51a0 2023-12-01T22:19:14.976 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:14.973+0000 7f20ca0cd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f20c40f51a0 0x7f20c41091d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:19:14.976 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:14.974+0000 7f20ca0cd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f20c40f51a0 0x7f20c41091d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:33166/0 (socket says 172.21.15.73:33166) 2023-12-01T22:19:14.976 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:14.974+0000 7f20ca0cd700 1 -- 172.21.15.73:0/2734799945 learned_addr learned my addr 172.21.15.73:0/2734799945 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:19:14.976 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:14.974+0000 7f20ca0cd700 1 -- 172.21.15.73:0/2734799945 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20b4005040 con 0x7f20c40f51a0 2023-12-01T22:19:14.976 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:14.974+0000 7f20ca0cd700 1 --2- 172.21.15.73:0/2734799945 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f20c40f51a0 0x7f20c41091d0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7f20b4013510 tx=0x7f20b400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:19:14.976 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:14.974+0000 7f20baffd700 1 -- 172.21.15.73:0/2734799945 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f20b4009320 con 0x7f20c40f51a0 2023-12-01T22:19:14.976 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:14.974+0000 7f20cb0cf700 1 -- 172.21.15.73:0/2734799945 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20c4060420 con 0x7f20c40f51a0 2023-12-01T22:19:14.976 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:14.974+0000 7f20baffd700 1 -- 172.21.15.73:0/2734799945 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f20b4013f00 con 0x7f20c40f51a0 2023-12-01T22:19:14.976 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:14.974+0000 7f20baffd700 1 -- 172.21.15.73:0/2734799945 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f20b4014dc0 con 0x7f20c40f51a0 2023-12-01T22:19:14.976 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:14.974+0000 7f20cb0cf700 1 -- 172.21.15.73:0/2734799945 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20c4109880 con 0x7f20c40f51a0 2023-12-01T22:19:14.977 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:14.975+0000 7f20baffd700 1 -- 172.21.15.73:0/2734799945 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f20b40095f0 con 0x7f20c40f51a0 2023-12-01T22:19:14.977 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:14.975+0000 7f20cb0cf700 1 -- 172.21.15.73:0/2734799945 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20c4059100 con 0x7f20c40f51a0 2023-12-01T22:19:14.977 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:14.975+0000 7f20baffd700 1 --2- 172.21.15.73:0/2734799945 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f20b002ff60 0x7f20b0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:19:14.977 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:14.975+0000 7f20baffd700 1 -- 172.21.15.73:0/2734799945 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f20b4041040 con 0x7f20c40f51a0 2023-12-01T22:19:14.977 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:14.975+0000 7f20c98cc700 1 --2- 172.21.15.73:0/2734799945 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f20b002ff60 0x7f20b0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:19:14.978 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:14.976+0000 7f20c98cc700 1 --2- 172.21.15.73:0/2734799945 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f20b002ff60 0x7f20b0032420 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f20c00096a0 tx=0x7f20c0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:19:14.981 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:14.980+0000 7f20baffd700 1 -- 172.21.15.73:0/2734799945 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f20b4014420 con 0x7f20c40f51a0 2023-12-01T22:19:15.189 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:15.189+0000 7f20cb0cf700 1 -- 172.21.15.73:0/2734799945 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f20c4045ad0 con 0x7f20c40f51a0 2023-12-01T22:19:15.190 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:15.189+0000 7f20baffd700 1 -- 172.21.15.73:0/2734799945 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f20b4010230 con 0x7f20c40f51a0 2023-12-01T22:19:15.190 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:19:15.190 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:19:15.192 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:15.191+0000 7f20cb0cf700 1 -- 172.21.15.73:0/2734799945 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f20b002ff60 msgr2=0x7f20b0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:19:15.192 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:15.191+0000 7f20cb0cf700 1 --2- 172.21.15.73:0/2734799945 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f20b002ff60 0x7f20b0032420 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f20c00096a0 tx=0x7f20c0006b40).stop 2023-12-01T22:19:15.192 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:15.191+0000 7f20cb0cf700 1 -- 172.21.15.73:0/2734799945 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f20c40f51a0 msgr2=0x7f20c41091d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:19:15.192 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:15.191+0000 7f20cb0cf700 1 --2- 172.21.15.73:0/2734799945 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f20c40f51a0 0x7f20c41091d0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7f20b4013510 tx=0x7f20b400b040).stop 2023-12-01T22:19:15.192 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:15.191+0000 7f20cb0cf700 1 -- 172.21.15.73:0/2734799945 shutdown_connections 2023-12-01T22:19:15.192 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:15.191+0000 7f20cb0cf700 1 --2- 172.21.15.73:0/2734799945 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f20b002ff60 0x7f20b0032420 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:19:15.192 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:15.191+0000 7f20cb0cf700 1 --2- 172.21.15.73:0/2734799945 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f20c40f51a0 0x7f20c41091d0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:19:15.192 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:15.191+0000 7f20cb0cf700 1 -- 172.21.15.73:0/2734799945 >> 172.21.15.73:0/2734799945 conn(0x7f20c40f09a0 msgr2=0x7f20c40f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:19:15.192 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:15.191+0000 7f20cb0cf700 1 -- 172.21.15.73:0/2734799945 shutdown_connections 2023-12-01T22:19:15.192 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:15.191+0000 7f20cb0cf700 1 -- 172.21.15.73:0/2734799945 wait complete. 2023-12-01T22:19:15.193 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:19:16.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:19:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:19:14.697972+0000 mgr.smithi029.loxrfu (mgr.14164) 249 : cluster [DBG] pgmap v235: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:19:16.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:19:15 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:19:15.189677+0000 mon.smithi029 (mon.0) 305 : audit [DBG] from='client.? 172.21.15.73:0/2734799945' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:19:16.591 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:19:16.592 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:19:17.771 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:17.769+0000 7f5bed78b700 1 -- 172.21.15.73:0/3087255968 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5be80f47c0 msgr2=0x7f5be80f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:19:17.771 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:17.769+0000 7f5bed78b700 1 --2- 172.21.15.73:0/3087255968 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5be80f47c0 0x7f5be80f4be0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 rx=0x7f5bd0009fb0 tx=0x7f5bd0009b30).stop 2023-12-01T22:19:17.771 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:17.769+0000 7f5bed78b700 1 -- 172.21.15.73:0/3087255968 shutdown_connections 2023-12-01T22:19:17.771 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:17.769+0000 7f5bed78b700 1 --2- 172.21.15.73:0/3087255968 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5be80f47c0 0x7f5be80f4be0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:19:17.771 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:17.769+0000 7f5bed78b700 1 -- 172.21.15.73:0/3087255968 >> 172.21.15.73:0/3087255968 conn(0x7f5be80f02e0 msgr2=0x7f5be80f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:19:17.771 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:17.769+0000 7f5bed78b700 1 -- 172.21.15.73:0/3087255968 shutdown_connections 2023-12-01T22:19:17.772 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:17.769+0000 7f5bed78b700 1 -- 172.21.15.73:0/3087255968 wait complete. 2023-12-01T22:19:17.772 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:17.770+0000 7f5bed78b700 1 Processor -- start 2023-12-01T22:19:17.772 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:17.770+0000 7f5bed78b700 1 -- start start 2023-12-01T22:19:17.772 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:17.770+0000 7f5bed78b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5be80f47c0 0x7f5be80f7370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:19:17.772 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:17.770+0000 7f5bed78b700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5be80f57f0 con 0x7f5be80f47c0 2023-12-01T22:19:17.772 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:17.770+0000 7f5be6ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5be80f47c0 0x7f5be80f7370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:19:17.772 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:17.770+0000 7f5be6ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5be80f47c0 0x7f5be80f7370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:33190/0 (socket says 172.21.15.73:33190) 2023-12-01T22:19:17.772 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:17.770+0000 7f5be6ffd700 1 -- 172.21.15.73:0/2956565090 learned_addr learned my addr 172.21.15.73:0/2956565090 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:19:17.772 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:17.771+0000 7f5be6ffd700 1 -- 172.21.15.73:0/2956565090 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bd0005040 con 0x7f5be80f47c0 2023-12-01T22:19:17.772 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:17.771+0000 7f5be6ffd700 1 --2- 172.21.15.73:0/2956565090 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5be80f47c0 0x7f5be80f7370 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7f5bd0013510 tx=0x7f5bd000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:19:17.773 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:17.771+0000 7f5bdf7fe700 1 -- 172.21.15.73:0/2956565090 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5bd0009320 con 0x7f5be80f47c0 2023-12-01T22:19:17.773 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:17.771+0000 7f5bed78b700 1 -- 172.21.15.73:0/2956565090 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5be80f5990 con 0x7f5be80f47c0 2023-12-01T22:19:17.774 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:17.771+0000 7f5bdf7fe700 1 -- 172.21.15.73:0/2956565090 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5bd0013f00 con 0x7f5be80f47c0 2023-12-01T22:19:17.774 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:17.771+0000 7f5bdf7fe700 1 -- 172.21.15.73:0/2956565090 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5bd0014d90 con 0x7f5be80f47c0 2023-12-01T22:19:17.774 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:17.771+0000 7f5bed78b700 1 -- 172.21.15.73:0/2956565090 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5be80f5e20 con 0x7f5be80f47c0 2023-12-01T22:19:17.774 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:17.772+0000 7f5bdf7fe700 1 -- 172.21.15.73:0/2956565090 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5bd0014420 con 0x7f5be80f47c0 2023-12-01T22:19:17.774 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:17.772+0000 7f5bed78b700 1 -- 172.21.15.73:0/2956565090 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5be8059100 con 0x7f5be80f47c0 2023-12-01T22:19:17.774 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:17.772+0000 7f5bdf7fe700 1 --2- 172.21.15.73:0/2956565090 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f5bd402ff60 0x7f5bd4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:19:17.774 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:17.772+0000 7f5bdf7fe700 1 -- 172.21.15.73:0/2956565090 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5bd00410c0 con 0x7f5be80f47c0 2023-12-01T22:19:17.774 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:17.772+0000 7f5be67fc700 1 --2- 172.21.15.73:0/2956565090 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f5bd402ff60 0x7f5bd4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:19:17.774 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:17.772+0000 7f5be67fc700 1 --2- 172.21.15.73:0/2956565090 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f5bd402ff60 0x7f5bd4032420 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f5bd80096a0 tx=0x7f5bd8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:19:17.777 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:17.777+0000 7f5bdf7fe700 1 -- 172.21.15.73:0/2956565090 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5bd00095f0 con 0x7f5be80f47c0 2023-12-01T22:19:17.984 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:17.983+0000 7f5bed78b700 1 -- 172.21.15.73:0/2956565090 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5be80fbf60 con 0x7f5be80f47c0 2023-12-01T22:19:17.984 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:17.984+0000 7f5bdf7fe700 1 -- 172.21.15.73:0/2956565090 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5bd001db70 con 0x7f5be80f47c0 2023-12-01T22:19:17.985 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:19:17.985 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:19:17.986 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:17.985+0000 7f5bed78b700 1 -- 172.21.15.73:0/2956565090 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f5bd402ff60 msgr2=0x7f5bd4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:19:17.986 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:17.985+0000 7f5bed78b700 1 --2- 172.21.15.73:0/2956565090 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f5bd402ff60 0x7f5bd4032420 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f5bd80096a0 tx=0x7f5bd8006b40).stop 2023-12-01T22:19:17.986 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:17.985+0000 7f5bed78b700 1 -- 172.21.15.73:0/2956565090 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5be80f47c0 msgr2=0x7f5be80f7370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:19:17.986 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:17.985+0000 7f5bed78b700 1 --2- 172.21.15.73:0/2956565090 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5be80f47c0 0x7f5be80f7370 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7f5bd0013510 tx=0x7f5bd000b040).stop 2023-12-01T22:19:17.986 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:17.985+0000 7f5bed78b700 1 -- 172.21.15.73:0/2956565090 shutdown_connections 2023-12-01T22:19:17.987 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:17.985+0000 7f5bed78b700 1 --2- 172.21.15.73:0/2956565090 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f5bd402ff60 0x7f5bd4032420 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:19:17.987 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:17.985+0000 7f5bed78b700 1 --2- 172.21.15.73:0/2956565090 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5be80f47c0 0x7f5be80f7370 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:19:17.987 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:17.985+0000 7f5bed78b700 1 -- 172.21.15.73:0/2956565090 >> 172.21.15.73:0/2956565090 conn(0x7f5be80f02e0 msgr2=0x7f5be80f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:19:17.987 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:17.985+0000 7f5bed78b700 1 -- 172.21.15.73:0/2956565090 shutdown_connections 2023-12-01T22:19:17.987 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:17.985+0000 7f5bed78b700 1 -- 172.21.15.73:0/2956565090 wait complete. 2023-12-01T22:19:17.987 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:19:18.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:19:17 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:19:16.698365+0000 mgr.smithi029.loxrfu (mgr.14164) 250 : cluster [DBG] pgmap v236: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:19:19.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:19:18 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:19:17.984281+0000 mon.smithi029 (mon.0) 306 : audit [DBG] from='client.? 172.21.15.73:0/2956565090' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:19:19.401 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:19:19.401 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:19:20.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:19:19 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:19:18.698665+0000 mgr.smithi029.loxrfu (mgr.14164) 251 : cluster [DBG] pgmap v237: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:19:20.510 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:20.509+0000 7fface8d4700 1 -- 172.21.15.73:0/2199134980 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffac80f51a0 msgr2=0x7ffac80f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:19:20.510 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:20.509+0000 7fface8d4700 1 --2- 172.21.15.73:0/2199134980 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffac80f51a0 0x7ffac80f55c0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 rx=0x7ffab8009fb0 tx=0x7ffab8009b30).stop 2023-12-01T22:19:20.510 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:20.509+0000 7fface8d4700 1 -- 172.21.15.73:0/2199134980 shutdown_connections 2023-12-01T22:19:20.510 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:20.509+0000 7fface8d4700 1 --2- 172.21.15.73:0/2199134980 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffac80f51a0 0x7ffac80f55c0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:19:20.510 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:20.509+0000 7fface8d4700 1 -- 172.21.15.73:0/2199134980 >> 172.21.15.73:0/2199134980 conn(0x7ffac80f09a0 msgr2=0x7ffac80f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:19:20.510 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:20.509+0000 7fface8d4700 1 -- 172.21.15.73:0/2199134980 shutdown_connections 2023-12-01T22:19:20.510 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:20.509+0000 7fface8d4700 1 -- 172.21.15.73:0/2199134980 wait complete. 2023-12-01T22:19:20.511 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:20.509+0000 7fface8d4700 1 Processor -- start 2023-12-01T22:19:20.511 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:20.509+0000 7fface8d4700 1 -- start start 2023-12-01T22:19:20.512 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:20.510+0000 7fface8d4700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffac80f51a0 0x7ffac81053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:19:20.512 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:20.510+0000 7fface8d4700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffac81058a0 con 0x7ffac80f51a0 2023-12-01T22:19:20.512 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:20.510+0000 7ffacd8d2700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffac80f51a0 0x7ffac81053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:19:20.513 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:20.510+0000 7ffacd8d2700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffac80f51a0 0x7ffac81053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:33204/0 (socket says 172.21.15.73:33204) 2023-12-01T22:19:20.513 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:20.510+0000 7ffacd8d2700 1 -- 172.21.15.73:0/4294016936 learned_addr learned my addr 172.21.15.73:0/4294016936 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:19:20.513 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:20.510+0000 7ffacd8d2700 1 -- 172.21.15.73:0/4294016936 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffab8005040 con 0x7ffac80f51a0 2023-12-01T22:19:20.513 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:20.510+0000 7ffacd8d2700 1 --2- 172.21.15.73:0/4294016936 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffac80f51a0 0x7ffac81053b0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7ffab8013510 tx=0x7ffab800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:19:20.513 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:20.511+0000 7ffabe7fc700 1 -- 172.21.15.73:0/4294016936 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffab8009320 con 0x7ffac80f51a0 2023-12-01T22:19:20.513 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:20.511+0000 7fface8d4700 1 -- 172.21.15.73:0/4294016936 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffac8105aa0 con 0x7ffac80f51a0 2023-12-01T22:19:20.513 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:20.511+0000 7ffabe7fc700 1 -- 172.21.15.73:0/4294016936 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ffab8013f00 con 0x7ffac80f51a0 2023-12-01T22:19:20.513 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:20.511+0000 7ffabe7fc700 1 -- 172.21.15.73:0/4294016936 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffab8014dc0 con 0x7ffac80f51a0 2023-12-01T22:19:20.513 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:20.511+0000 7fface8d4700 1 -- 172.21.15.73:0/4294016936 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffac8105f30 con 0x7ffac80f51a0 2023-12-01T22:19:20.513 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:20.511+0000 7ffabe7fc700 1 -- 172.21.15.73:0/4294016936 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ffab80095f0 con 0x7ffac80f51a0 2023-12-01T22:19:20.513 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:20.512+0000 7ffabe7fc700 1 --2- 172.21.15.73:0/4294016936 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7ffab402ff60 0x7ffab4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:19:20.513 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:20.512+0000 7ffabe7fc700 1 -- 172.21.15.73:0/4294016936 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ffab8041030 con 0x7ffac80f51a0 2023-12-01T22:19:20.513 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:20.512+0000 7ffacd0d1700 1 --2- 172.21.15.73:0/4294016936 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7ffab402ff60 0x7ffab4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:19:20.513 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:20.512+0000 7fface8d4700 1 -- 172.21.15.73:0/4294016936 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffac8059100 con 0x7ffac80f51a0 2023-12-01T22:19:20.514 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:20.512+0000 7ffacd0d1700 1 --2- 172.21.15.73:0/4294016936 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7ffab402ff60 0x7ffab4032420 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7ffac40096a0 tx=0x7ffac4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:19:20.517 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:20.517+0000 7ffabe7fc700 1 -- 172.21.15.73:0/4294016936 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ffab8014420 con 0x7ffac80f51a0 2023-12-01T22:19:20.726 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:20.726+0000 7fface8d4700 1 -- 172.21.15.73:0/4294016936 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ffac80fabe0 con 0x7ffac80f51a0 2023-12-01T22:19:20.727 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:20.726+0000 7ffabe7fc700 1 -- 172.21.15.73:0/4294016936 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ffab8010230 con 0x7ffac80f51a0 2023-12-01T22:19:20.727 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:19:20.727 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:19:20.729 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:20.728+0000 7fface8d4700 1 -- 172.21.15.73:0/4294016936 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7ffab402ff60 msgr2=0x7ffab4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:19:20.729 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:20.728+0000 7fface8d4700 1 --2- 172.21.15.73:0/4294016936 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7ffab402ff60 0x7ffab4032420 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7ffac40096a0 tx=0x7ffac4006b40).stop 2023-12-01T22:19:20.729 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:20.728+0000 7fface8d4700 1 -- 172.21.15.73:0/4294016936 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffac80f51a0 msgr2=0x7ffac81053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:19:20.729 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:20.728+0000 7fface8d4700 1 --2- 172.21.15.73:0/4294016936 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffac80f51a0 0x7ffac81053b0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7ffab8013510 tx=0x7ffab800b040).stop 2023-12-01T22:19:20.729 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:20.728+0000 7fface8d4700 1 -- 172.21.15.73:0/4294016936 shutdown_connections 2023-12-01T22:19:20.729 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:20.728+0000 7fface8d4700 1 --2- 172.21.15.73:0/4294016936 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7ffab402ff60 0x7ffab4032420 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:19:20.729 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:20.728+0000 7fface8d4700 1 --2- 172.21.15.73:0/4294016936 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffac80f51a0 0x7ffac81053b0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:19:20.729 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:20.728+0000 7fface8d4700 1 -- 172.21.15.73:0/4294016936 >> 172.21.15.73:0/4294016936 conn(0x7ffac80f09a0 msgr2=0x7ffac80f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:19:20.729 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:20.728+0000 7fface8d4700 1 -- 172.21.15.73:0/4294016936 shutdown_connections 2023-12-01T22:19:20.730 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:20.728+0000 7fface8d4700 1 -- 172.21.15.73:0/4294016936 wait complete. 2023-12-01T22:19:20.730 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:19:21.777 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:19:21 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:19:20.698995+0000 mgr.smithi029.loxrfu (mgr.14164) 252 : cluster [DBG] pgmap v238: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:19:21.777 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:19:21 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:19:20.726857+0000 mon.smithi029 (mon.0) 307 : audit [DBG] from='client.? 172.21.15.73:0/4294016936' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:19:22.154 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:19:22.155 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:19:23.318 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:23.316+0000 7f53c4da1700 1 -- 172.21.15.73:0/1781953102 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53c00f57d0 msgr2=0x7f53c00f5bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:19:23.318 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:23.316+0000 7f53c4da1700 1 --2- 172.21.15.73:0/1781953102 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53c00f57d0 0x7f53c00f5bf0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 rx=0x7f53a8009fb0 tx=0x7f53a8009b30).stop 2023-12-01T22:19:23.319 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:23.316+0000 7f53c4da1700 1 -- 172.21.15.73:0/1781953102 shutdown_connections 2023-12-01T22:19:23.319 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:23.316+0000 7f53c4da1700 1 --2- 172.21.15.73:0/1781953102 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53c00f57d0 0x7f53c00f5bf0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:19:23.319 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:23.316+0000 7f53c4da1700 1 -- 172.21.15.73:0/1781953102 >> 172.21.15.73:0/1781953102 conn(0x7f53c00f0120 msgr2=0x7f53c00f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:19:23.319 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:23.316+0000 7f53c4da1700 1 -- 172.21.15.73:0/1781953102 shutdown_connections 2023-12-01T22:19:23.319 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:23.316+0000 7f53c4da1700 1 -- 172.21.15.73:0/1781953102 wait complete. 2023-12-01T22:19:23.319 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:23.316+0000 7f53c4da1700 1 Processor -- start 2023-12-01T22:19:23.319 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:23.317+0000 7f53c4da1700 1 -- start start 2023-12-01T22:19:23.319 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:23.317+0000 7f53c4da1700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53c00f57d0 0x7f53c01064f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:19:23.319 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:23.317+0000 7f53c4da1700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53c01069e0 con 0x7f53c00f57d0 2023-12-01T22:19:23.319 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:23.317+0000 7f53bf7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53c00f57d0 0x7f53c01064f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:19:23.319 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:23.317+0000 7f53bf7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53c00f57d0 0x7f53c01064f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:33224/0 (socket says 172.21.15.73:33224) 2023-12-01T22:19:23.319 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:23.317+0000 7f53bf7fe700 1 -- 172.21.15.73:0/2613693098 learned_addr learned my addr 172.21.15.73:0/2613693098 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:19:23.319 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:23.317+0000 7f53bf7fe700 1 -- 172.21.15.73:0/2613693098 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53a8005040 con 0x7f53c00f57d0 2023-12-01T22:19:23.319 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:23.317+0000 7f53bf7fe700 1 --2- 172.21.15.73:0/2613693098 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53c00f57d0 0x7f53c01064f0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7f53a8013510 tx=0x7f53a800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:19:23.319 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:23.318+0000 7f53b7fff700 1 -- 172.21.15.73:0/2613693098 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f53a8009320 con 0x7f53c00f57d0 2023-12-01T22:19:23.320 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:23.318+0000 7f53c4da1700 1 -- 172.21.15.73:0/2613693098 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53c0106be0 con 0x7f53c00f57d0 2023-12-01T22:19:23.320 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:23.318+0000 7f53b7fff700 1 -- 172.21.15.73:0/2613693098 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f53a8013f00 con 0x7f53c00f57d0 2023-12-01T22:19:23.320 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:23.318+0000 7f53c4da1700 1 -- 172.21.15.73:0/2613693098 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53c0107070 con 0x7f53c00f57d0 2023-12-01T22:19:23.320 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:23.318+0000 7f53b7fff700 1 -- 172.21.15.73:0/2613693098 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f53a8014dc0 con 0x7f53c00f57d0 2023-12-01T22:19:23.320 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:23.318+0000 7f53c4da1700 1 -- 172.21.15.73:0/2613693098 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53c0045ad0 con 0x7f53c00f57d0 2023-12-01T22:19:23.321 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:23.319+0000 7f53b7fff700 1 -- 172.21.15.73:0/2613693098 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f53a80095f0 con 0x7f53c00f57d0 2023-12-01T22:19:23.321 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:23.319+0000 7f53b7fff700 1 --2- 172.21.15.73:0/2613693098 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f53ac02ff60 0x7f53ac032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:19:23.321 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:23.319+0000 7f53b7fff700 1 -- 172.21.15.73:0/2613693098 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f53a80410b0 con 0x7f53c00f57d0 2023-12-01T22:19:23.321 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:23.319+0000 7f53beffd700 1 --2- 172.21.15.73:0/2613693098 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f53ac02ff60 0x7f53ac032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:19:23.321 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:23.319+0000 7f53beffd700 1 --2- 172.21.15.73:0/2613693098 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f53ac02ff60 0x7f53ac032420 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f53b00096a0 tx=0x7f53b0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:19:23.324 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:23.323+0000 7f53b7fff700 1 -- 172.21.15.73:0/2613693098 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f53a8014420 con 0x7f53c00f57d0 2023-12-01T22:19:23.532 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:23.531+0000 7f53c4da1700 1 -- 172.21.15.73:0/2613693098 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f53c00fabe0 con 0x7f53c00f57d0 2023-12-01T22:19:23.533 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:23.532+0000 7f53b7fff700 1 -- 172.21.15.73:0/2613693098 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f53a8010230 con 0x7f53c00f57d0 2023-12-01T22:19:23.533 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:19:23.533 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:19:23.534 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:23.534+0000 7f53c4da1700 1 -- 172.21.15.73:0/2613693098 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f53ac02ff60 msgr2=0x7f53ac032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:19:23.538 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:23.534+0000 7f53c4da1700 1 --2- 172.21.15.73:0/2613693098 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f53ac02ff60 0x7f53ac032420 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f53b00096a0 tx=0x7f53b0006b40).stop 2023-12-01T22:19:23.538 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:23.534+0000 7f53c4da1700 1 -- 172.21.15.73:0/2613693098 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53c00f57d0 msgr2=0x7f53c01064f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:19:23.539 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:23.534+0000 7f53c4da1700 1 --2- 172.21.15.73:0/2613693098 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53c00f57d0 0x7f53c01064f0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7f53a8013510 tx=0x7f53a800b040).stop 2023-12-01T22:19:23.539 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:23.534+0000 7f53c4da1700 1 -- 172.21.15.73:0/2613693098 shutdown_connections 2023-12-01T22:19:23.539 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:23.534+0000 7f53c4da1700 1 --2- 172.21.15.73:0/2613693098 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f53ac02ff60 0x7f53ac032420 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:19:23.539 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:23.534+0000 7f53c4da1700 1 --2- 172.21.15.73:0/2613693098 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53c00f57d0 0x7f53c01064f0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:19:23.539 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:23.534+0000 7f53c4da1700 1 -- 172.21.15.73:0/2613693098 >> 172.21.15.73:0/2613693098 conn(0x7f53c00f0120 msgr2=0x7f53c00f4dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:19:23.539 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:23.534+0000 7f53c4da1700 1 -- 172.21.15.73:0/2613693098 shutdown_connections 2023-12-01T22:19:23.539 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:23.534+0000 7f53c4da1700 1 -- 172.21.15.73:0/2613693098 wait complete. 2023-12-01T22:19:23.540 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:19:24.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:19:23 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:19:22.699427+0000 mgr.smithi029.loxrfu (mgr.14164) 253 : cluster [DBG] pgmap v239: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:19:24.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:19:23 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:19:23.532526+0000 mon.smithi029 (mon.0) 308 : audit [DBG] from='client.? 172.21.15.73:0/2613693098' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:19:24.916 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:19:24.916 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:19:26.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:19:25 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:19:24.699943+0000 mgr.smithi029.loxrfu (mgr.14164) 254 : cluster [DBG] pgmap v240: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:19:26.093 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:26.091+0000 7f0fb5ab0700 1 -- 172.21.15.73:0/1236720811 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0fb00f5410 msgr2=0x7f0fb00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:19:26.093 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:26.091+0000 7f0fb5ab0700 1 --2- 172.21.15.73:0/1236720811 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0fb00f5410 0x7f0fb00f5830 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 rx=0x7f0fa4009fb0 tx=0x7f0fa4009b30).stop 2023-12-01T22:19:26.093 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:26.091+0000 7f0fb5ab0700 1 -- 172.21.15.73:0/1236720811 shutdown_connections 2023-12-01T22:19:26.093 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:26.091+0000 7f0fb5ab0700 1 --2- 172.21.15.73:0/1236720811 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0fb00f5410 0x7f0fb00f5830 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:19:26.093 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:26.091+0000 7f0fb5ab0700 1 -- 172.21.15.73:0/1236720811 >> 172.21.15.73:0/1236720811 conn(0x7f0fb00f0b70 msgr2=0x7f0fb00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:19:26.093 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:26.091+0000 7f0fb5ab0700 1 -- 172.21.15.73:0/1236720811 shutdown_connections 2023-12-01T22:19:26.093 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:26.091+0000 7f0fb5ab0700 1 -- 172.21.15.73:0/1236720811 wait complete. 2023-12-01T22:19:26.095 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:26.091+0000 7f0fb5ab0700 1 Processor -- start 2023-12-01T22:19:26.095 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:26.092+0000 7f0fb5ab0700 1 -- start start 2023-12-01T22:19:26.095 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:26.092+0000 7f0fb5ab0700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0fb00f5410 0x7f0fb0109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:19:26.095 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:26.092+0000 7f0fb5ab0700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fb0060220 con 0x7f0fb00f5410 2023-12-01T22:19:26.095 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:26.092+0000 7f0faf7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0fb00f5410 0x7f0fb0109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:19:26.095 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:26.092+0000 7f0faf7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0fb00f5410 0x7f0fb0109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:54664/0 (socket says 172.21.15.73:54664) 2023-12-01T22:19:26.095 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:26.092+0000 7f0faf7fe700 1 -- 172.21.15.73:0/3423393519 learned_addr learned my addr 172.21.15.73:0/3423393519 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:19:26.095 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:26.092+0000 7f0faf7fe700 1 -- 172.21.15.73:0/3423393519 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fa4005040 con 0x7f0fb00f5410 2023-12-01T22:19:26.095 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:26.092+0000 7f0faf7fe700 1 --2- 172.21.15.73:0/3423393519 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0fb00f5410 0x7f0fb0109450 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7f0fa4013510 tx=0x7f0fa400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:19:26.095 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:26.093+0000 7f0f97fff700 1 -- 172.21.15.73:0/3423393519 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0fa4009320 con 0x7f0fb00f5410 2023-12-01T22:19:26.095 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:26.093+0000 7f0fb5ab0700 1 -- 172.21.15.73:0/3423393519 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fb0060420 con 0x7f0fb00f5410 2023-12-01T22:19:26.095 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:26.093+0000 7f0f97fff700 1 -- 172.21.15.73:0/3423393519 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0fa4013f00 con 0x7f0fb00f5410 2023-12-01T22:19:26.095 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:26.093+0000 7f0f97fff700 1 -- 172.21.15.73:0/3423393519 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0fa4014d90 con 0x7f0fb00f5410 2023-12-01T22:19:26.095 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:26.093+0000 7f0fb5ab0700 1 -- 172.21.15.73:0/3423393519 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fb0109b00 con 0x7f0fb00f5410 2023-12-01T22:19:26.096 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:26.093+0000 7f0f97fff700 1 -- 172.21.15.73:0/3423393519 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0fa40095f0 con 0x7f0fb00f5410 2023-12-01T22:19:26.096 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:26.093+0000 7f0fb5ab0700 1 -- 172.21.15.73:0/3423393519 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fb0059100 con 0x7f0fb00f5410 2023-12-01T22:19:26.096 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:26.094+0000 7f0f97fff700 1 --2- 172.21.15.73:0/3423393519 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f0f9802ff60 0x7f0f98032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:19:26.097 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:26.094+0000 7f0f97fff700 1 -- 172.21.15.73:0/3423393519 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0fa4041000 con 0x7f0fb00f5410 2023-12-01T22:19:26.097 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:26.094+0000 7f0faeffd700 1 --2- 172.21.15.73:0/3423393519 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f0f9802ff60 0x7f0f98032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:19:26.097 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:26.094+0000 7f0faeffd700 1 --2- 172.21.15.73:0/3423393519 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f0f9802ff60 0x7f0f98032420 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f0fa00096a0 tx=0x7f0fa0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:19:26.100 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:26.098+0000 7f0f97fff700 1 -- 172.21.15.73:0/3423393519 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0fa4014420 con 0x7f0fb00f5410 2023-12-01T22:19:26.307 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:26.305+0000 7f0fb5ab0700 1 -- 172.21.15.73:0/3423393519 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0fb0045ad0 con 0x7f0fb00f5410 2023-12-01T22:19:26.307 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:26.305+0000 7f0f97fff700 1 -- 172.21.15.73:0/3423393519 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0fa4010230 con 0x7f0fb00f5410 2023-12-01T22:19:26.307 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:19:26.307 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:19:26.309 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:26.307+0000 7f0fb5ab0700 1 -- 172.21.15.73:0/3423393519 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f0f9802ff60 msgr2=0x7f0f98032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:19:26.309 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:26.307+0000 7f0fb5ab0700 1 --2- 172.21.15.73:0/3423393519 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f0f9802ff60 0x7f0f98032420 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f0fa00096a0 tx=0x7f0fa0006b40).stop 2023-12-01T22:19:26.309 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:26.307+0000 7f0fb5ab0700 1 -- 172.21.15.73:0/3423393519 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0fb00f5410 msgr2=0x7f0fb0109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:19:26.309 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:26.307+0000 7f0fb5ab0700 1 --2- 172.21.15.73:0/3423393519 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0fb00f5410 0x7f0fb0109450 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7f0fa4013510 tx=0x7f0fa400b040).stop 2023-12-01T22:19:26.309 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:26.307+0000 7f0fb5ab0700 1 -- 172.21.15.73:0/3423393519 shutdown_connections 2023-12-01T22:19:26.309 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:26.307+0000 7f0fb5ab0700 1 --2- 172.21.15.73:0/3423393519 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f0f9802ff60 0x7f0f98032420 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:19:26.309 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:26.307+0000 7f0fb5ab0700 1 --2- 172.21.15.73:0/3423393519 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0fb00f5410 0x7f0fb0109450 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:19:26.309 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:26.307+0000 7f0fb5ab0700 1 -- 172.21.15.73:0/3423393519 >> 172.21.15.73:0/3423393519 conn(0x7f0fb00f0b70 msgr2=0x7f0fb00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:19:26.309 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:26.307+0000 7f0fb5ab0700 1 -- 172.21.15.73:0/3423393519 shutdown_connections 2023-12-01T22:19:26.309 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:26.307+0000 7f0fb5ab0700 1 -- 172.21.15.73:0/3423393519 wait complete. 2023-12-01T22:19:26.310 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:19:27.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:19:26 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:19:26.307030+0000 mon.smithi029 (mon.0) 309 : audit [DBG] from='client.? 172.21.15.73:0/3423393519' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:19:27.690 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:19:27.691 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:19:28.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:19:27 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:19:26.700379+0000 mgr.smithi029.loxrfu (mgr.14164) 255 : cluster [DBG] pgmap v241: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:19:28.809 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:28.803+0000 7f94b60b6700 1 -- 172.21.15.73:0/860845475 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94b00f6510 msgr2=0x7f94b00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:19:28.809 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:28.803+0000 7f94b60b6700 1 --2- 172.21.15.73:0/860845475 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94b00f6510 0x7f94b00f6930 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 rx=0x7f94a4009fb0 tx=0x7f94a4009b30).stop 2023-12-01T22:19:28.809 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:28.803+0000 7f94b60b6700 1 -- 172.21.15.73:0/860845475 shutdown_connections 2023-12-01T22:19:28.810 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:28.803+0000 7f94b60b6700 1 --2- 172.21.15.73:0/860845475 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94b00f6510 0x7f94b00f6930 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:19:28.810 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:28.803+0000 7f94b60b6700 1 -- 172.21.15.73:0/860845475 >> 172.21.15.73:0/860845475 conn(0x7f94b00f1c70 msgr2=0x7f94b00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:19:28.810 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:28.803+0000 7f94b60b6700 1 -- 172.21.15.73:0/860845475 shutdown_connections 2023-12-01T22:19:28.810 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:28.803+0000 7f94b60b6700 1 -- 172.21.15.73:0/860845475 wait complete. 2023-12-01T22:19:28.810 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:28.803+0000 7f94b60b6700 1 Processor -- start 2023-12-01T22:19:28.810 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:28.804+0000 7f94b60b6700 1 -- start start 2023-12-01T22:19:28.810 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:28.804+0000 7f94b60b6700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94b00f6510 0x7f94b0108920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:19:28.811 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:28.804+0000 7f94b60b6700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94b0108e10 con 0x7f94b00f6510 2023-12-01T22:19:28.811 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:28.804+0000 7f94af7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94b00f6510 0x7f94b0108920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:19:28.811 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:28.804+0000 7f94af7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94b00f6510 0x7f94b0108920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:54684/0 (socket says 172.21.15.73:54684) 2023-12-01T22:19:28.811 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:28.804+0000 7f94af7fe700 1 -- 172.21.15.73:0/4217399100 learned_addr learned my addr 172.21.15.73:0/4217399100 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:19:28.811 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:28.804+0000 7f94af7fe700 1 -- 172.21.15.73:0/4217399100 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94a4005040 con 0x7f94b00f6510 2023-12-01T22:19:28.811 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:28.804+0000 7f94af7fe700 1 --2- 172.21.15.73:0/4217399100 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94b00f6510 0x7f94b0108920 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7f94a4013510 tx=0x7f94a400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:19:28.811 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:28.805+0000 7f9497fff700 1 -- 172.21.15.73:0/4217399100 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f94a4009320 con 0x7f94b00f6510 2023-12-01T22:19:28.811 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:28.805+0000 7f94b60b6700 1 -- 172.21.15.73:0/4217399100 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94b0109010 con 0x7f94b00f6510 2023-12-01T22:19:28.811 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:28.805+0000 7f9497fff700 1 -- 172.21.15.73:0/4217399100 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f94a4013f00 con 0x7f94b00f6510 2023-12-01T22:19:28.812 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:28.805+0000 7f9497fff700 1 -- 172.21.15.73:0/4217399100 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f94a4014d90 con 0x7f94b00f6510 2023-12-01T22:19:28.812 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:28.805+0000 7f94b60b6700 1 -- 172.21.15.73:0/4217399100 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94b01094a0 con 0x7f94b00f6510 2023-12-01T22:19:28.812 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:28.805+0000 7f9497fff700 1 -- 172.21.15.73:0/4217399100 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f94a40263a0 con 0x7f94b00f6510 2023-12-01T22:19:28.812 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:28.806+0000 7f9497fff700 1 --2- 172.21.15.73:0/4217399100 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f949802ff60 0x7f9498032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:19:28.812 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:28.806+0000 7f9497fff700 1 -- 172.21.15.73:0/4217399100 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f94a4040ff0 con 0x7f94b00f6510 2023-12-01T22:19:28.812 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:28.806+0000 7f94b60b6700 1 -- 172.21.15.73:0/4217399100 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94b0045ad0 con 0x7f94b00f6510 2023-12-01T22:19:28.812 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:28.806+0000 7f94aeffd700 1 --2- 172.21.15.73:0/4217399100 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f949802ff60 0x7f9498032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:19:28.812 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:28.806+0000 7f94aeffd700 1 --2- 172.21.15.73:0/4217399100 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f949802ff60 0x7f9498032420 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f94a00096a0 tx=0x7f94a0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:19:28.813 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:28.811+0000 7f9497fff700 1 -- 172.21.15.73:0/4217399100 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f94a4012920 con 0x7f94b00f6510 2023-12-01T22:19:29.019 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:29.018+0000 7f94b60b6700 1 -- 172.21.15.73:0/4217399100 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f94b00fc0b0 con 0x7f94b00f6510 2023-12-01T22:19:29.020 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:29.018+0000 7f9497fff700 1 -- 172.21.15.73:0/4217399100 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f94a402b020 con 0x7f94b00f6510 2023-12-01T22:19:29.020 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:19:29.020 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:19:29.022 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:29.020+0000 7f94b60b6700 1 -- 172.21.15.73:0/4217399100 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f949802ff60 msgr2=0x7f9498032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:19:29.022 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:29.020+0000 7f94b60b6700 1 --2- 172.21.15.73:0/4217399100 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f949802ff60 0x7f9498032420 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f94a00096a0 tx=0x7f94a0006b40).stop 2023-12-01T22:19:29.022 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:29.020+0000 7f94b60b6700 1 -- 172.21.15.73:0/4217399100 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94b00f6510 msgr2=0x7f94b0108920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:19:29.022 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:29.020+0000 7f94b60b6700 1 --2- 172.21.15.73:0/4217399100 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94b00f6510 0x7f94b0108920 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7f94a4013510 tx=0x7f94a400b040).stop 2023-12-01T22:19:29.022 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:29.020+0000 7f94b60b6700 1 -- 172.21.15.73:0/4217399100 shutdown_connections 2023-12-01T22:19:29.022 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:29.020+0000 7f94b60b6700 1 --2- 172.21.15.73:0/4217399100 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f949802ff60 0x7f9498032420 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:19:29.022 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:29.020+0000 7f94b60b6700 1 --2- 172.21.15.73:0/4217399100 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94b00f6510 0x7f94b0108920 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:19:29.022 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:29.020+0000 7f94b60b6700 1 -- 172.21.15.73:0/4217399100 >> 172.21.15.73:0/4217399100 conn(0x7f94b00f1c70 msgr2=0x7f94b00f26e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:19:29.022 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:29.020+0000 7f94b60b6700 1 -- 172.21.15.73:0/4217399100 shutdown_connections 2023-12-01T22:19:29.022 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:29.020+0000 7f94b60b6700 1 -- 172.21.15.73:0/4217399100 wait complete. 2023-12-01T22:19:29.023 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:19:30.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:19:29 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:19:28.700806+0000 mgr.smithi029.loxrfu (mgr.14164) 256 : cluster [DBG] pgmap v242: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:19:30.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:19:29 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:19:29.019835+0000 mon.smithi029 (mon.0) 310 : audit [DBG] from='client.? 172.21.15.73:0/4217399100' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:19:30.406 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:19:30.406 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:19:31.651 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:31.650+0000 7f7b51b1d700 1 -- 172.21.15.73:0/3121423445 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b4c0f5410 msgr2=0x7f7b4c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:19:31.652 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:31.650+0000 7f7b51b1d700 1 --2- 172.21.15.73:0/3121423445 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b4c0f5410 0x7f7b4c0f5830 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 rx=0x7f7b3c009fb0 tx=0x7f7b3c009b30).stop 2023-12-01T22:19:31.652 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:31.650+0000 7f7b51b1d700 1 -- 172.21.15.73:0/3121423445 shutdown_connections 2023-12-01T22:19:31.652 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:31.650+0000 7f7b51b1d700 1 --2- 172.21.15.73:0/3121423445 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b4c0f5410 0x7f7b4c0f5830 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:19:31.652 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:31.650+0000 7f7b51b1d700 1 -- 172.21.15.73:0/3121423445 >> 172.21.15.73:0/3121423445 conn(0x7f7b4c0f0b70 msgr2=0x7f7b4c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:19:31.652 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:31.650+0000 7f7b51b1d700 1 -- 172.21.15.73:0/3121423445 shutdown_connections 2023-12-01T22:19:31.652 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:31.650+0000 7f7b51b1d700 1 -- 172.21.15.73:0/3121423445 wait complete. 2023-12-01T22:19:31.652 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:31.650+0000 7f7b51b1d700 1 Processor -- start 2023-12-01T22:19:31.653 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:31.650+0000 7f7b51b1d700 1 -- start start 2023-12-01T22:19:31.653 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:31.650+0000 7f7b51b1d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b4c0f5410 0x7f7b4c0fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:19:31.653 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:31.650+0000 7f7b51b1d700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b4c0fb820 con 0x7f7b4c0f5410 2023-12-01T22:19:31.653 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:31.651+0000 7f7b4b7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b4c0f5410 0x7f7b4c0fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:19:31.653 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:31.651+0000 7f7b4b7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b4c0f5410 0x7f7b4c0fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:54712/0 (socket says 172.21.15.73:54712) 2023-12-01T22:19:31.654 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:31.651+0000 7f7b4b7fe700 1 -- 172.21.15.73:0/3863520033 learned_addr learned my addr 172.21.15.73:0/3863520033 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:19:31.654 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:31.651+0000 7f7b4b7fe700 1 -- 172.21.15.73:0/3863520033 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b3c005040 con 0x7f7b4c0f5410 2023-12-01T22:19:31.654 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:31.651+0000 7f7b4b7fe700 1 --2- 172.21.15.73:0/3863520033 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b4c0f5410 0x7f7b4c0fd3a0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7f7b3c013510 tx=0x7f7b3c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:19:31.654 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:31.651+0000 7f7b33fff700 1 -- 172.21.15.73:0/3863520033 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7b3c009320 con 0x7f7b4c0f5410 2023-12-01T22:19:31.654 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:31.651+0000 7f7b51b1d700 1 -- 172.21.15.73:0/3863520033 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b4c0fb9c0 con 0x7f7b4c0f5410 2023-12-01T22:19:31.654 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:31.651+0000 7f7b33fff700 1 -- 172.21.15.73:0/3863520033 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7b3c013f00 con 0x7f7b4c0f5410 2023-12-01T22:19:31.654 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:31.651+0000 7f7b33fff700 1 -- 172.21.15.73:0/3863520033 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7b3c014d90 con 0x7f7b4c0f5410 2023-12-01T22:19:31.654 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:31.651+0000 7f7b51b1d700 1 -- 172.21.15.73:0/3863520033 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b4c0fbe50 con 0x7f7b4c0f5410 2023-12-01T22:19:31.655 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:31.652+0000 7f7b33fff700 1 -- 172.21.15.73:0/3863520033 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7b3c0095f0 con 0x7f7b4c0f5410 2023-12-01T22:19:31.655 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:31.652+0000 7f7b51b1d700 1 -- 172.21.15.73:0/3863520033 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b4c059100 con 0x7f7b4c0f5410 2023-12-01T22:19:31.655 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:31.652+0000 7f7b33fff700 1 --2- 172.21.15.73:0/3863520033 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f7b3402ff60 0x7f7b34032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:19:31.655 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:31.652+0000 7f7b33fff700 1 -- 172.21.15.73:0/3863520033 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7b3c0410c0 con 0x7f7b4c0f5410 2023-12-01T22:19:31.655 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:31.653+0000 7f7b4affd700 1 --2- 172.21.15.73:0/3863520033 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f7b3402ff60 0x7f7b34032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:19:31.655 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:31.653+0000 7f7b4affd700 1 --2- 172.21.15.73:0/3863520033 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f7b3402ff60 0x7f7b34032420 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f7b400096a0 tx=0x7f7b40006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:19:31.659 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:31.657+0000 7f7b33fff700 1 -- 172.21.15.73:0/3863520033 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7b3c014420 con 0x7f7b4c0f5410 2023-12-01T22:19:31.776 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:19:31 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:19:30.701137+0000 mgr.smithi029.loxrfu (mgr.14164) 257 : cluster [DBG] pgmap v243: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:19:31.864 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:31.863+0000 7f7b51b1d700 1 -- 172.21.15.73:0/3863520033 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7b4c0fae60 con 0x7f7b4c0f5410 2023-12-01T22:19:31.865 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:31.863+0000 7f7b33fff700 1 -- 172.21.15.73:0/3863520033 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7b3c010230 con 0x7f7b4c0f5410 2023-12-01T22:19:31.866 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:19:31.866 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:19:31.867 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:31.865+0000 7f7b51b1d700 1 -- 172.21.15.73:0/3863520033 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f7b3402ff60 msgr2=0x7f7b34032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:19:31.868 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:31.865+0000 7f7b51b1d700 1 --2- 172.21.15.73:0/3863520033 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f7b3402ff60 0x7f7b34032420 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f7b400096a0 tx=0x7f7b40006b40).stop 2023-12-01T22:19:31.868 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:31.865+0000 7f7b51b1d700 1 -- 172.21.15.73:0/3863520033 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b4c0f5410 msgr2=0x7f7b4c0fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:19:31.868 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:31.865+0000 7f7b51b1d700 1 --2- 172.21.15.73:0/3863520033 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b4c0f5410 0x7f7b4c0fd3a0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7f7b3c013510 tx=0x7f7b3c00b040).stop 2023-12-01T22:19:31.868 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:31.865+0000 7f7b51b1d700 1 -- 172.21.15.73:0/3863520033 shutdown_connections 2023-12-01T22:19:31.868 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:31.865+0000 7f7b51b1d700 1 --2- 172.21.15.73:0/3863520033 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f7b3402ff60 0x7f7b34032420 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:19:31.868 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:31.865+0000 7f7b51b1d700 1 --2- 172.21.15.73:0/3863520033 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b4c0f5410 0x7f7b4c0fd3a0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:19:31.868 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:31.865+0000 7f7b51b1d700 1 -- 172.21.15.73:0/3863520033 >> 172.21.15.73:0/3863520033 conn(0x7f7b4c0f0b70 msgr2=0x7f7b4c05fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:19:31.869 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:31.865+0000 7f7b51b1d700 1 -- 172.21.15.73:0/3863520033 shutdown_connections 2023-12-01T22:19:31.869 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:31.865+0000 7f7b51b1d700 1 -- 172.21.15.73:0/3863520033 wait complete. 2023-12-01T22:19:31.869 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:19:32.776 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:19:32 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:19:31.864908+0000 mon.smithi029 (mon.0) 311 : audit [DBG] from='client.? 172.21.15.73:0/3863520033' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:19:33.270 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:19:33.270 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:19:33.776 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:19:33 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:19:32.701404+0000 mgr.smithi029.loxrfu (mgr.14164) 258 : cluster [DBG] pgmap v244: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:19:34.464 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:34.461+0000 7f9140cbc700 1 -- 172.21.15.73:0/1740145039 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f913c0f6510 msgr2=0x7f913c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:19:34.464 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:34.461+0000 7f9140cbc700 1 --2- 172.21.15.73:0/1740145039 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f913c0f6510 0x7f913c0f6930 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 rx=0x7f9124009fb0 tx=0x7f9124009b30).stop 2023-12-01T22:19:34.464 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:34.461+0000 7f9140cbc700 1 -- 172.21.15.73:0/1740145039 shutdown_connections 2023-12-01T22:19:34.464 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:34.461+0000 7f9140cbc700 1 --2- 172.21.15.73:0/1740145039 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f913c0f6510 0x7f913c0f6930 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:19:34.465 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:34.461+0000 7f9140cbc700 1 -- 172.21.15.73:0/1740145039 >> 172.21.15.73:0/1740145039 conn(0x7f913c0f1c70 msgr2=0x7f913c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:19:34.465 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:34.461+0000 7f9140cbc700 1 -- 172.21.15.73:0/1740145039 shutdown_connections 2023-12-01T22:19:34.465 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:34.461+0000 7f9140cbc700 1 -- 172.21.15.73:0/1740145039 wait complete. 2023-12-01T22:19:34.465 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:34.462+0000 7f9140cbc700 1 Processor -- start 2023-12-01T22:19:34.465 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:34.462+0000 7f9140cbc700 1 -- start start 2023-12-01T22:19:34.465 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:34.462+0000 7f9140cbc700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f913c0f6510 0x7f913c106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:19:34.465 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:34.462+0000 7f9140cbc700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f913c106c20 con 0x7f913c0f6510 2023-12-01T22:19:34.465 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:34.462+0000 7f913a59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f913c0f6510 0x7f913c106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:19:34.465 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:34.462+0000 7f913a59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f913c0f6510 0x7f913c106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:59824/0 (socket says 172.21.15.73:59824) 2023-12-01T22:19:34.465 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:34.462+0000 7f913a59c700 1 -- 172.21.15.73:0/1225024332 learned_addr learned my addr 172.21.15.73:0/1225024332 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:19:34.465 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:34.462+0000 7f913a59c700 1 -- 172.21.15.73:0/1225024332 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9124005040 con 0x7f913c0f6510 2023-12-01T22:19:34.465 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:34.463+0000 7f913a59c700 1 --2- 172.21.15.73:0/1225024332 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f913c0f6510 0x7f913c106730 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7f9124013510 tx=0x7f912400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:19:34.465 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:34.463+0000 7f9132ffd700 1 -- 172.21.15.73:0/1225024332 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9124009320 con 0x7f913c0f6510 2023-12-01T22:19:34.466 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:34.463+0000 7f9140cbc700 1 -- 172.21.15.73:0/1225024332 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f913c106e20 con 0x7f913c0f6510 2023-12-01T22:19:34.469 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:34.463+0000 7f9132ffd700 1 -- 172.21.15.73:0/1225024332 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9124013f00 con 0x7f913c0f6510 2023-12-01T22:19:34.469 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:34.463+0000 7f9132ffd700 1 -- 172.21.15.73:0/1225024332 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9124014d90 con 0x7f913c0f6510 2023-12-01T22:19:34.469 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:34.463+0000 7f9140cbc700 1 -- 172.21.15.73:0/1225024332 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f913c1072b0 con 0x7f913c0f6510 2023-12-01T22:19:34.469 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:34.464+0000 7f9132ffd700 1 -- 172.21.15.73:0/1225024332 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f91240263a0 con 0x7f913c0f6510 2023-12-01T22:19:34.469 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:34.464+0000 7f9132ffd700 1 --2- 172.21.15.73:0/1225024332 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f912802ff60 0x7f9128032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:19:34.469 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:34.464+0000 7f9132ffd700 1 -- 172.21.15.73:0/1225024332 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f91240412e0 con 0x7f913c0f6510 2023-12-01T22:19:34.469 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:34.464+0000 7f9140cbc700 1 -- 172.21.15.73:0/1225024332 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f913c045ad0 con 0x7f913c0f6510 2023-12-01T22:19:34.470 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:34.464+0000 7f9139d9b700 1 --2- 172.21.15.73:0/1225024332 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f912802ff60 0x7f9128032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:19:34.470 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:34.464+0000 7f9139d9b700 1 --2- 172.21.15.73:0/1225024332 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f912802ff60 0x7f9128032420 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f912c0096a0 tx=0x7f912c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:19:34.470 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:34.469+0000 7f9132ffd700 1 -- 172.21.15.73:0/1225024332 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9124012920 con 0x7f913c0f6510 2023-12-01T22:19:34.679 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:34.677+0000 7f9140cbc700 1 -- 172.21.15.73:0/1225024332 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f913c0fc050 con 0x7f913c0f6510 2023-12-01T22:19:34.679 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:34.678+0000 7f9132ffd700 1 -- 172.21.15.73:0/1225024332 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f912402b030 con 0x7f913c0f6510 2023-12-01T22:19:34.680 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:19:34.680 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:19:34.681 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:34.680+0000 7f9140cbc700 1 -- 172.21.15.73:0/1225024332 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f912802ff60 msgr2=0x7f9128032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:19:34.681 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:34.680+0000 7f9140cbc700 1 --2- 172.21.15.73:0/1225024332 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f912802ff60 0x7f9128032420 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f912c0096a0 tx=0x7f912c006b40).stop 2023-12-01T22:19:34.682 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:34.680+0000 7f9140cbc700 1 -- 172.21.15.73:0/1225024332 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f913c0f6510 msgr2=0x7f913c106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:19:34.682 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:34.680+0000 7f9140cbc700 1 --2- 172.21.15.73:0/1225024332 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f913c0f6510 0x7f913c106730 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7f9124013510 tx=0x7f912400b040).stop 2023-12-01T22:19:34.682 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:34.680+0000 7f9140cbc700 1 -- 172.21.15.73:0/1225024332 shutdown_connections 2023-12-01T22:19:34.682 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:34.680+0000 7f9140cbc700 1 --2- 172.21.15.73:0/1225024332 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f912802ff60 0x7f9128032420 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:19:34.682 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:34.680+0000 7f9140cbc700 1 --2- 172.21.15.73:0/1225024332 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f913c0f6510 0x7f913c106730 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:19:34.682 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:34.680+0000 7f9140cbc700 1 -- 172.21.15.73:0/1225024332 >> 172.21.15.73:0/1225024332 conn(0x7f913c0f1c70 msgr2=0x7f913c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:19:34.682 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:34.680+0000 7f9140cbc700 1 -- 172.21.15.73:0/1225024332 shutdown_connections 2023-12-01T22:19:34.682 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:34.680+0000 7f9140cbc700 1 -- 172.21.15.73:0/1225024332 wait complete. 2023-12-01T22:19:34.683 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:19:35.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:19:34 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:19:34.679253+0000 mon.smithi029 (mon.0) 312 : audit [DBG] from='client.? 172.21.15.73:0/1225024332' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:19:36.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:19:35 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:19:34.701643+0000 mgr.smithi029.loxrfu (mgr.14164) 259 : cluster [DBG] pgmap v245: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:19:36.071 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:19:36.071 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:19:37.304 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:37.298+0000 7f3c699d0700 1 -- 172.21.15.73:0/2044152734 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c640f5410 msgr2=0x7f3c640f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:19:37.304 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:37.298+0000 7f3c699d0700 1 --2- 172.21.15.73:0/2044152734 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c640f5410 0x7f3c640f5830 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 rx=0x7f3c4c009fb0 tx=0x7f3c4c009b30).stop 2023-12-01T22:19:37.304 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:37.298+0000 7f3c699d0700 1 -- 172.21.15.73:0/2044152734 shutdown_connections 2023-12-01T22:19:37.305 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:37.298+0000 7f3c699d0700 1 --2- 172.21.15.73:0/2044152734 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c640f5410 0x7f3c640f5830 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:19:37.305 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:37.298+0000 7f3c699d0700 1 -- 172.21.15.73:0/2044152734 >> 172.21.15.73:0/2044152734 conn(0x7f3c640f0b70 msgr2=0x7f3c640f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:19:37.305 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:37.298+0000 7f3c699d0700 1 -- 172.21.15.73:0/2044152734 shutdown_connections 2023-12-01T22:19:37.305 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:37.299+0000 7f3c699d0700 1 -- 172.21.15.73:0/2044152734 wait complete. 2023-12-01T22:19:37.305 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:37.299+0000 7f3c699d0700 1 Processor -- start 2023-12-01T22:19:37.305 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:37.299+0000 7f3c699d0700 1 -- start start 2023-12-01T22:19:37.305 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:37.299+0000 7f3c699d0700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c640f5410 0x7f3c64109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:19:37.305 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:37.299+0000 7f3c699d0700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c64060220 con 0x7f3c640f5410 2023-12-01T22:19:37.305 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:37.300+0000 7f3c62ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c640f5410 0x7f3c64109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:19:37.306 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:37.300+0000 7f3c62ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c640f5410 0x7f3c64109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:59846/0 (socket says 172.21.15.73:59846) 2023-12-01T22:19:37.306 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:37.300+0000 7f3c62ffd700 1 -- 172.21.15.73:0/456001266 learned_addr learned my addr 172.21.15.73:0/456001266 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:19:37.306 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:37.300+0000 7f3c62ffd700 1 -- 172.21.15.73:0/456001266 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c4c005040 con 0x7f3c640f5410 2023-12-01T22:19:37.306 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:37.300+0000 7f3c62ffd700 1 --2- 172.21.15.73:0/456001266 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c640f5410 0x7f3c64109450 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7f3c4c014000 tx=0x7f3c4c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:19:37.306 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:37.300+0000 7f3c5b7fe700 1 -- 172.21.15.73:0/456001266 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3c4c0095f0 con 0x7f3c640f5410 2023-12-01T22:19:37.306 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:37.300+0000 7f3c699d0700 1 -- 172.21.15.73:0/456001266 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c64060420 con 0x7f3c640f5410 2023-12-01T22:19:37.306 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:37.300+0000 7f3c5b7fe700 1 -- 172.21.15.73:0/456001266 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3c4c007e70 con 0x7f3c640f5410 2023-12-01T22:19:37.306 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:37.300+0000 7f3c5b7fe700 1 -- 172.21.15.73:0/456001266 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3c4c014ea0 con 0x7f3c640f5410 2023-12-01T22:19:37.307 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:37.300+0000 7f3c699d0700 1 -- 172.21.15.73:0/456001266 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c64109b00 con 0x7f3c640f5410 2023-12-01T22:19:37.307 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:37.301+0000 7f3c5b7fe700 1 -- 172.21.15.73:0/456001266 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3c4c014690 con 0x7f3c640f5410 2023-12-01T22:19:37.307 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:37.301+0000 7f3c699d0700 1 -- 172.21.15.73:0/456001266 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c64059100 con 0x7f3c640f5410 2023-12-01T22:19:37.307 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:37.301+0000 7f3c5b7fe700 1 --2- 172.21.15.73:0/456001266 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f3c5002ffb0 0x7f3c50032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:19:37.307 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:37.301+0000 7f3c5b7fe700 1 -- 172.21.15.73:0/456001266 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3c4c0408f0 con 0x7f3c640f5410 2023-12-01T22:19:37.307 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:37.301+0000 7f3c627fc700 1 --2- 172.21.15.73:0/456001266 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f3c5002ffb0 0x7f3c50032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:19:37.307 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:37.302+0000 7f3c627fc700 1 --2- 172.21.15.73:0/456001266 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f3c5002ffb0 0x7f3c50032470 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f3c540096a0 tx=0x7f3c54006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:19:37.308 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:37.306+0000 7f3c5b7fe700 1 -- 172.21.15.73:0/456001266 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3c4c00f300 con 0x7f3c640f5410 2023-12-01T22:19:37.511 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:37.510+0000 7f3c699d0700 1 -- 172.21.15.73:0/456001266 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3c64045ad0 con 0x7f3c640f5410 2023-12-01T22:19:37.512 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:37.510+0000 7f3c5b7fe700 1 -- 172.21.15.73:0/456001266 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3c4c00f300 con 0x7f3c640f5410 2023-12-01T22:19:37.512 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:19:37.512 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:19:37.514 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:37.512+0000 7f3c699d0700 1 -- 172.21.15.73:0/456001266 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f3c5002ffb0 msgr2=0x7f3c50032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:19:37.514 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:37.512+0000 7f3c699d0700 1 --2- 172.21.15.73:0/456001266 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f3c5002ffb0 0x7f3c50032470 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f3c540096a0 tx=0x7f3c54006b40).stop 2023-12-01T22:19:37.514 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:37.512+0000 7f3c699d0700 1 -- 172.21.15.73:0/456001266 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c640f5410 msgr2=0x7f3c64109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:19:37.514 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:37.512+0000 7f3c699d0700 1 --2- 172.21.15.73:0/456001266 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c640f5410 0x7f3c64109450 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7f3c4c014000 tx=0x7f3c4c00b040).stop 2023-12-01T22:19:37.514 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:37.512+0000 7f3c699d0700 1 -- 172.21.15.73:0/456001266 shutdown_connections 2023-12-01T22:19:37.514 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:37.512+0000 7f3c699d0700 1 --2- 172.21.15.73:0/456001266 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f3c5002ffb0 0x7f3c50032470 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:19:37.514 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:37.512+0000 7f3c699d0700 1 --2- 172.21.15.73:0/456001266 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c640f5410 0x7f3c64109450 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:19:37.514 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:37.512+0000 7f3c699d0700 1 -- 172.21.15.73:0/456001266 >> 172.21.15.73:0/456001266 conn(0x7f3c640f0b70 msgr2=0x7f3c640f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:19:37.514 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:37.512+0000 7f3c699d0700 1 -- 172.21.15.73:0/456001266 shutdown_connections 2023-12-01T22:19:37.515 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:37.512+0000 7f3c699d0700 1 -- 172.21.15.73:0/456001266 wait complete. 2023-12-01T22:19:37.515 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:19:38.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:19:37 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:19:36.701968+0000 mgr.smithi029.loxrfu (mgr.14164) 260 : cluster [DBG] pgmap v246: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:19:38.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:19:37 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:19:37.511949+0000 mon.smithi029 (mon.0) 313 : audit [DBG] from='client.? 172.21.15.73:0/456001266' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:19:38.899 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:19:38.899 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:19:40.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:19:39 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:19:38.702246+0000 mgr.smithi029.loxrfu (mgr.14164) 261 : cluster [DBG] pgmap v247: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:19:40.126 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:40.124+0000 7f3783d23700 1 -- 172.21.15.73:0/3766694088 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f377c0f5410 msgr2=0x7f377c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:19:40.126 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:40.124+0000 7f3783d23700 1 --2- 172.21.15.73:0/3766694088 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f377c0f5410 0x7f377c0f5830 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 rx=0x7f3778009fb0 tx=0x7f3778009b30).stop 2023-12-01T22:19:40.126 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:40.124+0000 7f3783d23700 1 -- 172.21.15.73:0/3766694088 shutdown_connections 2023-12-01T22:19:40.126 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:40.124+0000 7f3783d23700 1 --2- 172.21.15.73:0/3766694088 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f377c0f5410 0x7f377c0f5830 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:19:40.126 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:40.124+0000 7f3783d23700 1 -- 172.21.15.73:0/3766694088 >> 172.21.15.73:0/3766694088 conn(0x7f377c0f0b70 msgr2=0x7f377c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:19:40.126 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:40.124+0000 7f3783d23700 1 -- 172.21.15.73:0/3766694088 shutdown_connections 2023-12-01T22:19:40.126 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:40.124+0000 7f3783d23700 1 -- 172.21.15.73:0/3766694088 wait complete. 2023-12-01T22:19:40.126 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:40.124+0000 7f3783d23700 1 Processor -- start 2023-12-01T22:19:40.127 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:40.125+0000 7f3783d23700 1 -- start start 2023-12-01T22:19:40.128 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:40.125+0000 7f3783d23700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f377c0f5410 0x7f377c1093f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:19:40.128 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:40.125+0000 7f3783d23700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f377c060280 con 0x7f377c0f5410 2023-12-01T22:19:40.128 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:40.125+0000 7f3781abf700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f377c0f5410 0x7f377c1093f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:19:40.128 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:40.125+0000 7f3781abf700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f377c0f5410 0x7f377c1093f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:59872/0 (socket says 172.21.15.73:59872) 2023-12-01T22:19:40.128 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:40.125+0000 7f3781abf700 1 -- 172.21.15.73:0/2995508473 learned_addr learned my addr 172.21.15.73:0/2995508473 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:19:40.128 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:40.125+0000 7f3781abf700 1 -- 172.21.15.73:0/2995508473 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3778005040 con 0x7f377c0f5410 2023-12-01T22:19:40.128 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:40.126+0000 7f3781abf700 1 --2- 172.21.15.73:0/2995508473 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f377c0f5410 0x7f377c1093f0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7f3778013510 tx=0x7f377800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:19:40.128 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:40.126+0000 7f37727fc700 1 -- 172.21.15.73:0/2995508473 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3778009320 con 0x7f377c0f5410 2023-12-01T22:19:40.128 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:40.126+0000 7f3783d23700 1 -- 172.21.15.73:0/2995508473 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f377c060480 con 0x7f377c0f5410 2023-12-01T22:19:40.128 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:40.126+0000 7f37727fc700 1 -- 172.21.15.73:0/2995508473 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3778013f00 con 0x7f377c0f5410 2023-12-01T22:19:40.128 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:40.126+0000 7f37727fc700 1 -- 172.21.15.73:0/2995508473 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3778014d90 con 0x7f377c0f5410 2023-12-01T22:19:40.128 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:40.126+0000 7f3783d23700 1 -- 172.21.15.73:0/2995508473 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f377c109b20 con 0x7f377c0f5410 2023-12-01T22:19:40.129 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:40.127+0000 7f37727fc700 1 -- 172.21.15.73:0/2995508473 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f37780095f0 con 0x7f377c0f5410 2023-12-01T22:19:40.129 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:40.127+0000 7f3783d23700 1 -- 172.21.15.73:0/2995508473 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f377c045ad0 con 0x7f377c0f5410 2023-12-01T22:19:40.129 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:40.127+0000 7f37727fc700 1 --2- 172.21.15.73:0/2995508473 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f376802ff60 0x7f3768032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:19:40.129 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:40.127+0000 7f37727fc700 1 -- 172.21.15.73:0/2995508473 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3778041000 con 0x7f377c0f5410 2023-12-01T22:19:40.130 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:40.127+0000 7f37812be700 1 --2- 172.21.15.73:0/2995508473 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f376802ff60 0x7f3768032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:19:40.130 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:40.128+0000 7f37812be700 1 --2- 172.21.15.73:0/2995508473 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f376802ff60 0x7f3768032420 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f376c0096a0 tx=0x7f376c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:19:40.133 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:40.132+0000 7f37727fc700 1 -- 172.21.15.73:0/2995508473 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3778014420 con 0x7f377c0f5410 2023-12-01T22:19:40.338 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:40.336+0000 7f3783d23700 1 -- 172.21.15.73:0/2995508473 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f377c10a340 con 0x7f377c0f5410 2023-12-01T22:19:40.339 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:40.337+0000 7f37727fc700 1 -- 172.21.15.73:0/2995508473 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f377801d7f0 con 0x7f377c0f5410 2023-12-01T22:19:40.339 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:19:40.339 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:19:40.340 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:40.339+0000 7f3783d23700 1 -- 172.21.15.73:0/2995508473 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f376802ff60 msgr2=0x7f3768032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:19:40.341 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:40.339+0000 7f3783d23700 1 --2- 172.21.15.73:0/2995508473 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f376802ff60 0x7f3768032420 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f376c0096a0 tx=0x7f376c006b40).stop 2023-12-01T22:19:40.341 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:40.339+0000 7f3783d23700 1 -- 172.21.15.73:0/2995508473 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f377c0f5410 msgr2=0x7f377c1093f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:19:40.341 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:40.339+0000 7f3783d23700 1 --2- 172.21.15.73:0/2995508473 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f377c0f5410 0x7f377c1093f0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7f3778013510 tx=0x7f377800b040).stop 2023-12-01T22:19:40.341 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:40.339+0000 7f3783d23700 1 -- 172.21.15.73:0/2995508473 shutdown_connections 2023-12-01T22:19:40.341 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:40.339+0000 7f3783d23700 1 --2- 172.21.15.73:0/2995508473 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f376802ff60 0x7f3768032420 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:19:40.341 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:40.339+0000 7f3783d23700 1 --2- 172.21.15.73:0/2995508473 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f377c0f5410 0x7f377c1093f0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:19:40.341 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:40.339+0000 7f3783d23700 1 -- 172.21.15.73:0/2995508473 >> 172.21.15.73:0/2995508473 conn(0x7f377c0f0b70 msgr2=0x7f377c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:19:40.341 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:40.339+0000 7f3783d23700 1 -- 172.21.15.73:0/2995508473 shutdown_connections 2023-12-01T22:19:40.341 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:40.339+0000 7f3783d23700 1 -- 172.21.15.73:0/2995508473 wait complete. 2023-12-01T22:19:40.341 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:19:41.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:19:40 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:19:40.338557+0000 mon.smithi029 (mon.0) 314 : audit [DBG] from='client.? 172.21.15.73:0/2995508473' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:19:41.745 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:19:41.745 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:19:42.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:19:41 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:19:40.702584+0000 mgr.smithi029.loxrfu (mgr.14164) 262 : cluster [DBG] pgmap v248: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:19:42.951 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:42.948+0000 7f89c0cc0700 1 -- 172.21.15.73:0/1678182467 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f89bc0f5410 msgr2=0x7f89bc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:19:42.951 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:42.948+0000 7f89c0cc0700 1 --2- 172.21.15.73:0/1678182467 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f89bc0f5410 0x7f89bc0f5830 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 rx=0x7f89b0009fb0 tx=0x7f89b0009b30).stop 2023-12-01T22:19:42.951 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:42.948+0000 7f89c0cc0700 1 -- 172.21.15.73:0/1678182467 shutdown_connections 2023-12-01T22:19:42.951 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:42.948+0000 7f89c0cc0700 1 --2- 172.21.15.73:0/1678182467 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f89bc0f5410 0x7f89bc0f5830 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:19:42.951 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:42.948+0000 7f89c0cc0700 1 -- 172.21.15.73:0/1678182467 >> 172.21.15.73:0/1678182467 conn(0x7f89bc0f0b70 msgr2=0x7f89bc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:19:42.951 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:42.948+0000 7f89c0cc0700 1 -- 172.21.15.73:0/1678182467 shutdown_connections 2023-12-01T22:19:42.951 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:42.948+0000 7f89c0cc0700 1 -- 172.21.15.73:0/1678182467 wait complete. 2023-12-01T22:19:42.951 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:42.948+0000 7f89c0cc0700 1 Processor -- start 2023-12-01T22:19:42.951 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:42.948+0000 7f89c0cc0700 1 -- start start 2023-12-01T22:19:42.951 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:42.949+0000 7f89c0cc0700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f89bc0f5410 0x7f89bc108910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:19:42.951 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:42.949+0000 7f89c0cc0700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89bc108e00 con 0x7f89bc0f5410 2023-12-01T22:19:42.951 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:42.949+0000 7f89ba59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f89bc0f5410 0x7f89bc108910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:19:42.952 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:42.949+0000 7f89ba59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f89bc0f5410 0x7f89bc108910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:59894/0 (socket says 172.21.15.73:59894) 2023-12-01T22:19:42.952 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:42.949+0000 7f89ba59c700 1 -- 172.21.15.73:0/1614036619 learned_addr learned my addr 172.21.15.73:0/1614036619 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:19:42.952 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:42.949+0000 7f89ba59c700 1 -- 172.21.15.73:0/1614036619 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89b0005040 con 0x7f89bc0f5410 2023-12-01T22:19:42.953 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:42.949+0000 7f89ba59c700 1 --2- 172.21.15.73:0/1614036619 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f89bc0f5410 0x7f89bc108910 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7f89b0013510 tx=0x7f89b000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:19:42.953 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:42.950+0000 7f89aaffd700 1 -- 172.21.15.73:0/1614036619 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f89b0009320 con 0x7f89bc0f5410 2023-12-01T22:19:42.953 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:42.950+0000 7f89aaffd700 1 -- 172.21.15.73:0/1614036619 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f89b0013f00 con 0x7f89bc0f5410 2023-12-01T22:19:42.953 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:42.950+0000 7f89aaffd700 1 -- 172.21.15.73:0/1614036619 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f89b0014d90 con 0x7f89bc0f5410 2023-12-01T22:19:42.953 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:42.950+0000 7f89c0cc0700 1 -- 172.21.15.73:0/1614036619 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89bc109000 con 0x7f89bc0f5410 2023-12-01T22:19:42.953 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:42.950+0000 7f89c0cc0700 1 -- 172.21.15.73:0/1614036619 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89bc109490 con 0x7f89bc0f5410 2023-12-01T22:19:42.953 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:42.950+0000 7f89aaffd700 1 -- 172.21.15.73:0/1614036619 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f89b00095f0 con 0x7f89bc0f5410 2023-12-01T22:19:42.953 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:42.950+0000 7f89c0cc0700 1 -- 172.21.15.73:0/1614036619 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89bc045ad0 con 0x7f89bc0f5410 2023-12-01T22:19:42.953 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:42.951+0000 7f89aaffd700 1 --2- 172.21.15.73:0/1614036619 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f89a402ff60 0x7f89a4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:19:42.953 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:42.951+0000 7f89aaffd700 1 -- 172.21.15.73:0/1614036619 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f89b0041020 con 0x7f89bc0f5410 2023-12-01T22:19:42.953 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:42.951+0000 7f89b9d9b700 1 --2- 172.21.15.73:0/1614036619 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f89a402ff60 0x7f89a4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:19:42.953 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:42.951+0000 7f89b9d9b700 1 --2- 172.21.15.73:0/1614036619 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f89a402ff60 0x7f89a4032420 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f89ac0096a0 tx=0x7f89ac006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:19:42.957 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:42.956+0000 7f89aaffd700 1 -- 172.21.15.73:0/1614036619 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f89b0014420 con 0x7f89bc0f5410 2023-12-01T22:19:43.160 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:43.158+0000 7f89c0cc0700 1 -- 172.21.15.73:0/1614036619 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f89bc0fae60 con 0x7f89bc0f5410 2023-12-01T22:19:43.161 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:43.159+0000 7f89aaffd700 1 -- 172.21.15.73:0/1614036619 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f89b001d7f0 con 0x7f89bc0f5410 2023-12-01T22:19:43.161 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:19:43.161 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:19:43.162 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:43.160+0000 7f89c0cc0700 1 -- 172.21.15.73:0/1614036619 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f89a402ff60 msgr2=0x7f89a4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:19:43.162 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:43.160+0000 7f89c0cc0700 1 --2- 172.21.15.73:0/1614036619 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f89a402ff60 0x7f89a4032420 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f89ac0096a0 tx=0x7f89ac006b40).stop 2023-12-01T22:19:43.162 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:43.160+0000 7f89c0cc0700 1 -- 172.21.15.73:0/1614036619 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f89bc0f5410 msgr2=0x7f89bc108910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:19:43.163 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:43.160+0000 7f89c0cc0700 1 --2- 172.21.15.73:0/1614036619 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f89bc0f5410 0x7f89bc108910 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7f89b0013510 tx=0x7f89b000b040).stop 2023-12-01T22:19:43.163 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:43.160+0000 7f89c0cc0700 1 -- 172.21.15.73:0/1614036619 shutdown_connections 2023-12-01T22:19:43.163 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:43.160+0000 7f89c0cc0700 1 --2- 172.21.15.73:0/1614036619 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f89a402ff60 0x7f89a4032420 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:19:43.163 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:43.161+0000 7f89c0cc0700 1 --2- 172.21.15.73:0/1614036619 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f89bc0f5410 0x7f89bc108910 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:19:43.163 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:43.161+0000 7f89c0cc0700 1 -- 172.21.15.73:0/1614036619 >> 172.21.15.73:0/1614036619 conn(0x7f89bc0f0b70 msgr2=0x7f89bc05fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:19:43.163 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:43.161+0000 7f89c0cc0700 1 -- 172.21.15.73:0/1614036619 shutdown_connections 2023-12-01T22:19:43.163 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:43.161+0000 7f89c0cc0700 1 -- 172.21.15.73:0/1614036619 wait complete. 2023-12-01T22:19:43.163 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:19:44.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:19:43 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:19:42.703008+0000 mgr.smithi029.loxrfu (mgr.14164) 263 : cluster [DBG] pgmap v249: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:19:44.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:19:43 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:19:43.160472+0000 mon.smithi029 (mon.0) 315 : audit [DBG] from='client.? 172.21.15.73:0/1614036619' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:19:44.541 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:19:44.541 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:19:45.744 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:45.741+0000 7f0d82d17700 1 -- 172.21.15.73:0/499931509 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0d7c0601e0 msgr2=0x7f0d7c0f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:19:45.744 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:45.741+0000 7f0d82d17700 1 --2- 172.21.15.73:0/499931509 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0d7c0601e0 0x7f0d7c0f8f20 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 rx=0x7f0d78009fb0 tx=0x7f0d78009b30).stop 2023-12-01T22:19:45.744 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:45.741+0000 7f0d82d17700 1 -- 172.21.15.73:0/499931509 shutdown_connections 2023-12-01T22:19:45.744 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:45.741+0000 7f0d82d17700 1 --2- 172.21.15.73:0/499931509 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0d7c0601e0 0x7f0d7c0f8f20 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:19:45.745 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:45.741+0000 7f0d82d17700 1 -- 172.21.15.73:0/499931509 >> 172.21.15.73:0/499931509 conn(0x7f0d7c0f09a0 msgr2=0x7f0d7c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:19:45.745 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:45.742+0000 7f0d82d17700 1 -- 172.21.15.73:0/499931509 shutdown_connections 2023-12-01T22:19:45.745 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:45.742+0000 7f0d82d17700 1 -- 172.21.15.73:0/499931509 wait complete. 2023-12-01T22:19:45.745 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:45.742+0000 7f0d82d17700 1 Processor -- start 2023-12-01T22:19:45.745 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:45.742+0000 7f0d82d17700 1 -- start start 2023-12-01T22:19:45.745 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:45.742+0000 7f0d82d17700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0d7c0601e0 0x7f0d7c106500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:19:45.745 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:45.742+0000 7f0d82d17700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d7c1069f0 con 0x7f0d7c0601e0 2023-12-01T22:19:45.745 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:45.743+0000 7f0d81d15700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0d7c0601e0 0x7f0d7c106500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:19:45.745 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:45.743+0000 7f0d81d15700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0d7c0601e0 0x7f0d7c106500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:41500/0 (socket says 172.21.15.73:41500) 2023-12-01T22:19:45.745 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:45.743+0000 7f0d81d15700 1 -- 172.21.15.73:0/237350569 learned_addr learned my addr 172.21.15.73:0/237350569 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:19:45.747 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:45.743+0000 7f0d81d15700 1 -- 172.21.15.73:0/237350569 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d78005040 con 0x7f0d7c0601e0 2023-12-01T22:19:45.747 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:45.743+0000 7f0d81d15700 1 --2- 172.21.15.73:0/237350569 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0d7c0601e0 0x7f0d7c106500 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7f0d78013510 tx=0x7f0d7800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:19:45.747 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:45.743+0000 7f0d727fc700 1 -- 172.21.15.73:0/237350569 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0d78009320 con 0x7f0d7c0601e0 2023-12-01T22:19:45.747 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:45.743+0000 7f0d82d17700 1 -- 172.21.15.73:0/237350569 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d7c106bf0 con 0x7f0d7c0601e0 2023-12-01T22:19:45.747 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:45.743+0000 7f0d727fc700 1 -- 172.21.15.73:0/237350569 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0d78013f00 con 0x7f0d7c0601e0 2023-12-01T22:19:45.747 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:45.743+0000 7f0d727fc700 1 -- 172.21.15.73:0/237350569 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0d78014d90 con 0x7f0d7c0601e0 2023-12-01T22:19:45.747 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:45.743+0000 7f0d82d17700 1 -- 172.21.15.73:0/237350569 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d7c107080 con 0x7f0d7c0601e0 2023-12-01T22:19:45.747 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:45.744+0000 7f0d727fc700 1 -- 172.21.15.73:0/237350569 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0d78014420 con 0x7f0d7c0601e0 2023-12-01T22:19:45.747 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:45.744+0000 7f0d82d17700 1 -- 172.21.15.73:0/237350569 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d7c045ad0 con 0x7f0d7c0601e0 2023-12-01T22:19:45.747 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:45.744+0000 7f0d727fc700 1 --2- 172.21.15.73:0/237350569 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f0d6802ff60 0x7f0d68032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:19:45.747 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:45.744+0000 7f0d727fc700 1 -- 172.21.15.73:0/237350569 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0d78041050 con 0x7f0d7c0601e0 2023-12-01T22:19:45.747 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:45.744+0000 7f0d81514700 1 --2- 172.21.15.73:0/237350569 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f0d6802ff60 0x7f0d68032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:19:45.747 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:45.745+0000 7f0d81514700 1 --2- 172.21.15.73:0/237350569 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f0d6802ff60 0x7f0d68032420 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f0d6c0096a0 tx=0x7f0d6c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:19:45.751 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:45.749+0000 7f0d727fc700 1 -- 172.21.15.73:0/237350569 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0d780095f0 con 0x7f0d7c0601e0 2023-12-01T22:19:45.954 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:45.953+0000 7f0d82d17700 1 -- 172.21.15.73:0/237350569 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0d7c107ed0 con 0x7f0d7c0601e0 2023-12-01T22:19:45.955 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:45.953+0000 7f0d727fc700 1 -- 172.21.15.73:0/237350569 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0d7801db70 con 0x7f0d7c0601e0 2023-12-01T22:19:45.955 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:19:45.955 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:19:45.957 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:45.955+0000 7f0d82d17700 1 -- 172.21.15.73:0/237350569 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f0d6802ff60 msgr2=0x7f0d68032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:19:45.957 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:45.955+0000 7f0d82d17700 1 --2- 172.21.15.73:0/237350569 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f0d6802ff60 0x7f0d68032420 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f0d6c0096a0 tx=0x7f0d6c006b40).stop 2023-12-01T22:19:45.957 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:45.955+0000 7f0d82d17700 1 -- 172.21.15.73:0/237350569 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0d7c0601e0 msgr2=0x7f0d7c106500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:19:45.957 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:45.955+0000 7f0d82d17700 1 --2- 172.21.15.73:0/237350569 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0d7c0601e0 0x7f0d7c106500 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7f0d78013510 tx=0x7f0d7800b040).stop 2023-12-01T22:19:45.957 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:45.955+0000 7f0d82d17700 1 -- 172.21.15.73:0/237350569 shutdown_connections 2023-12-01T22:19:45.957 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:45.955+0000 7f0d82d17700 1 --2- 172.21.15.73:0/237350569 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f0d6802ff60 0x7f0d68032420 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:19:45.957 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:45.955+0000 7f0d82d17700 1 --2- 172.21.15.73:0/237350569 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0d7c0601e0 0x7f0d7c106500 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:19:45.957 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:45.955+0000 7f0d82d17700 1 -- 172.21.15.73:0/237350569 >> 172.21.15.73:0/237350569 conn(0x7f0d7c0f09a0 msgr2=0x7f0d7c0f1420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:19:45.957 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:45.955+0000 7f0d82d17700 1 -- 172.21.15.73:0/237350569 shutdown_connections 2023-12-01T22:19:45.957 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:45.955+0000 7f0d82d17700 1 -- 172.21.15.73:0/237350569 wait complete. 2023-12-01T22:19:45.958 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:19:46.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:19:45 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:19:44.703276+0000 mgr.smithi029.loxrfu (mgr.14164) 264 : cluster [DBG] pgmap v250: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:19:47.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:19:46 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:19:45.954844+0000 mon.smithi029 (mon.0) 316 : audit [DBG] from='client.? 172.21.15.73:0/237350569' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:19:47.347 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:19:47.347 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:19:48.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:19:47 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:19:46.703633+0000 mgr.smithi029.loxrfu (mgr.14164) 265 : cluster [DBG] pgmap v251: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:19:48.591 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:48.589+0000 7f2d95035700 1 -- 172.21.15.73:0/425358028 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d900601e0 msgr2=0x7f2d900f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:19:48.591 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:48.589+0000 7f2d95035700 1 --2- 172.21.15.73:0/425358028 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d900601e0 0x7f2d900f8f20 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 rx=0x7f2d80009fb0 tx=0x7f2d80009b30).stop 2023-12-01T22:19:48.591 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:48.589+0000 7f2d95035700 1 -- 172.21.15.73:0/425358028 shutdown_connections 2023-12-01T22:19:48.591 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:48.589+0000 7f2d95035700 1 --2- 172.21.15.73:0/425358028 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d900601e0 0x7f2d900f8f20 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:19:48.591 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:48.589+0000 7f2d95035700 1 -- 172.21.15.73:0/425358028 >> 172.21.15.73:0/425358028 conn(0x7f2d900f09a0 msgr2=0x7f2d900f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:19:48.592 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:48.589+0000 7f2d95035700 1 -- 172.21.15.73:0/425358028 shutdown_connections 2023-12-01T22:19:48.592 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:48.589+0000 7f2d95035700 1 -- 172.21.15.73:0/425358028 wait complete. 2023-12-01T22:19:48.592 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:48.589+0000 7f2d95035700 1 Processor -- start 2023-12-01T22:19:48.592 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:48.589+0000 7f2d95035700 1 -- start start 2023-12-01T22:19:48.592 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:48.589+0000 7f2d95035700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d900601e0 0x7f2d900f69a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:19:48.592 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:48.589+0000 7f2d95035700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d900f4e20 con 0x7f2d900601e0 2023-12-01T22:19:48.593 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:48.590+0000 7f2d8ffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d900601e0 0x7f2d900f69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:19:48.594 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:48.590+0000 7f2d8ffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d900601e0 0x7f2d900f69a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:41530/0 (socket says 172.21.15.73:41530) 2023-12-01T22:19:48.594 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:48.590+0000 7f2d8ffff700 1 -- 172.21.15.73:0/465380015 learned_addr learned my addr 172.21.15.73:0/465380015 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:19:48.594 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:48.590+0000 7f2d8ffff700 1 -- 172.21.15.73:0/465380015 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d80005040 con 0x7f2d900601e0 2023-12-01T22:19:48.594 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:48.590+0000 7f2d8ffff700 1 --2- 172.21.15.73:0/465380015 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d900601e0 0x7f2d900f69a0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7f2d80013510 tx=0x7f2d8000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:19:48.594 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:48.590+0000 7f2d8cff9700 1 -- 172.21.15.73:0/465380015 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2d80009320 con 0x7f2d900601e0 2023-12-01T22:19:48.594 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:48.590+0000 7f2d95035700 1 -- 172.21.15.73:0/465380015 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d900f4fc0 con 0x7f2d900601e0 2023-12-01T22:19:48.594 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:48.590+0000 7f2d8cff9700 1 -- 172.21.15.73:0/465380015 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2d80013f00 con 0x7f2d900601e0 2023-12-01T22:19:48.594 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:48.590+0000 7f2d8cff9700 1 -- 172.21.15.73:0/465380015 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2d80014d90 con 0x7f2d900601e0 2023-12-01T22:19:48.594 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:48.590+0000 7f2d95035700 1 -- 172.21.15.73:0/465380015 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d900f5450 con 0x7f2d900601e0 2023-12-01T22:19:48.594 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:48.591+0000 7f2d8cff9700 1 -- 172.21.15.73:0/465380015 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2d800095f0 con 0x7f2d900601e0 2023-12-01T22:19:48.594 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:48.591+0000 7f2d95035700 1 -- 172.21.15.73:0/465380015 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d90059100 con 0x7f2d900601e0 2023-12-01T22:19:48.594 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:48.591+0000 7f2d8cff9700 1 --2- 172.21.15.73:0/465380015 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f2d7802ff60 0x7f2d78032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:19:48.594 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:48.591+0000 7f2d8cff9700 1 -- 172.21.15.73:0/465380015 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2d800410a0 con 0x7f2d900601e0 2023-12-01T22:19:48.595 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:48.592+0000 7f2d8f7fe700 1 --2- 172.21.15.73:0/465380015 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f2d7802ff60 0x7f2d78032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:19:48.595 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:48.592+0000 7f2d8f7fe700 1 --2- 172.21.15.73:0/465380015 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f2d7802ff60 0x7f2d78032420 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f2d840096a0 tx=0x7f2d84006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:19:48.598 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:48.596+0000 7f2d8cff9700 1 -- 172.21.15.73:0/465380015 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2d8001d7f0 con 0x7f2d900601e0 2023-12-01T22:19:48.802 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:48.800+0000 7f2d95035700 1 -- 172.21.15.73:0/465380015 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2d900fbcd0 con 0x7f2d900601e0 2023-12-01T22:19:48.803 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:48.801+0000 7f2d8cff9700 1 -- 172.21.15.73:0/465380015 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2d80010230 con 0x7f2d900601e0 2023-12-01T22:19:48.803 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:19:48.803 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:19:48.805 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:48.802+0000 7f2d95035700 1 -- 172.21.15.73:0/465380015 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f2d7802ff60 msgr2=0x7f2d78032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:19:48.805 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:48.802+0000 7f2d95035700 1 --2- 172.21.15.73:0/465380015 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f2d7802ff60 0x7f2d78032420 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f2d840096a0 tx=0x7f2d84006b40).stop 2023-12-01T22:19:48.805 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:48.802+0000 7f2d95035700 1 -- 172.21.15.73:0/465380015 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d900601e0 msgr2=0x7f2d900f69a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:19:48.805 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:48.802+0000 7f2d95035700 1 --2- 172.21.15.73:0/465380015 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d900601e0 0x7f2d900f69a0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7f2d80013510 tx=0x7f2d8000b040).stop 2023-12-01T22:19:48.805 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:48.802+0000 7f2d95035700 1 -- 172.21.15.73:0/465380015 shutdown_connections 2023-12-01T22:19:48.806 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:48.802+0000 7f2d95035700 1 --2- 172.21.15.73:0/465380015 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f2d7802ff60 0x7f2d78032420 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:19:48.806 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:48.802+0000 7f2d95035700 1 --2- 172.21.15.73:0/465380015 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d900601e0 0x7f2d900f69a0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:19:48.806 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:48.802+0000 7f2d95035700 1 -- 172.21.15.73:0/465380015 >> 172.21.15.73:0/465380015 conn(0x7f2d900f09a0 msgr2=0x7f2d900f1420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:19:48.806 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:48.803+0000 7f2d95035700 1 -- 172.21.15.73:0/465380015 shutdown_connections 2023-12-01T22:19:48.806 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:48.803+0000 7f2d95035700 1 -- 172.21.15.73:0/465380015 wait complete. 2023-12-01T22:19:48.806 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:19:50.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:19:49 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:19:48.703936+0000 mgr.smithi029.loxrfu (mgr.14164) 266 : cluster [DBG] pgmap v252: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:19:50.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:19:49 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:19:48.802409+0000 mon.smithi029 (mon.0) 317 : audit [DBG] from='client.? 172.21.15.73:0/465380015' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:19:50.197 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:19:50.198 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:19:51.405 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:51.403+0000 7fc80759e700 1 -- 172.21.15.73:0/565401015 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc8080f62a0 msgr2=0x7fc8080f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:19:51.405 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:51.403+0000 7fc80759e700 1 --2- 172.21.15.73:0/565401015 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc8080f62a0 0x7fc8080f66c0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 rx=0x7fc7f8009fb0 tx=0x7fc7f8009b30).stop 2023-12-01T22:19:51.405 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:51.403+0000 7fc80759e700 1 -- 172.21.15.73:0/565401015 shutdown_connections 2023-12-01T22:19:51.405 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:51.403+0000 7fc80759e700 1 --2- 172.21.15.73:0/565401015 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc8080f62a0 0x7fc8080f66c0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:19:51.405 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:51.403+0000 7fc80759e700 1 -- 172.21.15.73:0/565401015 >> 172.21.15.73:0/565401015 conn(0x7fc8080f1aa0 msgr2=0x7fc8080f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:19:51.405 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:51.403+0000 7fc80759e700 1 -- 172.21.15.73:0/565401015 shutdown_connections 2023-12-01T22:19:51.405 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:51.403+0000 7fc80759e700 1 -- 172.21.15.73:0/565401015 wait complete. 2023-12-01T22:19:51.406 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:51.403+0000 7fc80759e700 1 Processor -- start 2023-12-01T22:19:51.406 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:51.403+0000 7fc80759e700 1 -- start start 2023-12-01T22:19:51.406 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:51.404+0000 7fc80759e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc8080f62a0 0x7fc8081064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:19:51.406 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:51.404+0000 7fc80759e700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8081069a0 con 0x7fc8080f62a0 2023-12-01T22:19:51.407 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:51.404+0000 7fc80659c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc8080f62a0 0x7fc8081064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:19:51.407 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:51.404+0000 7fc80659c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc8080f62a0 0x7fc8081064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:41550/0 (socket says 172.21.15.73:41550) 2023-12-01T22:19:51.407 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:51.404+0000 7fc80659c700 1 -- 172.21.15.73:0/613331278 learned_addr learned my addr 172.21.15.73:0/613331278 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:19:51.407 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:51.404+0000 7fc80659c700 1 -- 172.21.15.73:0/613331278 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7f8005040 con 0x7fc8080f62a0 2023-12-01T22:19:51.408 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:51.404+0000 7fc80659c700 1 --2- 172.21.15.73:0/613331278 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc8080f62a0 0x7fc8081064b0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7fc7f8013510 tx=0x7fc7f800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:19:51.408 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:51.404+0000 7fc7f6ffd700 1 -- 172.21.15.73:0/613331278 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc7f8009320 con 0x7fc8080f62a0 2023-12-01T22:19:51.408 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:51.404+0000 7fc7f6ffd700 1 -- 172.21.15.73:0/613331278 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc7f8013f00 con 0x7fc8080f62a0 2023-12-01T22:19:51.408 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:51.404+0000 7fc80759e700 1 -- 172.21.15.73:0/613331278 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc808106ba0 con 0x7fc8080f62a0 2023-12-01T22:19:51.408 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:51.404+0000 7fc7f6ffd700 1 -- 172.21.15.73:0/613331278 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc7f8014dc0 con 0x7fc8080f62a0 2023-12-01T22:19:51.408 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:51.405+0000 7fc80759e700 1 -- 172.21.15.73:0/613331278 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc808107030 con 0x7fc8080f62a0 2023-12-01T22:19:51.408 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:51.405+0000 7fc7f6ffd700 1 -- 172.21.15.73:0/613331278 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc7f80095f0 con 0x7fc8080f62a0 2023-12-01T22:19:51.408 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:51.405+0000 7fc80759e700 1 -- 172.21.15.73:0/613331278 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc808045ad0 con 0x7fc8080f62a0 2023-12-01T22:19:51.408 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:51.405+0000 7fc7f6ffd700 1 --2- 172.21.15.73:0/613331278 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fc7f002ff60 0x7fc7f0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:19:51.408 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:51.405+0000 7fc7f6ffd700 1 -- 172.21.15.73:0/613331278 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc7f8041000 con 0x7fc8080f62a0 2023-12-01T22:19:51.408 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:51.406+0000 7fc805d9b700 1 --2- 172.21.15.73:0/613331278 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fc7f002ff60 0x7fc7f0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:19:51.409 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:51.406+0000 7fc805d9b700 1 --2- 172.21.15.73:0/613331278 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fc7f002ff60 0x7fc7f0032420 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7fc7fc0096a0 tx=0x7fc7fc006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:19:51.412 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:51.411+0000 7fc7f6ffd700 1 -- 172.21.15.73:0/613331278 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc7f801c790 con 0x7fc8080f62a0 2023-12-01T22:19:51.616 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:51.615+0000 7fc80759e700 1 -- 172.21.15.73:0/613331278 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc808107ee0 con 0x7fc8080f62a0 2023-12-01T22:19:51.617 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:51.615+0000 7fc7f6ffd700 1 -- 172.21.15.73:0/613331278 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc7f8010230 con 0x7fc8080f62a0 2023-12-01T22:19:51.617 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:19:51.617 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:19:51.619 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:51.617+0000 7fc80759e700 1 -- 172.21.15.73:0/613331278 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fc7f002ff60 msgr2=0x7fc7f0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:19:51.619 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:51.617+0000 7fc80759e700 1 --2- 172.21.15.73:0/613331278 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fc7f002ff60 0x7fc7f0032420 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7fc7fc0096a0 tx=0x7fc7fc006b40).stop 2023-12-01T22:19:51.619 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:51.617+0000 7fc80759e700 1 -- 172.21.15.73:0/613331278 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc8080f62a0 msgr2=0x7fc8081064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:19:51.619 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:51.617+0000 7fc80759e700 1 --2- 172.21.15.73:0/613331278 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc8080f62a0 0x7fc8081064b0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7fc7f8013510 tx=0x7fc7f800b040).stop 2023-12-01T22:19:51.619 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:51.617+0000 7fc80759e700 1 -- 172.21.15.73:0/613331278 shutdown_connections 2023-12-01T22:19:51.619 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:51.617+0000 7fc80759e700 1 --2- 172.21.15.73:0/613331278 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fc7f002ff60 0x7fc7f0032420 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:19:51.619 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:51.617+0000 7fc80759e700 1 --2- 172.21.15.73:0/613331278 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc8080f62a0 0x7fc8081064b0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:19:51.620 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:51.617+0000 7fc80759e700 1 -- 172.21.15.73:0/613331278 >> 172.21.15.73:0/613331278 conn(0x7fc8080f1aa0 msgr2=0x7fc8080f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:19:51.620 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:51.617+0000 7fc80759e700 1 -- 172.21.15.73:0/613331278 shutdown_connections 2023-12-01T22:19:51.620 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:51.617+0000 7fc80759e700 1 -- 172.21.15.73:0/613331278 wait complete. 2023-12-01T22:19:51.620 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:19:51.776 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:19:51 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:19:50.704251+0000 mgr.smithi029.loxrfu (mgr.14164) 267 : cluster [DBG] pgmap v253: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:19:52.777 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:19:52 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:19:51.616855+0000 mon.smithi029 (mon.0) 318 : audit [DBG] from='client.? 172.21.15.73:0/613331278' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:19:53.023 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:19:53.023 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:19:53.776 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:19:53 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:19:52.704460+0000 mgr.smithi029.loxrfu (mgr.14164) 268 : cluster [DBG] pgmap v254: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:19:54.212 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:54.209+0000 7f20c1645700 1 -- 172.21.15.73:0/3636513790 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f20bc0f5410 msgr2=0x7f20bc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:19:54.213 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:54.209+0000 7f20c1645700 1 --2- 172.21.15.73:0/3636513790 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f20bc0f5410 0x7f20bc0f5830 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 rx=0x7f20b0009fb0 tx=0x7f20b0009b30).stop 2023-12-01T22:19:54.213 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:54.209+0000 7f20c1645700 1 -- 172.21.15.73:0/3636513790 shutdown_connections 2023-12-01T22:19:54.213 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:54.209+0000 7f20c1645700 1 --2- 172.21.15.73:0/3636513790 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f20bc0f5410 0x7f20bc0f5830 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:19:54.213 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:54.209+0000 7f20c1645700 1 -- 172.21.15.73:0/3636513790 >> 172.21.15.73:0/3636513790 conn(0x7f20bc0f0b70 msgr2=0x7f20bc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:19:54.213 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:54.209+0000 7f20c1645700 1 -- 172.21.15.73:0/3636513790 shutdown_connections 2023-12-01T22:19:54.213 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:54.209+0000 7f20c1645700 1 -- 172.21.15.73:0/3636513790 wait complete. 2023-12-01T22:19:54.213 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:54.210+0000 7f20c1645700 1 Processor -- start 2023-12-01T22:19:54.213 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:54.210+0000 7f20c1645700 1 -- start start 2023-12-01T22:19:54.213 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:54.210+0000 7f20c1645700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f20bc0f5410 0x7f20bc109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:19:54.213 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:54.210+0000 7f20c1645700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20bc060240 con 0x7f20bc0f5410 2023-12-01T22:19:54.213 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:54.210+0000 7f20baffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f20bc0f5410 0x7f20bc109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:19:54.213 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:54.210+0000 7f20baffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f20bc0f5410 0x7f20bc109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:41564/0 (socket says 172.21.15.73:41564) 2023-12-01T22:19:54.213 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:54.210+0000 7f20baffd700 1 -- 172.21.15.73:0/4111977696 learned_addr learned my addr 172.21.15.73:0/4111977696 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:19:54.213 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:54.210+0000 7f20baffd700 1 -- 172.21.15.73:0/4111977696 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20b0005040 con 0x7f20bc0f5410 2023-12-01T22:19:54.214 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:54.211+0000 7f20baffd700 1 --2- 172.21.15.73:0/4111977696 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f20bc0f5410 0x7f20bc109430 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7f20b0013510 tx=0x7f20b000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:19:54.214 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:54.211+0000 7f20a37fe700 1 -- 172.21.15.73:0/4111977696 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f20b0009320 con 0x7f20bc0f5410 2023-12-01T22:19:54.214 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:54.211+0000 7f20c1645700 1 -- 172.21.15.73:0/4111977696 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20bc060440 con 0x7f20bc0f5410 2023-12-01T22:19:54.214 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:54.211+0000 7f20a37fe700 1 -- 172.21.15.73:0/4111977696 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f20b0013f00 con 0x7f20bc0f5410 2023-12-01T22:19:54.214 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:54.211+0000 7f20a37fe700 1 -- 172.21.15.73:0/4111977696 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f20b0014d90 con 0x7f20bc0f5410 2023-12-01T22:19:54.214 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:54.211+0000 7f20c1645700 1 -- 172.21.15.73:0/4111977696 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20bc109ae0 con 0x7f20bc0f5410 2023-12-01T22:19:54.215 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:54.212+0000 7f20a37fe700 1 -- 172.21.15.73:0/4111977696 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f20b00095f0 con 0x7f20bc0f5410 2023-12-01T22:19:54.215 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:54.212+0000 7f20c1645700 1 -- 172.21.15.73:0/4111977696 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20bc045ad0 con 0x7f20bc0f5410 2023-12-01T22:19:54.215 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:54.212+0000 7f20a37fe700 1 --2- 172.21.15.73:0/4111977696 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f20a402ff60 0x7f20a4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:19:54.215 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:54.212+0000 7f20a37fe700 1 -- 172.21.15.73:0/4111977696 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f20b0040fe0 con 0x7f20bc0f5410 2023-12-01T22:19:54.215 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:54.212+0000 7f20ba7fc700 1 --2- 172.21.15.73:0/4111977696 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f20a402ff60 0x7f20a4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:19:54.215 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:54.213+0000 7f20ba7fc700 1 --2- 172.21.15.73:0/4111977696 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f20a402ff60 0x7f20a4032420 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f20ac0096a0 tx=0x7f20ac006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:19:54.218 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:54.217+0000 7f20a37fe700 1 -- 172.21.15.73:0/4111977696 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f20b001d7f0 con 0x7f20bc0f5410 2023-12-01T22:19:54.425 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:54.423+0000 7f20c1645700 1 -- 172.21.15.73:0/4111977696 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f20bc0fae60 con 0x7f20bc0f5410 2023-12-01T22:19:54.425 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:54.424+0000 7f20a37fe700 1 -- 172.21.15.73:0/4111977696 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f20b0010230 con 0x7f20bc0f5410 2023-12-01T22:19:54.426 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:19:54.426 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:19:54.427 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:54.425+0000 7f20c1645700 1 -- 172.21.15.73:0/4111977696 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f20a402ff60 msgr2=0x7f20a4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:19:54.427 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:54.425+0000 7f20c1645700 1 --2- 172.21.15.73:0/4111977696 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f20a402ff60 0x7f20a4032420 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f20ac0096a0 tx=0x7f20ac006b40).stop 2023-12-01T22:19:54.427 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:54.425+0000 7f20c1645700 1 -- 172.21.15.73:0/4111977696 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f20bc0f5410 msgr2=0x7f20bc109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:19:54.427 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:54.425+0000 7f20c1645700 1 --2- 172.21.15.73:0/4111977696 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f20bc0f5410 0x7f20bc109430 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7f20b0013510 tx=0x7f20b000b040).stop 2023-12-01T22:19:54.427 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:54.425+0000 7f20c1645700 1 -- 172.21.15.73:0/4111977696 shutdown_connections 2023-12-01T22:19:54.427 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:54.425+0000 7f20c1645700 1 --2- 172.21.15.73:0/4111977696 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f20a402ff60 0x7f20a4032420 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:19:54.427 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:54.425+0000 7f20c1645700 1 --2- 172.21.15.73:0/4111977696 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f20bc0f5410 0x7f20bc109430 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:19:54.427 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:54.425+0000 7f20c1645700 1 -- 172.21.15.73:0/4111977696 >> 172.21.15.73:0/4111977696 conn(0x7f20bc0f0b70 msgr2=0x7f20bc0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:19:54.428 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:54.425+0000 7f20c1645700 1 -- 172.21.15.73:0/4111977696 shutdown_connections 2023-12-01T22:19:54.428 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:54.425+0000 7f20c1645700 1 -- 172.21.15.73:0/4111977696 wait complete. 2023-12-01T22:19:54.428 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:19:54.776 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:19:54 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:19:54.425212+0000 mon.smithi029 (mon.0) 319 : audit [DBG] from='client.? 172.21.15.73:0/4111977696' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:19:55.776 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:19:55 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:19:54.704790+0000 mgr.smithi029.loxrfu (mgr.14164) 269 : cluster [DBG] pgmap v255: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:19:55.802 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:19:55.802 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:19:56.965 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:56.964+0000 7f074818a700 1 -- 172.21.15.73:0/1859410123 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0740060220 msgr2=0x7f07400f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:19:56.965 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:56.964+0000 7f074818a700 1 --2- 172.21.15.73:0/1859410123 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0740060220 0x7f07400f9150 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 rx=0x7f073c009fb0 tx=0x7f073c009b30).stop 2023-12-01T22:19:56.965 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:56.964+0000 7f074818a700 1 -- 172.21.15.73:0/1859410123 shutdown_connections 2023-12-01T22:19:56.965 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:56.964+0000 7f074818a700 1 --2- 172.21.15.73:0/1859410123 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0740060220 0x7f07400f9150 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:19:56.966 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:56.964+0000 7f074818a700 1 -- 172.21.15.73:0/1859410123 >> 172.21.15.73:0/1859410123 conn(0x7f07400f0b70 msgr2=0x7f07400f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:19:56.966 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:56.964+0000 7f074818a700 1 -- 172.21.15.73:0/1859410123 shutdown_connections 2023-12-01T22:19:56.966 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:56.964+0000 7f074818a700 1 -- 172.21.15.73:0/1859410123 wait complete. 2023-12-01T22:19:56.966 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:56.964+0000 7f074818a700 1 Processor -- start 2023-12-01T22:19:56.967 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:56.965+0000 7f074818a700 1 -- start start 2023-12-01T22:19:56.968 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:56.965+0000 7f074818a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0740060220 0x7f07401066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:19:56.968 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:56.965+0000 7f074818a700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0740106bd0 con 0x7f0740060220 2023-12-01T22:19:56.968 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:56.965+0000 7f0745f26700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0740060220 0x7f07401066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:19:56.968 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:56.965+0000 7f0745f26700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0740060220 0x7f07401066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:47116/0 (socket says 172.21.15.73:47116) 2023-12-01T22:19:56.968 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:56.965+0000 7f0745f26700 1 -- 172.21.15.73:0/1583015223 learned_addr learned my addr 172.21.15.73:0/1583015223 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:19:56.968 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:56.965+0000 7f0745f26700 1 -- 172.21.15.73:0/1583015223 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f073c005040 con 0x7f0740060220 2023-12-01T22:19:56.968 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:56.966+0000 7f0745f26700 1 --2- 172.21.15.73:0/1583015223 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0740060220 0x7f07401066e0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7f073c013510 tx=0x7f073c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:19:56.968 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:56.966+0000 7f07367fc700 1 -- 172.21.15.73:0/1583015223 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f073c009320 con 0x7f0740060220 2023-12-01T22:19:56.968 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:56.966+0000 7f074818a700 1 -- 172.21.15.73:0/1583015223 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0740106dd0 con 0x7f0740060220 2023-12-01T22:19:56.969 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:56.966+0000 7f07367fc700 1 -- 172.21.15.73:0/1583015223 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f073c013f00 con 0x7f0740060220 2023-12-01T22:19:56.969 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:56.966+0000 7f07367fc700 1 -- 172.21.15.73:0/1583015223 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f073c014dc0 con 0x7f0740060220 2023-12-01T22:19:56.970 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:56.966+0000 7f074818a700 1 -- 172.21.15.73:0/1583015223 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0740107260 con 0x7f0740060220 2023-12-01T22:19:56.970 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:56.967+0000 7f07367fc700 1 -- 172.21.15.73:0/1583015223 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f073c0095f0 con 0x7f0740060220 2023-12-01T22:19:56.970 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:56.967+0000 7f07367fc700 1 --2- 172.21.15.73:0/1583015223 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f072c02ff60 0x7f072c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:19:56.970 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:56.967+0000 7f074818a700 1 -- 172.21.15.73:0/1583015223 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0740045ad0 con 0x7f0740060220 2023-12-01T22:19:56.970 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:56.967+0000 7f07367fc700 1 -- 172.21.15.73:0/1583015223 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f073c041000 con 0x7f0740060220 2023-12-01T22:19:56.970 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:56.967+0000 7f0745725700 1 --2- 172.21.15.73:0/1583015223 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f072c02ff60 0x7f072c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:19:56.970 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:56.967+0000 7f0745725700 1 --2- 172.21.15.73:0/1583015223 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f072c02ff60 0x7f072c032420 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f07300096a0 tx=0x7f0730006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:19:56.973 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:56.972+0000 7f07367fc700 1 -- 172.21.15.73:0/1583015223 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f073c014420 con 0x7f0740060220 2023-12-01T22:19:57.181 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:57.180+0000 7f074818a700 1 -- 172.21.15.73:0/1583015223 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0740108110 con 0x7f0740060220 2023-12-01T22:19:57.182 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:57.181+0000 7f07367fc700 1 -- 172.21.15.73:0/1583015223 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f073c02b020 con 0x7f0740060220 2023-12-01T22:19:57.182 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:19:57.182 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:19:57.183 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:57.183+0000 7f074818a700 1 -- 172.21.15.73:0/1583015223 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f072c02ff60 msgr2=0x7f072c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:19:57.184 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:57.183+0000 7f074818a700 1 --2- 172.21.15.73:0/1583015223 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f072c02ff60 0x7f072c032420 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f07300096a0 tx=0x7f0730006b40).stop 2023-12-01T22:19:57.184 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:57.183+0000 7f074818a700 1 -- 172.21.15.73:0/1583015223 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0740060220 msgr2=0x7f07401066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:19:57.184 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:57.183+0000 7f074818a700 1 --2- 172.21.15.73:0/1583015223 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0740060220 0x7f07401066e0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7f073c013510 tx=0x7f073c00b040).stop 2023-12-01T22:19:57.184 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:57.183+0000 7f074818a700 1 -- 172.21.15.73:0/1583015223 shutdown_connections 2023-12-01T22:19:57.184 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:57.183+0000 7f074818a700 1 --2- 172.21.15.73:0/1583015223 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f072c02ff60 0x7f072c032420 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:19:57.184 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:57.183+0000 7f074818a700 1 --2- 172.21.15.73:0/1583015223 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0740060220 0x7f07401066e0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:19:57.184 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:57.183+0000 7f074818a700 1 -- 172.21.15.73:0/1583015223 >> 172.21.15.73:0/1583015223 conn(0x7f07400f0b70 msgr2=0x7f07400f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:19:57.184 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:57.183+0000 7f074818a700 1 -- 172.21.15.73:0/1583015223 shutdown_connections 2023-12-01T22:19:57.184 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:57.183+0000 7f074818a700 1 -- 172.21.15.73:0/1583015223 wait complete. 2023-12-01T22:19:57.184 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:19:58.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:19:57 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:19:56.705134+0000 mgr.smithi029.loxrfu (mgr.14164) 270 : cluster [DBG] pgmap v256: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:19:58.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:19:57 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:19:57.181085+0000 mon.smithi029 (mon.0) 320 : audit [DBG] from='client.? 172.21.15.73:0/1583015223' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:19:58.570 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:19:58.571 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:19:59.703 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:59.700+0000 7f8e9a6ca700 1 -- 172.21.15.73:0/4098734269 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8e940f6510 msgr2=0x7f8e940f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:19:59.703 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:59.700+0000 7f8e9a6ca700 1 --2- 172.21.15.73:0/4098734269 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8e940f6510 0x7f8e940f6930 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 rx=0x7f8e84009fb0 tx=0x7f8e84009b30).stop 2023-12-01T22:19:59.703 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:59.700+0000 7f8e9a6ca700 1 -- 172.21.15.73:0/4098734269 shutdown_connections 2023-12-01T22:19:59.703 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:59.700+0000 7f8e9a6ca700 1 --2- 172.21.15.73:0/4098734269 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8e940f6510 0x7f8e940f6930 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:19:59.703 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:59.700+0000 7f8e9a6ca700 1 -- 172.21.15.73:0/4098734269 >> 172.21.15.73:0/4098734269 conn(0x7f8e940f1c70 msgr2=0x7f8e940f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:19:59.703 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:59.700+0000 7f8e9a6ca700 1 -- 172.21.15.73:0/4098734269 shutdown_connections 2023-12-01T22:19:59.703 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:59.700+0000 7f8e9a6ca700 1 -- 172.21.15.73:0/4098734269 wait complete. 2023-12-01T22:19:59.704 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:59.701+0000 7f8e9a6ca700 1 Processor -- start 2023-12-01T22:19:59.704 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:59.701+0000 7f8e9a6ca700 1 -- start start 2023-12-01T22:19:59.704 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:59.701+0000 7f8e9a6ca700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8e940f6510 0x7f8e94106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:19:59.704 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:59.701+0000 7f8e9a6ca700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e94106c20 con 0x7f8e940f6510 2023-12-01T22:19:59.704 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:59.701+0000 7f8e93fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8e940f6510 0x7f8e94106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:19:59.704 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:59.701+0000 7f8e93fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8e940f6510 0x7f8e94106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:47134/0 (socket says 172.21.15.73:47134) 2023-12-01T22:19:59.704 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:59.701+0000 7f8e93fff700 1 -- 172.21.15.73:0/1830311325 learned_addr learned my addr 172.21.15.73:0/1830311325 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:19:59.704 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:59.702+0000 7f8e93fff700 1 -- 172.21.15.73:0/1830311325 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e84005040 con 0x7f8e940f6510 2023-12-01T22:19:59.704 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:59.702+0000 7f8e93fff700 1 --2- 172.21.15.73:0/1830311325 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8e940f6510 0x7f8e94106730 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7f8e84013510 tx=0x7f8e8400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:19:59.704 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:59.702+0000 7f8e90ff9700 1 -- 172.21.15.73:0/1830311325 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8e84009320 con 0x7f8e940f6510 2023-12-01T22:19:59.704 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:59.702+0000 7f8e90ff9700 1 -- 172.21.15.73:0/1830311325 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8e84013f00 con 0x7f8e940f6510 2023-12-01T22:19:59.704 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:59.702+0000 7f8e9a6ca700 1 -- 172.21.15.73:0/1830311325 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e94106e20 con 0x7f8e940f6510 2023-12-01T22:19:59.705 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:59.702+0000 7f8e90ff9700 1 -- 172.21.15.73:0/1830311325 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8e84014d90 con 0x7f8e940f6510 2023-12-01T22:19:59.705 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:59.702+0000 7f8e9a6ca700 1 -- 172.21.15.73:0/1830311325 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e941072b0 con 0x7f8e940f6510 2023-12-01T22:19:59.705 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:59.703+0000 7f8e90ff9700 1 -- 172.21.15.73:0/1830311325 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8e840095f0 con 0x7f8e940f6510 2023-12-01T22:19:59.706 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:59.703+0000 7f8e9a6ca700 1 -- 172.21.15.73:0/1830311325 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e94045ad0 con 0x7f8e940f6510 2023-12-01T22:19:59.706 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:59.703+0000 7f8e90ff9700 1 --2- 172.21.15.73:0/1830311325 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f8e7c02ff60 0x7f8e7c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:19:59.706 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:59.703+0000 7f8e90ff9700 1 -- 172.21.15.73:0/1830311325 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8e84040fe0 con 0x7f8e940f6510 2023-12-01T22:19:59.706 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:59.703+0000 7f8e937fe700 1 --2- 172.21.15.73:0/1830311325 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f8e7c02ff60 0x7f8e7c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:19:59.706 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:59.703+0000 7f8e937fe700 1 --2- 172.21.15.73:0/1830311325 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f8e7c02ff60 0x7f8e7c032420 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f8e880096a0 tx=0x7f8e88006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:19:59.708 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:59.708+0000 7f8e90ff9700 1 -- 172.21.15.73:0/1830311325 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8e8401d7f0 con 0x7f8e940f6510 2023-12-01T22:19:59.913 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:59.913+0000 7f8e9a6ca700 1 -- 172.21.15.73:0/1830311325 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8e940fbf60 con 0x7f8e940f6510 2023-12-01T22:19:59.914 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:59.913+0000 7f8e90ff9700 1 -- 172.21.15.73:0/1830311325 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8e84010230 con 0x7f8e940f6510 2023-12-01T22:19:59.914 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:19:59.914 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:19:59.915 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:59.915+0000 7f8e9a6ca700 1 -- 172.21.15.73:0/1830311325 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f8e7c02ff60 msgr2=0x7f8e7c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:19:59.915 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:59.915+0000 7f8e9a6ca700 1 --2- 172.21.15.73:0/1830311325 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f8e7c02ff60 0x7f8e7c032420 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f8e880096a0 tx=0x7f8e88006b40).stop 2023-12-01T22:19:59.916 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:59.915+0000 7f8e9a6ca700 1 -- 172.21.15.73:0/1830311325 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8e940f6510 msgr2=0x7f8e94106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:19:59.916 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:59.915+0000 7f8e9a6ca700 1 --2- 172.21.15.73:0/1830311325 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8e940f6510 0x7f8e94106730 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7f8e84013510 tx=0x7f8e8400b040).stop 2023-12-01T22:19:59.916 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:59.915+0000 7f8e9a6ca700 1 -- 172.21.15.73:0/1830311325 shutdown_connections 2023-12-01T22:19:59.916 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:59.915+0000 7f8e9a6ca700 1 --2- 172.21.15.73:0/1830311325 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f8e7c02ff60 0x7f8e7c032420 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:19:59.916 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:59.915+0000 7f8e9a6ca700 1 --2- 172.21.15.73:0/1830311325 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8e940f6510 0x7f8e94106730 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:19:59.916 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:59.915+0000 7f8e9a6ca700 1 -- 172.21.15.73:0/1830311325 >> 172.21.15.73:0/1830311325 conn(0x7f8e940f1c70 msgr2=0x7f8e940f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:19:59.916 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:59.915+0000 7f8e9a6ca700 1 -- 172.21.15.73:0/1830311325 shutdown_connections 2023-12-01T22:19:59.916 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:19:59.915+0000 7f8e9a6ca700 1 -- 172.21.15.73:0/1830311325 wait complete. 2023-12-01T22:19:59.916 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:20:00.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:19:59 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:19:58.705374+0000 mgr.smithi029.loxrfu (mgr.14164) 271 : cluster [DBG] pgmap v257: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:20:01.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:20:00 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:19:59.913655+0000 mon.smithi029 (mon.0) 321 : audit [DBG] from='client.? 172.21.15.73:0/1830311325' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:20:01.253 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:20:01.253 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:20:02.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:20:01 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:20:00.705710+0000 mgr.smithi029.loxrfu (mgr.14164) 272 : cluster [DBG] pgmap v258: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:20:02.451 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:02.450+0000 7fd87a597700 1 -- 172.21.15.73:0/3309824486 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd8740f6a10 msgr2=0x7fd8740f6e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:20:02.451 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:02.450+0000 7fd87a597700 1 --2- 172.21.15.73:0/3309824486 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd8740f6a10 0x7fd8740f6e30 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 rx=0x7fd868009fb0 tx=0x7fd868009b30).stop 2023-12-01T22:20:02.451 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:02.450+0000 7fd87a597700 1 -- 172.21.15.73:0/3309824486 shutdown_connections 2023-12-01T22:20:02.452 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:02.450+0000 7fd87a597700 1 --2- 172.21.15.73:0/3309824486 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd8740f6a10 0x7fd8740f6e30 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:20:02.452 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:02.450+0000 7fd87a597700 1 -- 172.21.15.73:0/3309824486 >> 172.21.15.73:0/3309824486 conn(0x7fd8740f02e0 msgr2=0x7fd8740f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:20:02.452 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:02.450+0000 7fd87a597700 1 -- 172.21.15.73:0/3309824486 shutdown_connections 2023-12-01T22:20:02.452 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:02.450+0000 7fd87a597700 1 -- 172.21.15.73:0/3309824486 wait complete. 2023-12-01T22:20:02.452 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:02.450+0000 7fd87a597700 1 Processor -- start 2023-12-01T22:20:02.452 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:02.451+0000 7fd87a597700 1 -- start start 2023-12-01T22:20:02.452 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:02.451+0000 7fd87a597700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd8740f6a10 0x7fd8741093f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:20:02.452 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:02.451+0000 7fd87a597700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8740f5330 con 0x7fd8740f6a10 2023-12-01T22:20:02.454 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:02.451+0000 7fd873fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd8740f6a10 0x7fd8741093f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:20:02.454 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:02.451+0000 7fd873fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd8740f6a10 0x7fd8741093f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:47154/0 (socket says 172.21.15.73:47154) 2023-12-01T22:20:02.454 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:02.451+0000 7fd873fff700 1 -- 172.21.15.73:0/4050635795 learned_addr learned my addr 172.21.15.73:0/4050635795 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:20:02.454 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:02.451+0000 7fd873fff700 1 -- 172.21.15.73:0/4050635795 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd868005040 con 0x7fd8740f6a10 2023-12-01T22:20:02.454 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:02.451+0000 7fd873fff700 1 --2- 172.21.15.73:0/4050635795 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd8740f6a10 0x7fd8741093f0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7fd868013510 tx=0x7fd86800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:20:02.454 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:02.452+0000 7fd870ff9700 1 -- 172.21.15.73:0/4050635795 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd868009320 con 0x7fd8740f6a10 2023-12-01T22:20:02.454 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:02.452+0000 7fd87a597700 1 -- 172.21.15.73:0/4050635795 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8740f5530 con 0x7fd8740f6a10 2023-12-01T22:20:02.454 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:02.452+0000 7fd870ff9700 1 -- 172.21.15.73:0/4050635795 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd868013f00 con 0x7fd8740f6a10 2023-12-01T22:20:02.454 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:02.452+0000 7fd870ff9700 1 -- 172.21.15.73:0/4050635795 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd868014d90 con 0x7fd8740f6a10 2023-12-01T22:20:02.454 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:02.452+0000 7fd87a597700 1 -- 172.21.15.73:0/4050635795 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd874109aa0 con 0x7fd8740f6a10 2023-12-01T22:20:02.454 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:02.452+0000 7fd870ff9700 1 -- 172.21.15.73:0/4050635795 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd8680263a0 con 0x7fd8740f6a10 2023-12-01T22:20:02.454 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:02.453+0000 7fd87a597700 1 -- 172.21.15.73:0/4050635795 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd874059100 con 0x7fd8740f6a10 2023-12-01T22:20:02.454 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:02.453+0000 7fd870ff9700 1 --2- 172.21.15.73:0/4050635795 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fd85c02ff60 0x7fd85c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:20:02.455 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:02.453+0000 7fd870ff9700 1 -- 172.21.15.73:0/4050635795 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd8680413c0 con 0x7fd8740f6a10 2023-12-01T22:20:02.455 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:02.453+0000 7fd8737fe700 1 --2- 172.21.15.73:0/4050635795 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fd85c02ff60 0x7fd85c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:20:02.455 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:02.453+0000 7fd8737fe700 1 --2- 172.21.15.73:0/4050635795 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fd85c02ff60 0x7fd85c032420 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7fd8640096a0 tx=0x7fd864006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:20:02.458 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:02.458+0000 7fd870ff9700 1 -- 172.21.15.73:0/4050635795 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd86801d3d0 con 0x7fd8740f6a10 2023-12-01T22:20:02.664 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:02.663+0000 7fd87a597700 1 -- 172.21.15.73:0/4050635795 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd874045ad0 con 0x7fd8740f6a10 2023-12-01T22:20:02.665 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:02.664+0000 7fd870ff9700 1 -- 172.21.15.73:0/4050635795 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd86801cb10 con 0x7fd8740f6a10 2023-12-01T22:20:02.665 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:20:02.665 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:20:02.667 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:02.666+0000 7fd87a597700 1 -- 172.21.15.73:0/4050635795 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fd85c02ff60 msgr2=0x7fd85c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:20:02.667 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:02.666+0000 7fd87a597700 1 --2- 172.21.15.73:0/4050635795 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fd85c02ff60 0x7fd85c032420 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7fd8640096a0 tx=0x7fd864006b40).stop 2023-12-01T22:20:02.667 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:02.666+0000 7fd87a597700 1 -- 172.21.15.73:0/4050635795 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd8740f6a10 msgr2=0x7fd8741093f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:20:02.667 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:02.666+0000 7fd87a597700 1 --2- 172.21.15.73:0/4050635795 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd8740f6a10 0x7fd8741093f0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7fd868013510 tx=0x7fd86800b040).stop 2023-12-01T22:20:02.667 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:02.666+0000 7fd87a597700 1 -- 172.21.15.73:0/4050635795 shutdown_connections 2023-12-01T22:20:02.667 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:02.666+0000 7fd87a597700 1 --2- 172.21.15.73:0/4050635795 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fd85c02ff60 0x7fd85c032420 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:20:02.668 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:02.666+0000 7fd87a597700 1 --2- 172.21.15.73:0/4050635795 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd8740f6a10 0x7fd8741093f0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:20:02.668 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:02.666+0000 7fd87a597700 1 -- 172.21.15.73:0/4050635795 >> 172.21.15.73:0/4050635795 conn(0x7fd8740f02e0 msgr2=0x7fd8740f0da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:20:02.668 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:02.666+0000 7fd87a597700 1 -- 172.21.15.73:0/4050635795 shutdown_connections 2023-12-01T22:20:02.668 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:02.666+0000 7fd87a597700 1 -- 172.21.15.73:0/4050635795 wait complete. 2023-12-01T22:20:02.668 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:20:03.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:20:02 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:20:02.664517+0000 mon.smithi029 (mon.0) 322 : audit [DBG] from='client.? 172.21.15.73:0/4050635795' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:20:04.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:20:03 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:20:02.706045+0000 mgr.smithi029.loxrfu (mgr.14164) 273 : cluster [DBG] pgmap v259: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:20:04.064 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:20:04.064 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:20:05.274 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:05.272+0000 7fb5fd79a700 1 -- 172.21.15.73:0/329394622 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5f8060280 msgr2=0x7fb5f80f8ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:20:05.274 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:05.272+0000 7fb5fd79a700 1 --2- 172.21.15.73:0/329394622 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5f8060280 0x7fb5f80f8ea0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 rx=0x7fb5e8009fb0 tx=0x7fb5e8009b30).stop 2023-12-01T22:20:05.274 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:05.272+0000 7fb5fd79a700 1 -- 172.21.15.73:0/329394622 shutdown_connections 2023-12-01T22:20:05.274 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:05.272+0000 7fb5fd79a700 1 --2- 172.21.15.73:0/329394622 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5f8060280 0x7fb5f80f8ea0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:20:05.274 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:05.272+0000 7fb5fd79a700 1 -- 172.21.15.73:0/329394622 >> 172.21.15.73:0/329394622 conn(0x7fb5f80f0920 msgr2=0x7fb5f80f2d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:20:05.274 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:05.272+0000 7fb5fd79a700 1 -- 172.21.15.73:0/329394622 shutdown_connections 2023-12-01T22:20:05.274 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:05.272+0000 7fb5fd79a700 1 -- 172.21.15.73:0/329394622 wait complete. 2023-12-01T22:20:05.274 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:05.272+0000 7fb5fd79a700 1 Processor -- start 2023-12-01T22:20:05.274 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:05.272+0000 7fb5fd79a700 1 -- start start 2023-12-01T22:20:05.274 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:05.272+0000 7fb5fd79a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5f8060280 0x7fb5f8106500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:20:05.275 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:05.272+0000 7fb5fd79a700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5f81069f0 con 0x7fb5f8060280 2023-12-01T22:20:05.275 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:05.273+0000 7fb5f6ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5f8060280 0x7fb5f8106500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:20:05.275 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:05.273+0000 7fb5f6ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5f8060280 0x7fb5f8106500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:51608/0 (socket says 172.21.15.73:51608) 2023-12-01T22:20:05.275 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:05.273+0000 7fb5f6ffd700 1 -- 172.21.15.73:0/4143715643 learned_addr learned my addr 172.21.15.73:0/4143715643 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:20:05.275 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:05.273+0000 7fb5f6ffd700 1 -- 172.21.15.73:0/4143715643 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5e8005040 con 0x7fb5f8060280 2023-12-01T22:20:05.275 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:05.273+0000 7fb5f6ffd700 1 --2- 172.21.15.73:0/4143715643 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5f8060280 0x7fb5f8106500 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7fb5e8013510 tx=0x7fb5e800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:20:05.275 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:05.273+0000 7fb5df7fe700 1 -- 172.21.15.73:0/4143715643 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb5e8009320 con 0x7fb5f8060280 2023-12-01T22:20:05.276 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:05.273+0000 7fb5fd79a700 1 -- 172.21.15.73:0/4143715643 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5f8106bf0 con 0x7fb5f8060280 2023-12-01T22:20:05.276 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:05.273+0000 7fb5df7fe700 1 -- 172.21.15.73:0/4143715643 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb5e8013f00 con 0x7fb5f8060280 2023-12-01T22:20:05.276 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:05.273+0000 7fb5df7fe700 1 -- 172.21.15.73:0/4143715643 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb5e8014dc0 con 0x7fb5f8060280 2023-12-01T22:20:05.276 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:05.273+0000 7fb5fd79a700 1 -- 172.21.15.73:0/4143715643 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5f8107080 con 0x7fb5f8060280 2023-12-01T22:20:05.276 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:05.274+0000 7fb5df7fe700 1 -- 172.21.15.73:0/4143715643 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb5e8014420 con 0x7fb5f8060280 2023-12-01T22:20:05.276 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:05.274+0000 7fb5fd79a700 1 -- 172.21.15.73:0/4143715643 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5f8045ad0 con 0x7fb5f8060280 2023-12-01T22:20:05.276 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:05.274+0000 7fb5df7fe700 1 --2- 172.21.15.73:0/4143715643 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fb5e002ff60 0x7fb5e0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:20:05.276 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:05.274+0000 7fb5df7fe700 1 -- 172.21.15.73:0/4143715643 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb5e80410d0 con 0x7fb5f8060280 2023-12-01T22:20:05.276 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:05.275+0000 7fb5f67fc700 1 --2- 172.21.15.73:0/4143715643 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fb5e002ff60 0x7fb5e0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:20:05.276 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:05.275+0000 7fb5f67fc700 1 --2- 172.21.15.73:0/4143715643 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fb5e002ff60 0x7fb5e0032420 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7fb5ec0096a0 tx=0x7fb5ec006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:20:05.280 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:05.279+0000 7fb5df7fe700 1 -- 172.21.15.73:0/4143715643 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb5e80095f0 con 0x7fb5f8060280 2023-12-01T22:20:05.486 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:05.485+0000 7fb5fd79a700 1 -- 172.21.15.73:0/4143715643 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb5f8107e80 con 0x7fb5f8060280 2023-12-01T22:20:05.487 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:05.486+0000 7fb5df7fe700 1 -- 172.21.15.73:0/4143715643 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb5e801cb10 con 0x7fb5f8060280 2023-12-01T22:20:05.487 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:20:05.487 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:20:05.489 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:05.488+0000 7fb5fd79a700 1 -- 172.21.15.73:0/4143715643 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fb5e002ff60 msgr2=0x7fb5e0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:20:05.489 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:05.488+0000 7fb5fd79a700 1 --2- 172.21.15.73:0/4143715643 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fb5e002ff60 0x7fb5e0032420 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7fb5ec0096a0 tx=0x7fb5ec006b40).stop 2023-12-01T22:20:05.489 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:05.488+0000 7fb5fd79a700 1 -- 172.21.15.73:0/4143715643 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5f8060280 msgr2=0x7fb5f8106500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:20:05.489 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:05.488+0000 7fb5fd79a700 1 --2- 172.21.15.73:0/4143715643 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5f8060280 0x7fb5f8106500 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7fb5e8013510 tx=0x7fb5e800b040).stop 2023-12-01T22:20:05.489 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:05.488+0000 7fb5fd79a700 1 -- 172.21.15.73:0/4143715643 shutdown_connections 2023-12-01T22:20:05.489 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:05.488+0000 7fb5fd79a700 1 --2- 172.21.15.73:0/4143715643 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7fb5e002ff60 0x7fb5e0032420 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:20:05.489 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:05.488+0000 7fb5fd79a700 1 --2- 172.21.15.73:0/4143715643 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5f8060280 0x7fb5f8106500 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:20:05.489 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:05.488+0000 7fb5fd79a700 1 -- 172.21.15.73:0/4143715643 >> 172.21.15.73:0/4143715643 conn(0x7fb5f80f0920 msgr2=0x7fb5f80f13f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:20:05.489 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:05.488+0000 7fb5fd79a700 1 -- 172.21.15.73:0/4143715643 shutdown_connections 2023-12-01T22:20:05.489 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:05.488+0000 7fb5fd79a700 1 -- 172.21.15.73:0/4143715643 wait complete. 2023-12-01T22:20:05.489 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:20:06.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:20:05 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:20:04.706294+0000 mgr.smithi029.loxrfu (mgr.14164) 274 : cluster [DBG] pgmap v260: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:20:06.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:20:05 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:20:05.486469+0000 mon.smithi029 (mon.0) 323 : audit [DBG] from='client.? 172.21.15.73:0/4143715643' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:20:06.881 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:20:06.882 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:20:08.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:20:07 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:20:06.706608+0000 mgr.smithi029.loxrfu (mgr.14164) 275 : cluster [DBG] pgmap v261: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:20:08.042 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:08.039+0000 7f87c2eb0700 1 -- 172.21.15.73:0/1819683007 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87bc060120 msgr2=0x7f87bc0f8fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:20:08.042 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:08.039+0000 7f87c2eb0700 1 --2- 172.21.15.73:0/1819683007 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87bc060120 0x7f87bc0f8fe0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 rx=0x7f87b8009fb0 tx=0x7f87b8009b30).stop 2023-12-01T22:20:08.042 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:08.039+0000 7f87c2eb0700 1 -- 172.21.15.73:0/1819683007 shutdown_connections 2023-12-01T22:20:08.042 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:08.039+0000 7f87c2eb0700 1 --2- 172.21.15.73:0/1819683007 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87bc060120 0x7f87bc0f8fe0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:20:08.042 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:08.039+0000 7f87c2eb0700 1 -- 172.21.15.73:0/1819683007 >> 172.21.15.73:0/1819683007 conn(0x7f87bc0f09a0 msgr2=0x7f87bc0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:20:08.042 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:08.039+0000 7f87c2eb0700 1 -- 172.21.15.73:0/1819683007 shutdown_connections 2023-12-01T22:20:08.042 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:08.039+0000 7f87c2eb0700 1 -- 172.21.15.73:0/1819683007 wait complete. 2023-12-01T22:20:08.043 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:08.039+0000 7f87c2eb0700 1 Processor -- start 2023-12-01T22:20:08.043 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:08.039+0000 7f87c2eb0700 1 -- start start 2023-12-01T22:20:08.043 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:08.039+0000 7f87c2eb0700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87bc060120 0x7f87bc1064d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:20:08.043 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:08.039+0000 7f87c2eb0700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87bc1069c0 con 0x7f87bc060120 2023-12-01T22:20:08.043 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:08.040+0000 7f87c1eae700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87bc060120 0x7f87bc1064d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:20:08.043 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:08.040+0000 7f87c1eae700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87bc060120 0x7f87bc1064d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:51620/0 (socket says 172.21.15.73:51620) 2023-12-01T22:20:08.043 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:08.040+0000 7f87c1eae700 1 -- 172.21.15.73:0/994246431 learned_addr learned my addr 172.21.15.73:0/994246431 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:20:08.043 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:08.040+0000 7f87c1eae700 1 -- 172.21.15.73:0/994246431 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87b8005040 con 0x7f87bc060120 2023-12-01T22:20:08.043 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:08.040+0000 7f87c1eae700 1 --2- 172.21.15.73:0/994246431 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87bc060120 0x7f87bc1064d0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7f87b8013510 tx=0x7f87b800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:20:08.043 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:08.040+0000 7f87b27fc700 1 -- 172.21.15.73:0/994246431 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f87b8009320 con 0x7f87bc060120 2023-12-01T22:20:08.043 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:08.040+0000 7f87c2eb0700 1 -- 172.21.15.73:0/994246431 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87bc106bc0 con 0x7f87bc060120 2023-12-01T22:20:08.043 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:08.040+0000 7f87b27fc700 1 -- 172.21.15.73:0/994246431 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f87b8013f00 con 0x7f87bc060120 2023-12-01T22:20:08.043 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:08.040+0000 7f87b27fc700 1 -- 172.21.15.73:0/994246431 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f87b8014d90 con 0x7f87bc060120 2023-12-01T22:20:08.043 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:08.040+0000 7f87c2eb0700 1 -- 172.21.15.73:0/994246431 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87bc107050 con 0x7f87bc060120 2023-12-01T22:20:08.044 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:08.041+0000 7f87b27fc700 1 -- 172.21.15.73:0/994246431 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f87b8014420 con 0x7f87bc060120 2023-12-01T22:20:08.044 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:08.041+0000 7f87c2eb0700 1 -- 172.21.15.73:0/994246431 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87bc045ad0 con 0x7f87bc060120 2023-12-01T22:20:08.044 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:08.041+0000 7f87b27fc700 1 --2- 172.21.15.73:0/994246431 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f87a802ffb0 0x7f87a8032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:20:08.044 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:08.041+0000 7f87b27fc700 1 -- 172.21.15.73:0/994246431 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f87b80410c0 con 0x7f87bc060120 2023-12-01T22:20:08.045 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:08.042+0000 7f87c16ad700 1 --2- 172.21.15.73:0/994246431 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f87a802ffb0 0x7f87a8032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:20:08.045 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:08.042+0000 7f87c16ad700 1 --2- 172.21.15.73:0/994246431 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f87a802ffb0 0x7f87a8032470 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f87ac0096a0 tx=0x7f87ac006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:20:08.048 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:08.046+0000 7f87b27fc700 1 -- 172.21.15.73:0/994246431 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f87b80095f0 con 0x7f87bc060120 2023-12-01T22:20:08.256 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:08.254+0000 7f87c2eb0700 1 -- 172.21.15.73:0/994246431 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f87bc107f00 con 0x7f87bc060120 2023-12-01T22:20:08.256 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:08.255+0000 7f87b27fc700 1 -- 172.21.15.73:0/994246431 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f87b801db70 con 0x7f87bc060120 2023-12-01T22:20:08.257 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:20:08.257 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:20:08.258 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:08.256+0000 7f87c2eb0700 1 -- 172.21.15.73:0/994246431 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f87a802ffb0 msgr2=0x7f87a8032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:20:08.258 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:08.256+0000 7f87c2eb0700 1 --2- 172.21.15.73:0/994246431 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f87a802ffb0 0x7f87a8032470 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f87ac0096a0 tx=0x7f87ac006b40).stop 2023-12-01T22:20:08.258 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:08.256+0000 7f87c2eb0700 1 -- 172.21.15.73:0/994246431 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87bc060120 msgr2=0x7f87bc1064d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:20:08.258 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:08.256+0000 7f87c2eb0700 1 --2- 172.21.15.73:0/994246431 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87bc060120 0x7f87bc1064d0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7f87b8013510 tx=0x7f87b800b040).stop 2023-12-01T22:20:08.258 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:08.256+0000 7f87c2eb0700 1 -- 172.21.15.73:0/994246431 shutdown_connections 2023-12-01T22:20:08.258 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:08.256+0000 7f87c2eb0700 1 --2- 172.21.15.73:0/994246431 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f87a802ffb0 0x7f87a8032470 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:20:08.258 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:08.256+0000 7f87c2eb0700 1 --2- 172.21.15.73:0/994246431 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87bc060120 0x7f87bc1064d0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:20:08.259 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:08.256+0000 7f87c2eb0700 1 -- 172.21.15.73:0/994246431 >> 172.21.15.73:0/994246431 conn(0x7f87bc0f09a0 msgr2=0x7f87bc0f1450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:20:08.259 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:08.256+0000 7f87c2eb0700 1 -- 172.21.15.73:0/994246431 shutdown_connections 2023-12-01T22:20:08.259 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:08.256+0000 7f87c2eb0700 1 -- 172.21.15.73:0/994246431 wait complete. 2023-12-01T22:20:08.259 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:20:09.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:20:08 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:20:08.256311+0000 mon.smithi029 (mon.0) 324 : audit [DBG] from='client.? 172.21.15.73:0/994246431' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:20:09.584 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:20:09.584 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:20:10.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:20:09 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:20:08.706909+0000 mgr.smithi029.loxrfu (mgr.14164) 276 : cluster [DBG] pgmap v262: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:20:10.728 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:10.725+0000 7f2583515700 1 -- 172.21.15.73:0/2137608517 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f257c060280 msgr2=0x7f257c0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:20:10.729 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:10.725+0000 7f2583515700 1 --2- 172.21.15.73:0/2137608517 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f257c060280 0x7f257c0f90f0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 rx=0x7f256c009fb0 tx=0x7f256c009b30).stop 2023-12-01T22:20:10.729 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:10.725+0000 7f2583515700 1 -- 172.21.15.73:0/2137608517 shutdown_connections 2023-12-01T22:20:10.729 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:10.725+0000 7f2583515700 1 --2- 172.21.15.73:0/2137608517 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f257c060280 0x7f257c0f90f0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:20:10.729 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:10.725+0000 7f2583515700 1 -- 172.21.15.73:0/2137608517 >> 172.21.15.73:0/2137608517 conn(0x7f257c0f0b70 msgr2=0x7f257c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:20:10.729 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:10.725+0000 7f2583515700 1 -- 172.21.15.73:0/2137608517 shutdown_connections 2023-12-01T22:20:10.729 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:10.725+0000 7f2583515700 1 -- 172.21.15.73:0/2137608517 wait complete. 2023-12-01T22:20:10.729 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:10.726+0000 7f2583515700 1 Processor -- start 2023-12-01T22:20:10.729 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:10.726+0000 7f2583515700 1 -- start start 2023-12-01T22:20:10.729 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:10.726+0000 7f2583515700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f257c060280 0x7f257c0f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:20:10.729 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:10.726+0000 7f2583515700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f257c0f4ff0 con 0x7f257c060280 2023-12-01T22:20:10.730 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:10.726+0000 7f25812b1700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f257c060280 0x7f257c0f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:20:10.730 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:10.726+0000 7f25812b1700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f257c060280 0x7f257c0f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:51638/0 (socket says 172.21.15.73:51638) 2023-12-01T22:20:10.730 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:10.726+0000 7f25812b1700 1 -- 172.21.15.73:0/4134754081 learned_addr learned my addr 172.21.15.73:0/4134754081 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:20:10.730 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:10.727+0000 7f25812b1700 1 -- 172.21.15.73:0/4134754081 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f256c005040 con 0x7f257c060280 2023-12-01T22:20:10.730 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:10.727+0000 7f25812b1700 1 --2- 172.21.15.73:0/4134754081 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f257c060280 0x7f257c0f6b70 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7f256c013510 tx=0x7f256c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:20:10.730 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:10.727+0000 7f2571ffb700 1 -- 172.21.15.73:0/4134754081 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f256c009320 con 0x7f257c060280 2023-12-01T22:20:10.730 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:10.727+0000 7f2583515700 1 -- 172.21.15.73:0/4134754081 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f257c0f5190 con 0x7f257c060280 2023-12-01T22:20:10.730 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:10.727+0000 7f2571ffb700 1 -- 172.21.15.73:0/4134754081 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f256c013f00 con 0x7f257c060280 2023-12-01T22:20:10.730 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:10.727+0000 7f2571ffb700 1 -- 172.21.15.73:0/4134754081 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f256c014dc0 con 0x7f257c060280 2023-12-01T22:20:10.730 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:10.727+0000 7f2583515700 1 -- 172.21.15.73:0/4134754081 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f257c0f5620 con 0x7f257c060280 2023-12-01T22:20:10.731 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:10.728+0000 7f2583515700 1 -- 172.21.15.73:0/4134754081 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f257c059100 con 0x7f257c060280 2023-12-01T22:20:10.731 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:10.728+0000 7f2571ffb700 1 -- 172.21.15.73:0/4134754081 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f256c0095f0 con 0x7f257c060280 2023-12-01T22:20:10.731 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:10.728+0000 7f2571ffb700 1 --2- 172.21.15.73:0/4134754081 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f256802ff60 0x7f2568032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:20:10.731 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:10.728+0000 7f2571ffb700 1 -- 172.21.15.73:0/4134754081 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f256c0410e0 con 0x7f257c060280 2023-12-01T22:20:10.731 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:10.728+0000 7f2580ab0700 1 --2- 172.21.15.73:0/4134754081 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f256802ff60 0x7f2568032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:20:10.732 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:10.729+0000 7f2580ab0700 1 --2- 172.21.15.73:0/4134754081 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f256802ff60 0x7f2568032420 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f25780096a0 tx=0x7f2578006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:20:10.734 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:10.733+0000 7f2571ffb700 1 -- 172.21.15.73:0/4134754081 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f256c014420 con 0x7f257c060280 2023-12-01T22:20:10.940 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:10.938+0000 7f2583515700 1 -- 172.21.15.73:0/4134754081 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f257c045ad0 con 0x7f257c060280 2023-12-01T22:20:10.940 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:10.939+0000 7f2571ffb700 1 -- 172.21.15.73:0/4134754081 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f256c01c920 con 0x7f257c060280 2023-12-01T22:20:10.941 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:20:10.941 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:20:10.942 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:10.940+0000 7f2583515700 1 -- 172.21.15.73:0/4134754081 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f256802ff60 msgr2=0x7f2568032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:20:10.942 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:10.940+0000 7f2583515700 1 --2- 172.21.15.73:0/4134754081 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f256802ff60 0x7f2568032420 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f25780096a0 tx=0x7f2578006b40).stop 2023-12-01T22:20:10.942 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:10.940+0000 7f2583515700 1 -- 172.21.15.73:0/4134754081 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f257c060280 msgr2=0x7f257c0f6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:20:10.943 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:10.940+0000 7f2583515700 1 --2- 172.21.15.73:0/4134754081 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f257c060280 0x7f257c0f6b70 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7f256c013510 tx=0x7f256c00b040).stop 2023-12-01T22:20:10.943 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:10.940+0000 7f2583515700 1 -- 172.21.15.73:0/4134754081 shutdown_connections 2023-12-01T22:20:10.943 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:10.940+0000 7f2583515700 1 --2- 172.21.15.73:0/4134754081 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f256802ff60 0x7f2568032420 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:20:10.943 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:10.940+0000 7f2583515700 1 --2- 172.21.15.73:0/4134754081 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f257c060280 0x7f257c0f6b70 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:20:10.943 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:10.940+0000 7f2583515700 1 -- 172.21.15.73:0/4134754081 >> 172.21.15.73:0/4134754081 conn(0x7f257c0f0b70 msgr2=0x7f257c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:20:10.943 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:10.940+0000 7f2583515700 1 -- 172.21.15.73:0/4134754081 shutdown_connections 2023-12-01T22:20:10.943 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:10.941+0000 7f2583515700 1 -- 172.21.15.73:0/4134754081 wait complete. 2023-12-01T22:20:10.943 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:20:11.776 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:20:11 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:20:10.707267+0000 mgr.smithi029.loxrfu (mgr.14164) 277 : cluster [DBG] pgmap v263: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:20:11.777 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:20:11 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:20:10.940300+0000 mon.smithi029 (mon.0) 325 : audit [DBG] from='client.? 172.21.15.73:0/4134754081' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:20:12.321 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-01T22:20:12.321 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-01T22:20:13.511 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:13.508+0000 7f7448708700 1 -- 172.21.15.73:0/1676509149 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74400f5410 msgr2=0x7f74400f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:20:13.511 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:13.508+0000 7f7448708700 1 --2- 172.21.15.73:0/1676509149 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74400f5410 0x7f74400f5830 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 rx=0x7f7430009fb0 tx=0x7f7430009b30).stop 2023-12-01T22:20:13.512 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:13.508+0000 7f7448708700 1 -- 172.21.15.73:0/1676509149 shutdown_connections 2023-12-01T22:20:13.512 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:13.508+0000 7f7448708700 1 --2- 172.21.15.73:0/1676509149 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74400f5410 0x7f74400f5830 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:20:13.512 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:13.508+0000 7f7448708700 1 -- 172.21.15.73:0/1676509149 >> 172.21.15.73:0/1676509149 conn(0x7f74400f0b70 msgr2=0x7f74400f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:20:13.512 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:13.508+0000 7f7448708700 1 -- 172.21.15.73:0/1676509149 shutdown_connections 2023-12-01T22:20:13.512 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:13.508+0000 7f7448708700 1 -- 172.21.15.73:0/1676509149 wait complete. 2023-12-01T22:20:13.512 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:13.509+0000 7f7448708700 1 Processor -- start 2023-12-01T22:20:13.512 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:13.509+0000 7f7448708700 1 -- start start 2023-12-01T22:20:13.512 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:13.509+0000 7f7448708700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74400f5410 0x7f74401066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:20:13.512 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:13.509+0000 7f7448708700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7440106bc0 con 0x7f74400f5410 2023-12-01T22:20:13.512 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:13.509+0000 7f74464a4700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74400f5410 0x7f74401066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:20:13.512 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:13.509+0000 7f74464a4700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74400f5410 0x7f74401066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.73:51658/0 (socket says 172.21.15.73:51658) 2023-12-01T22:20:13.512 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:13.509+0000 7f74464a4700 1 -- 172.21.15.73:0/246083282 learned_addr learned my addr 172.21.15.73:0/246083282 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-12-01T22:20:13.512 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:13.509+0000 7f74464a4700 1 -- 172.21.15.73:0/246083282 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7430005040 con 0x7f74400f5410 2023-12-01T22:20:13.512 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:13.509+0000 7f74464a4700 1 --2- 172.21.15.73:0/246083282 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74400f5410 0x7f74401066d0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 rx=0x7f7430013510 tx=0x7f743000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:20:13.513 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:13.510+0000 7f7436ffd700 1 -- 172.21.15.73:0/246083282 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7430009320 con 0x7f74400f5410 2023-12-01T22:20:13.513 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:13.510+0000 7f7448708700 1 -- 172.21.15.73:0/246083282 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7440106dc0 con 0x7f74400f5410 2023-12-01T22:20:13.513 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:13.510+0000 7f7436ffd700 1 -- 172.21.15.73:0/246083282 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7430013f00 con 0x7f74400f5410 2023-12-01T22:20:13.513 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:13.510+0000 7f7436ffd700 1 -- 172.21.15.73:0/246083282 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7430014d90 con 0x7f74400f5410 2023-12-01T22:20:13.513 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:13.510+0000 7f7448708700 1 -- 172.21.15.73:0/246083282 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7440107250 con 0x7f74400f5410 2023-12-01T22:20:13.514 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:13.510+0000 7f7436ffd700 1 -- 172.21.15.73:0/246083282 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f74300095f0 con 0x7f74400f5410 2023-12-01T22:20:13.514 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:13.510+0000 7f7448708700 1 -- 172.21.15.73:0/246083282 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7440045ad0 con 0x7f74400f5410 2023-12-01T22:20:13.514 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:13.511+0000 7f7436ffd700 1 --2- 172.21.15.73:0/246083282 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f742c02ff60 0x7f742c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-01T22:20:13.514 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:13.511+0000 7f7436ffd700 1 -- 172.21.15.73:0/246083282 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f74300410c0 con 0x7f74400f5410 2023-12-01T22:20:13.514 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:13.511+0000 7f7445ca3700 1 --2- 172.21.15.73:0/246083282 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f742c02ff60 0x7f742c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-01T22:20:13.514 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:13.511+0000 7f7445ca3700 1 --2- 172.21.15.73:0/246083282 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f742c02ff60 0x7f742c032420 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7f743c0096a0 tx=0x7f743c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T22:20:13.517 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:13.515+0000 7f7436ffd700 1 -- 172.21.15.73:0/246083282 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7430014420 con 0x7f74400f5410 2023-12-01T22:20:13.723 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:13.721+0000 7f7448708700 1 -- 172.21.15.73:0/246083282 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7440108100 con 0x7f74400f5410 2023-12-01T22:20:13.724 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:13.722+0000 7f7436ffd700 1 -- 172.21.15.73:0/246083282 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7430010230 con 0x7f74400f5410 2023-12-01T22:20:13.724 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-12-01T22:20:13.724 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"fsid":"5558ab6c-9096-11ee-95a2-87774f69a715","modified":"2023-12-01T22:10:10.228477Z","created":"2023-12-01T22:10:10.228477Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-01T22:20:13.725 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:13.724+0000 7f7448708700 1 -- 172.21.15.73:0/246083282 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f742c02ff60 msgr2=0x7f742c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:20:13.725 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:13.724+0000 7f7448708700 1 --2- 172.21.15.73:0/246083282 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f742c02ff60 0x7f742c032420 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7f743c0096a0 tx=0x7f743c006b40).stop 2023-12-01T22:20:13.725 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:13.724+0000 7f7448708700 1 -- 172.21.15.73:0/246083282 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74400f5410 msgr2=0x7f74401066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T22:20:13.725 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:13.724+0000 7f7448708700 1 --2- 172.21.15.73:0/246083282 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74400f5410 0x7f74401066d0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 rx=0x7f7430013510 tx=0x7f743000b040).stop 2023-12-01T22:20:13.726 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:13.724+0000 7f7448708700 1 -- 172.21.15.73:0/246083282 shutdown_connections 2023-12-01T22:20:13.726 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:13.724+0000 7f7448708700 1 --2- 172.21.15.73:0/246083282 >> [v2:172.21.15.29:6800/3698425734,v1:172.21.15.29:6801/3698425734] conn(0x7f742c02ff60 0x7f742c032420 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:20:13.726 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:13.724+0000 7f7448708700 1 --2- 172.21.15.73:0/246083282 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74400f5410 0x7f74401066d0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-01T22:20:13.726 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:13.724+0000 7f7448708700 1 -- 172.21.15.73:0/246083282 >> 172.21.15.73:0/246083282 conn(0x7f74400f0b70 msgr2=0x7f744005fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T22:20:13.726 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:13.724+0000 7f7448708700 1 -- 172.21.15.73:0/246083282 shutdown_connections 2023-12-01T22:20:13.726 INFO:teuthology.orchestra.run.smithi073.stderr:2023-12-01T22:20:13.724+0000 7f7448708700 1 -- 172.21.15.73:0/246083282 wait complete. 2023-12-01T22:20:13.726 INFO:teuthology.orchestra.run.smithi073.stderr:dumped monmap epoch 1 2023-12-01T22:20:14.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:20:13 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: cluster 2023-12-01T22:20:12.707685+0000 mgr.smithi029.loxrfu (mgr.14164) 278 : cluster [DBG] pgmap v264: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-01T22:20:14.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:20:13 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:20:12.785278+0000 mon.smithi029 (mon.0) 326 : audit [INF] from='mgr.14164 172.21.15.29:0/3195927111' entity='mgr.smithi029.loxrfu' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi029.loxrfu/mirror_snapshot_schedule"}]: dispatch 2023-12-01T22:20:14.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:20:13 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:20:12.788143+0000 mon.smithi029 (mon.0) 327 : audit [INF] from='mgr.14164 172.21.15.29:0/3195927111' entity='mgr.smithi029.loxrfu' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi029.loxrfu/trash_purge_schedule"}]: dispatch 2023-12-01T22:20:14.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:20:13 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: audit 2023-12-01T22:20:13.723606+0000 mon.smithi029 (mon.0) 328 : audit [DBG] from='client.? 172.21.15.73:0/246083282' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-01T22:20:14.132 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_0dc074ce0b0b3b4f393f56dd32ef36884ebf45a9/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds 2023-12-01T22:20:14.134 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2023-12-01T22:20:14.134 DEBUG:teuthology.orchestra.run.smithi029:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-12-01T22:20:14.160 DEBUG:teuthology.orchestra.run.smithi073:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-12-01T22:20:14.178 INFO:tasks.cephadm:Stopping all daemons... 2023-12-01T22:20:14.179 INFO:tasks.cephadm.mon.smithi029:Stopping mon.smithi029... 2023-12-01T22:20:14.179 DEBUG:teuthology.orchestra.run.smithi029:> sudo systemctl stop ceph-5558ab6c-9096-11ee-95a2-87774f69a715@mon.smithi029 2023-12-01T22:20:14.527 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:20:14 smithi029 systemd[1]: Stopping Ceph mon.smithi029 for 5558ab6c-9096-11ee-95a2-87774f69a715... 2023-12-01T22:20:14.527 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:20:14 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:20:14.309+0000 7f66ea27b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi029 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2023-12-01T22:20:14.527 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Dec 01 22:20:14 smithi029 ceph-5558ab6c-9096-11ee-95a2-87774f69a715-mon.smithi029[111775]: debug 2023-12-01T22:20:14.310+0000 7f66ea27b700 -1 mon.smithi029@0(leader) e1 *** Got Signal Terminated *** 2023-12-01T22:20:14.753 DEBUG:teuthology.orchestra.run.smithi029:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5558ab6c-9096-11ee-95a2-87774f69a715@mon.smithi029.service' 2023-12-01T22:20:14.813 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-12-01T22:20:14.813 INFO:tasks.cephadm.mon.smithi029:Stopped mon.smithi029 2023-12-01T22:20:14.813 INFO:tasks.cephadm.mon.smithi073:Stopping mon.smithi073... 2023-12-01T22:20:14.813 DEBUG:teuthology.orchestra.run.smithi073:> sudo systemctl stop ceph-5558ab6c-9096-11ee-95a2-87774f69a715@mon.smithi073 2023-12-01T22:20:14.850 INFO:teuthology.orchestra.run.smithi073.stderr:Failed to stop ceph-5558ab6c-9096-11ee-95a2-87774f69a715@mon.smithi073.service: Unit ceph-5558ab6c-9096-11ee-95a2-87774f69a715@mon.smithi073.service not loaded. 2023-12-01T22:20:14.851 DEBUG:teuthology.orchestra.run:got remote process result: 5 2023-12-01T22:20:14.852 ERROR:tasks.cephadm:Failed to stop "ceph.mon.smithi073" Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_0dc074ce0b0b3b4f393f56dd32ef36884ebf45a9/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_0dc074ce0b0b3b4f393f56dd32ef36884ebf45a9/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_0dc074ce0b0b3b4f393f56dd32ef36884ebf45a9/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi073 with status 5: 'sudo systemctl stop ceph-5558ab6c-9096-11ee-95a2-87774f69a715@mon.smithi073' 2023-12-01T22:20:14.853 INFO:tasks.cephadm:Archiving crash dumps... 2023-12-01T22:20:14.854 DEBUG:teuthology.misc:Transferring archived files from smithi029:/var/lib/ceph/5558ab6c-9096-11ee-95a2-87774f69a715/crash to /home/teuthworker/archive/yuriw-2023-12-01_16:53:58-rados-wip-yuri8-testing-2023-11-29-0706-pacific-distro-default-smithi/7474315/remote/ubuntu@smithi029.front.sepia.ceph.com/crash 2023-12-01T22:20:14.855 DEBUG:teuthology.orchestra.run.smithi029:> sudo tar c -f - -C /var/lib/ceph/5558ab6c-9096-11ee-95a2-87774f69a715/crash -- . 2023-12-01T22:20:14.907 DEBUG:teuthology.misc:Transferring archived files from smithi073:/var/lib/ceph/5558ab6c-9096-11ee-95a2-87774f69a715/crash to /home/teuthworker/archive/yuriw-2023-12-01_16:53:58-rados-wip-yuri8-testing-2023-11-29-0706-pacific-distro-default-smithi/7474315/remote/ubuntu@smithi073.front.sepia.ceph.com/crash 2023-12-01T22:20:14.908 DEBUG:teuthology.orchestra.run.smithi073:> sudo tar c -f - -C /var/lib/ceph/5558ab6c-9096-11ee-95a2-87774f69a715/crash -- . 2023-12-01T22:20:14.954 INFO:teuthology.orchestra.run.smithi073.stderr:tar: /var/lib/ceph/5558ab6c-9096-11ee-95a2-87774f69a715/crash: Cannot open: No such file or directory 2023-12-01T22:20:14.954 INFO:teuthology.orchestra.run.smithi073.stderr:tar: Error is not recoverable: exiting now 2023-12-01T22:20:14.955 INFO:tasks.cephadm:Checking cluster log for badness... 2023-12-01T22:20:14.955 DEBUG:teuthology.orchestra.run.smithi029:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/5558ab6c-9096-11ee-95a2-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2023-12-01T22:20:14.993 INFO:teuthology.orchestra.run.smithi029.stderr:grep: /var/log/ceph/5558ab6c-9096-11ee-95a2-87774f69a715/ceph.log: No such file or directory 2023-12-01T22:20:14.996 INFO:tasks.cephadm:Compressing logs... 2023-12-01T22:20:14.996 DEBUG:teuthology.orchestra.run.smithi029:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-12-01T22:20:15.039 DEBUG:teuthology.orchestra.run.smithi073:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-12-01T22:20:15.070 INFO:teuthology.orchestra.run.smithi073.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2023-12-01T22:20:15.070 INFO:teuthology.orchestra.run.smithi073.stderr:: No such file or directory 2023-12-01T22:20:15.071 INFO:teuthology.orchestra.run.smithi029.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2023-12-01T22:20:15.072 INFO:teuthology.orchestra.run.smithi029.stderr:: No such file or directory 2023-12-01T22:20:15.073 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/5558ab6c-9096-11ee-95a2-87774f69a715/ceph-volume.log 2023-12-01T22:20:15.073 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/5558ab6c-9096-11ee-95a2-87774f69a715/ceph-mon.smithi029.log 2023-12-01T22:20:15.073 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/5558ab6c-9096-11ee-95a2-87774f69a715/ceph-mgr.smithi029.loxrfu.log 2023-12-01T22:20:15.073 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/cephadm.log: 96.1% -- replaced with /var/log/ceph/cephadm.log.gz 2023-12-01T22:20:15.074 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/5558ab6c-9096-11ee-95a2-87774f69a715/ceph-volume.log: /var/log/ceph/5558ab6c-9096-11ee-95a2-87774f69a715/ceph-mon.smithi029.log: 88.6% -- replaced with /var/log/ceph/5558ab6c-9096-11ee-95a2-87774f69a715/ceph-volume.log.gz 2023-12-01T22:20:15.074 INFO:teuthology.orchestra.run.smithi073.stderr: 2023-12-01T22:20:15.075 INFO:teuthology.orchestra.run.smithi073.stderr:real 0m0.015s 2023-12-01T22:20:15.075 INFO:teuthology.orchestra.run.smithi073.stderr:user 0m0.011s 2023-12-01T22:20:15.075 INFO:teuthology.orchestra.run.smithi073.stderr:sys 0m0.014s 2023-12-01T22:20:15.080 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/5558ab6c-9096-11ee-95a2-87774f69a715/ceph-mgr.smithi029.loxrfu.log: 91.7% -- replaced with /var/log/ceph/cephadm.log.gz 2023-12-01T22:20:15.159 INFO:teuthology.orchestra.run.smithi029.stderr: 92.6% -- replaced with /var/log/ceph/5558ab6c-9096-11ee-95a2-87774f69a715/ceph-mgr.smithi029.loxrfu.log.gz 2023-12-01T22:20:15.231 INFO:teuthology.orchestra.run.smithi029.stderr: 90.6% -- replaced with /var/log/ceph/5558ab6c-9096-11ee-95a2-87774f69a715/ceph-mon.smithi029.log.gz 2023-12-01T22:20:15.231 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-12-01T22:20:15.231 INFO:teuthology.orchestra.run.smithi029.stderr:real 0m0.171s 2023-12-01T22:20:15.231 INFO:teuthology.orchestra.run.smithi029.stderr:user 0m0.255s 2023-12-01T22:20:15.231 INFO:teuthology.orchestra.run.smithi029.stderr:sys 0m0.023s 2023-12-01T22:20:15.231 INFO:tasks.cephadm:Archiving logs... 2023-12-01T22:20:15.231 DEBUG:teuthology.misc:Transferring archived files from smithi029:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-12-01_16:53:58-rados-wip-yuri8-testing-2023-11-29-0706-pacific-distro-default-smithi/7474315/remote/ubuntu@smithi029.front.sepia.ceph.com/log 2023-12-01T22:20:15.232 DEBUG:teuthology.orchestra.run.smithi029:> sudo tar c -f - -C /var/log/ceph -- . 2023-12-01T22:20:15.353 DEBUG:teuthology.misc:Transferring archived files from smithi073:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-12-01_16:53:58-rados-wip-yuri8-testing-2023-11-29-0706-pacific-distro-default-smithi/7474315/remote/ubuntu@smithi073.front.sepia.ceph.com/log 2023-12-01T22:20:15.354 DEBUG:teuthology.orchestra.run.smithi073:> sudo tar c -f - -C /var/log/ceph -- . 2023-12-01T22:20:15.387 INFO:tasks.cephadm:Removing cluster... 2023-12-01T22:20:15.388 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 --force 2023-12-01T22:21:12.309 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 5558ab6c-9096-11ee-95a2-87774f69a715 --force 2023-12-01T22:21:12.608 INFO:tasks.cephadm:Removing cephadm ... 2023-12-01T22:21:12.608 DEBUG:teuthology.orchestra.run.smithi029:> rm -rf /home/ubuntu/cephtest/cephadm 2023-12-01T22:21:12.632 DEBUG:teuthology.orchestra.run.smithi073:> rm -rf /home/ubuntu/cephtest/cephadm 2023-12-01T22:21:12.650 INFO:tasks.cephadm:Teardown complete 2023-12-01T22:21:12.650 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_0dc074ce0b0b3b4f393f56dd32ef36884ebf45a9/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_0dc074ce0b0b3b4f393f56dd32ef36884ebf45a9/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_0dc074ce0b0b3b4f393f56dd32ef36884ebf45a9/qa/tasks/cephadm.py", line 1463, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_0dc074ce0b0b3b4f393f56dd32ef36884ebf45a9/qa/tasks/cephadm.py", line 1376, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_0dc074ce0b0b3b4f393f56dd32ef36884ebf45a9/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi073 with status 5: 'sudo systemctl stop ceph-5558ab6c-9096-11ee-95a2-87774f69a715@mon.smithi073' 2023-12-01T22:21:12.746 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=8c992d15e58c487cae15e2d453fbfcad Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_0dc074ce0b0b3b4f393f56dd32ef36884ebf45a9/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_0dc074ce0b0b3b4f393f56dd32ef36884ebf45a9/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_0dc074ce0b0b3b4f393f56dd32ef36884ebf45a9/qa/tasks/cephadm.py", line 1463, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_0dc074ce0b0b3b4f393f56dd32ef36884ebf45a9/qa/tasks/cephadm.py", line 1376, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_0dc074ce0b0b3b4f393f56dd32ef36884ebf45a9/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi073 with status 5: 'sudo systemctl stop ceph-5558ab6c-9096-11ee-95a2-87774f69a715@mon.smithi073' 2023-12-01T22:21:12.752 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2023-12-01T22:21:12.762 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-12-01T22:21:12.771 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-12-01T22:21:12.771 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-12-01T22:21:12.795 DEBUG:teuthology.orchestra.run.smithi073:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-12-01T22:21:12.874 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-12-01T22:21:12.884 INFO:teuthology.task.clock:Checking final clock skew... 2023-12-01T22:21:12.884 DEBUG:teuthology.orchestra.run.smithi029:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-01T22:21:12.887 DEBUG:teuthology.orchestra.run.smithi073:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-01T22:21:12.904 INFO:teuthology.orchestra.run.smithi073.stderr:bash: ntpq: command not found 2023-12-01T22:21:12.907 INFO:teuthology.orchestra.run.smithi029.stderr:bash: ntpq: command not found 2023-12-01T22:21:12.910 INFO:teuthology.orchestra.run.smithi073.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-01T22:21:12.910 INFO:teuthology.orchestra.run.smithi073.stdout:=============================================================================== 2023-12-01T22:21:12.910 INFO:teuthology.orchestra.run.smithi073.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 45 -448us[ -458us] +/- 88ms 2023-12-01T22:21:12.910 INFO:teuthology.orchestra.run.smithi073.stdout:^* hv02.front.sepia.ceph.com 3 6 377 44 -917us[ -927us] +/- 55ms 2023-12-01T22:21:12.911 INFO:teuthology.orchestra.run.smithi073.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 46 +1427us[+1417us] +/- 69ms 2023-12-01T22:21:12.911 INFO:teuthology.orchestra.run.smithi073.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-12-01T22:21:12.912 INFO:teuthology.orchestra.run.smithi029.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-01T22:21:12.912 INFO:teuthology.orchestra.run.smithi029.stdout:=============================================================================== 2023-12-01T22:21:12.913 INFO:teuthology.orchestra.run.smithi029.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 45 -418us[ -418us] +/- 88ms 2023-12-01T22:21:12.913 INFO:teuthology.orchestra.run.smithi029.stdout:^* hv02.front.sepia.ceph.com 3 6 377 46 -901us[ -895us] +/- 55ms 2023-12-01T22:21:12.913 INFO:teuthology.orchestra.run.smithi029.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 47 +1454us[+1460us] +/- 69ms 2023-12-01T22:21:12.913 INFO:teuthology.orchestra.run.smithi029.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-12-01T22:21:12.913 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-12-01T22:21:12.923 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-12-01T22:21:12.923 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-12-01T22:21:12.932 DEBUG:teuthology.orchestra.run.smithi029:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-12-01T22:21:13.057 DEBUG:teuthology.orchestra.run.smithi073:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-12-01T22:21:13.147 DEBUG:teuthology.orchestra.run.smithi029:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-01T22:21:13.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-01T22:21:13.186 DEBUG:teuthology.orchestra.run.smithi073:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-01T22:21:13.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-01T22:21:13.219 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-12-01T22:21:13.231 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-12-01T22:21:13.240 INFO:teuthology.task.internal:Duration was 1464.474276 seconds 2023-12-01T22:21:13.240 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-12-01T22:21:13.249 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-12-01T22:21:13.250 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-12-01T22:21:13.252 DEBUG:teuthology.orchestra.run.smithi073:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-12-01T22:21:13.305 INFO:teuthology.orchestra.run.smithi073.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-01T22:21:13.309 INFO:teuthology.orchestra.run.smithi029.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-01T22:21:13.541 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-12-01T22:21:13.541 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi029.front.sepia.ceph.com 2023-12-01T22:21:13.542 DEBUG:teuthology.orchestra.run.smithi029:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-12-01T22:21:13.572 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi073.front.sepia.ceph.com 2023-12-01T22:21:13.572 DEBUG:teuthology.orchestra.run.smithi073:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-12-01T22:21:13.594 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-12-01T22:21:13.594 DEBUG:teuthology.orchestra.run.smithi029:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-12-01T22:21:13.615 DEBUG:teuthology.orchestra.run.smithi073:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-12-01T22:21:13.679 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-12-01T22:21:13.680 DEBUG:teuthology.orchestra.run.smithi029:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-12-01T22:21:13.682 DEBUG:teuthology.orchestra.run.smithi073:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-12-01T22:21:13.785 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-12-01T22:21:13.795 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-12-01T22:21:13.796 DEBUG:teuthology.orchestra.run.smithi029:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-12-01T22:21:13.827 DEBUG:teuthology.orchestra.run.smithi073:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-12-01T22:21:13.854 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-12-01T22:21:13.865 DEBUG:teuthology.orchestra.run.smithi029:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-12-01T22:21:13.871 DEBUG:teuthology.orchestra.run.smithi073:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-12-01T22:21:13.901 INFO:teuthology.orchestra.run.smithi029.stdout:kernel.core_pattern = core 2023-12-01T22:21:13.921 INFO:teuthology.orchestra.run.smithi073.stdout:kernel.core_pattern = core 2023-12-01T22:21:13.940 DEBUG:teuthology.orchestra.run.smithi029:> test -e /home/ubuntu/cephtest/archive/coredump 2023-12-01T22:21:13.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-01T22:21:13.990 DEBUG:teuthology.orchestra.run.smithi073:> test -e /home/ubuntu/cephtest/archive/coredump 2023-12-01T22:21:14.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-01T22:21:14.007 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-12-01T22:21:14.018 INFO:teuthology.task.internal:Transferring archived files... 2023-12-01T22:21:14.019 DEBUG:teuthology.misc:Transferring archived files from smithi029:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-12-01_16:53:58-rados-wip-yuri8-testing-2023-11-29-0706-pacific-distro-default-smithi/7474315/remote/smithi029 2023-12-01T22:21:14.020 DEBUG:teuthology.orchestra.run.smithi029:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-12-01T22:21:14.077 DEBUG:teuthology.misc:Transferring archived files from smithi073:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-12-01_16:53:58-rados-wip-yuri8-testing-2023-11-29-0706-pacific-distro-default-smithi/7474315/remote/smithi073 2023-12-01T22:21:14.078 DEBUG:teuthology.orchestra.run.smithi073:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-12-01T22:21:14.116 INFO:teuthology.task.internal:Removing archive directory... 2023-12-01T22:21:14.116 DEBUG:teuthology.orchestra.run.smithi029:> rm -rf -- /home/ubuntu/cephtest/archive 2023-12-01T22:21:14.119 DEBUG:teuthology.orchestra.run.smithi073:> rm -rf -- /home/ubuntu/cephtest/archive 2023-12-01T22:21:14.168 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-12-01T22:21:14.179 INFO:teuthology.task.internal:Not uploading archives. 2023-12-01T22:21:14.180 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-12-01T22:21:14.189 INFO:teuthology.task.internal:Tidying up after the test... 2023-12-01T22:21:14.190 DEBUG:teuthology.orchestra.run.smithi029:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-12-01T22:21:14.193 DEBUG:teuthology.orchestra.run.smithi073:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-12-01T22:21:14.212 INFO:teuthology.orchestra.run.smithi029.stdout: 525364 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Dec 1 22:21 /home/ubuntu/cephtest 2023-12-01T22:21:14.225 INFO:teuthology.orchestra.run.smithi073.stdout: 526080 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Dec 1 22:21 /home/ubuntu/cephtest 2023-12-01T22:21:14.227 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-12-01T22:21:14.237 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-12-01T22:21:14.261 INFO:teuthology.nuke:Checking targets against current locks 2023-12-01T22:21:14.288 DEBUG:teuthology.nuke:shortname: smithi029 2023-12-01T22:21:14.288 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-01T22:21:14.313 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi029.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-01_16:53:58-rados-wip-yuri8-testing-2023-11-29-0706-pacific-distro-default-smithi/7474315', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-01 21:47:57.719678', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE/Gu7jodpkoe5qWBuiwaFjNwhjul3l3KPVahRuQC55JMjyXx24DHCmCGW02tLw76baQ3mep7UA/wI5eV8bv24M='} 2023-12-01T22:21:14.316 DEBUG:teuthology.nuke:shortname: smithi073 2023-12-01T22:21:14.316 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-01T22:21:14.338 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi073.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-01_16:53:58-rados-wip-yuri8-testing-2023-11-29-0706-pacific-distro-default-smithi/7474315', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-01 21:47:57.721648', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDvGhdhX/gmwksEZ0RDbX9usvX71/SkQPZMnaOpgQuILm1Bcrctk4U7dn2bknhIe8bsWP/eN604nGENK8O4gMZ8='} 2023-12-01T22:21:14.367 INFO:teuthology.orchestra.console.smithi029:Power off 2023-12-01T22:21:14.367 DEBUG:teuthology.orchestra.console.smithi029:pexpect command: ipmitool -H smithi029.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-12-01T22:21:14.395 DEBUG:teuthology.orchestra.console.smithi029:power off output: Chassis Power Control: Down/Off 2023-12-01T22:21:14.395 DEBUG:teuthology.orchestra.console.smithi029:pexpect command: ipmitool -H smithi029.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-01T22:21:14.415 INFO:teuthology.orchestra.console.smithi073:Power off 2023-12-01T22:21:14.415 DEBUG:teuthology.orchestra.console.smithi073:pexpect command: ipmitool -H smithi073.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-12-01T22:21:14.425 DEBUG:teuthology.orchestra.console.smithi029:check power output: Chassis Power is on 2023-12-01T22:21:14.438 DEBUG:teuthology.orchestra.console.smithi073:power off output: Chassis Power Control: Down/Off 2023-12-01T22:21:14.439 DEBUG:teuthology.orchestra.console.smithi073:pexpect command: ipmitool -H smithi073.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-01T22:21:14.463 DEBUG:teuthology.orchestra.console.smithi073:check power output: Chassis Power is on 2023-12-01T22:21:18.427 DEBUG:teuthology.orchestra.console.smithi029:pexpect command: ipmitool -H smithi029.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-01T22:21:18.464 DEBUG:teuthology.orchestra.console.smithi073:pexpect command: ipmitool -H smithi073.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-01T22:21:18.537 DEBUG:teuthology.orchestra.console.smithi029:check power output: Chassis Power is on 2023-12-01T22:21:18.575 DEBUG:teuthology.orchestra.console.smithi073:check power output: Chassis Power is on 2023-12-01T22:21:22.542 DEBUG:teuthology.orchestra.console.smithi029:pexpect command: ipmitool -H smithi029.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-01T22:21:22.576 DEBUG:teuthology.orchestra.console.smithi073:pexpect command: ipmitool -H smithi073.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-01T22:21:22.653 DEBUG:teuthology.orchestra.console.smithi029:check power output: Chassis Power is off 2023-12-01T22:21:22.687 DEBUG:teuthology.orchestra.console.smithi073:check power output: Chassis Power is on 2023-12-01T22:21:22.755 INFO:teuthology.orchestra.console.smithi029:Power off completed 2023-12-01T22:21:26.690 DEBUG:teuthology.orchestra.console.smithi073:pexpect command: ipmitool -H smithi073.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-01T22:21:26.803 DEBUG:teuthology.orchestra.console.smithi073:check power output: Chassis Power is off 2023-12-01T22:21:26.904 INFO:teuthology.orchestra.console.smithi073:Power off completed 2023-12-01T22:21:27.009 INFO:teuthology.run:Summary data: description: rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} duration: 1464.474276304245 failure_reason: 'Command failed on smithi073 with status 5: ''sudo systemctl stop ceph-5558ab6c-9096-11ee-95a2-87774f69a715@mon.smithi073''' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=8c992d15e58c487cae15e2d453fbfcad status: fail success: false 2023-12-01T22:21:27.009 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-01T22:21:27.085 INFO:teuthology.run:FAIL