2023-12-25T19:50:29.176 INFO:root:teuthology version: 0.0.1.dev226+g9680d2a 2023-12-25T19:50:29.176 DEBUG:teuthology.run:Teuthology command: teuthology --name yuriw-2023-12-25_17:28:36-rados-wip-yuri5-testing-2023-12-15-0747-pacific-distro-default-smithi --description rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} --verbose --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2023-12-25_17:28:36-rados-wip-yuri5-testing-2023-12-15-0747-pacific-distro-default-smithi/7500869 -- /home/teuthworker/archive/yuriw-2023-12-25_17:28:36-rados-wip-yuri5-testing-2023-12-15-0747-pacific-distro-default-smithi/7500869/orig.config.yaml 2023-12-25T19:50:29.226 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-25T19:50:29.310 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-12-25_17:28:36-rados-wip-yuri5-testing-2023-12-15-0747-pacific-distro-default-smithi/7500869 branch: wip-yuri5-testing-2023-12-15-0747-pacific description: rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} email: yweinste@redhat.com first_in_suite: false job_id: '7500869' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'setup ceph/pacific v16.2.4 ' name: yuriw-2023-12-25_17:28:36-rados-wip-yuri5-testing-2023-12-15-0747-pacific-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-yuri5-testing-2023-12-15-0747-pacific ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: mon pg warn min per osd: 0 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding sha1: b14a0a106b8cffae0144ff9fb83d9ce1af1a4bd2 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: b14a0a106b8cffae0144ff9fb83d9ce1af1a4bd2 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri5-testing-2023-12-15-0747-pacific sha1: b14a0a106b8cffae0144ff9fb83d9ce1af1a4bd2 owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - host.a - client.0 - osd.0 - osd.1 - osd.2 - - host.b - client.1 - osd.3 - osd.4 - osd.5 seed: 2834 sha1: b14a0a106b8cffae0144ff9fb83d9ce1af1a4bd2 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri5-testing-2023-12-15-0747-pacific suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_b14a0a106b8cffae0144ff9fb83d9ce1af1a4bd2/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: b14a0a106b8cffae0144ff9fb83d9ce1af1a4bd2 targets: smithi050.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDUisXIgHIseMJVWCQr2pJaStGQPrO9ixhqevO3+QgZvApgL+eVstcZ9+0GeOLVR5I3NJcbS+YnnDVLmLTWmD+U= smithi160.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMh/duOsT8rOcQLFOpEZX5ZqrEYscseh2K7m3kgPRUTPE+IcOg6z9u92kDmVc8jMA2wITbTMMP0bfNGrH9JJZyc= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - pexec: clients: - sudo modprobe -r ceph - sudo modprobe ceph disable_send_metrics=on - install: exclude_packages: - ceph-volume tag: v16.2.4 - print: '**** done install task...' - cephadm: avoid_pacific_features: true cephadm_branch: v16.2.4 cephadm_git_url: https://github.com/ceph/ceph image: docker.io/ceph/ceph:v16.2.4 roleless: true - print: '**** done starting v16.2.4' - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: host.a: - ceph fs volume create cephfs --placement=4 - ceph fs dump - cephadm.shell: host.a: - ceph fs set cephfs max_mds 1 - cephadm.shell: host.a: - ceph fs set cephfs allow_standby_replay false - cephadm.shell: host.a: - ceph fs set cephfs inline_data false - cephadm.shell: host.a: - ceph fs dump - ceph --format=json fs dump | jq -e ".filesystems | length == 1" - while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done - fs.pre_upgrade_save: null - kclient: null - print: '**** done client' - parallel: - upgrade-tasks - workload-tasks - cephadm.shell: host.a: - ceph fs dump - fs.post_upgrade_checks: null teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 9680d2aba1e76f64ade295c81599c97907457848 timestamp: 2023-12-25_17:28:36 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.1485078 workload-tasks: sequential: - workunit: clients: all: - suites/fsstress.sh 2023-12-25T19:50:29.310 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_b14a0a106b8cffae0144ff9fb83d9ce1af1a4bd2/qa; will attempt to use it 2023-12-25T19:50:29.311 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_b14a0a106b8cffae0144ff9fb83d9ce1af1a4bd2/qa/tasks 2023-12-25T19:50:29.311 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-12-25T19:50:29.312 INFO:teuthology.task.internal:Checking packages... 2023-12-25T19:50:29.335 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'b14a0a106b8cffae0144ff9fb83d9ce1af1a4bd2' 2023-12-25T19:50:29.336 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-12-25T19:50:29.336 INFO:teuthology.packaging:ref: None 2023-12-25T19:50:29.336 INFO:teuthology.packaging:tag: None 2023-12-25T19:50:29.336 INFO:teuthology.packaging:branch: wip-yuri5-testing-2023-12-15-0747-pacific 2023-12-25T19:50:29.336 INFO:teuthology.packaging:sha1: b14a0a106b8cffae0144ff9fb83d9ce1af1a4bd2 2023-12-25T19:50:29.336 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri5-testing-2023-12-15-0747-pacific 2023-12-25T19:50:29.486 INFO:teuthology.task.internal:Found packages for ceph version 16.2.14-385.gb14a0a10 2023-12-25T19:50:29.488 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-12-25T19:50:29.495 INFO:teuthology.task.internal:no buildpackages task found 2023-12-25T19:50:29.496 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-12-25T19:50:29.503 INFO:teuthology.task.internal:Saving configuration 2023-12-25T19:50:29.523 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-12-25T19:50:29.529 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-25T19:50:29.553 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi050.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-25_17:28:36-rados-wip-yuri5-testing-2023-12-15-0747-pacific-distro-default-smithi/7500869', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-25 19:45:06.249870', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDUisXIgHIseMJVWCQr2pJaStGQPrO9ixhqevO3+QgZvApgL+eVstcZ9+0GeOLVR5I3NJcbS+YnnDVLmLTWmD+U='} 2023-12-25T19:50:29.582 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi160.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-25_17:28:36-rados-wip-yuri5-testing-2023-12-15-0747-pacific-distro-default-smithi/7500869', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-25 19:45:06.248307', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMh/duOsT8rOcQLFOpEZX5ZqrEYscseh2K7m3kgPRUTPE+IcOg6z9u92kDmVc8jMA2wITbTMMP0bfNGrH9JJZyc='} 2023-12-25T19:50:29.582 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-12-25T19:50:29.595 INFO:teuthology.task.internal:roles: ubuntu@smithi050.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2023-12-25T19:50:29.595 INFO:teuthology.task.internal:roles: ubuntu@smithi160.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2023-12-25T19:50:29.595 INFO:teuthology.run_tasks:Running task console_log... 2023-12-25T19:50:29.689 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f299689ae50>, signals=[15]) 2023-12-25T19:50:29.690 INFO:teuthology.run_tasks:Running task internal.connect... 2023-12-25T19:50:29.703 INFO:teuthology.task.internal:Opening connections... 2023-12-25T19:50:29.703 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi050.front.sepia.ceph.com 2023-12-25T19:50:29.704 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi050.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-25T19:50:29.774 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi160.front.sepia.ceph.com 2023-12-25T19:50:29.775 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-25T19:50:29.840 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-12-25T19:50:29.847 DEBUG:teuthology.orchestra.run.smithi050:> uname -m 2023-12-25T19:50:29.871 INFO:teuthology.orchestra.run.smithi050.stdout:x86_64 2023-12-25T19:50:29.871 DEBUG:teuthology.orchestra.run.smithi050:> cat /etc/os-release 2023-12-25T19:50:29.933 INFO:teuthology.orchestra.run.smithi050.stdout:NAME="CentOS Stream" 2023-12-25T19:50:29.933 INFO:teuthology.orchestra.run.smithi050.stdout:VERSION="8" 2023-12-25T19:50:29.933 INFO:teuthology.orchestra.run.smithi050.stdout:ID="centos" 2023-12-25T19:50:29.933 INFO:teuthology.orchestra.run.smithi050.stdout:ID_LIKE="rhel fedora" 2023-12-25T19:50:29.933 INFO:teuthology.orchestra.run.smithi050.stdout:VERSION_ID="8" 2023-12-25T19:50:29.934 INFO:teuthology.orchestra.run.smithi050.stdout:PLATFORM_ID="platform:el8" 2023-12-25T19:50:29.934 INFO:teuthology.orchestra.run.smithi050.stdout:PRETTY_NAME="CentOS Stream 8" 2023-12-25T19:50:29.934 INFO:teuthology.orchestra.run.smithi050.stdout:ANSI_COLOR="0;31" 2023-12-25T19:50:29.934 INFO:teuthology.orchestra.run.smithi050.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-12-25T19:50:29.934 INFO:teuthology.orchestra.run.smithi050.stdout:HOME_URL="https://centos.org/" 2023-12-25T19:50:29.934 INFO:teuthology.orchestra.run.smithi050.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-12-25T19:50:29.934 INFO:teuthology.orchestra.run.smithi050.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-12-25T19:50:29.934 INFO:teuthology.orchestra.run.smithi050.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-12-25T19:50:29.935 INFO:teuthology.lock.ops:Updating smithi050.front.sepia.ceph.com on lock server 2023-12-25T19:50:29.955 DEBUG:teuthology.orchestra.run.smithi160:> uname -m 2023-12-25T19:50:29.977 INFO:teuthology.orchestra.run.smithi160.stdout:x86_64 2023-12-25T19:50:29.978 DEBUG:teuthology.orchestra.run.smithi160:> cat /etc/os-release 2023-12-25T19:50:30.039 INFO:teuthology.orchestra.run.smithi160.stdout:NAME="CentOS Stream" 2023-12-25T19:50:30.039 INFO:teuthology.orchestra.run.smithi160.stdout:VERSION="8" 2023-12-25T19:50:30.039 INFO:teuthology.orchestra.run.smithi160.stdout:ID="centos" 2023-12-25T19:50:30.039 INFO:teuthology.orchestra.run.smithi160.stdout:ID_LIKE="rhel fedora" 2023-12-25T19:50:30.039 INFO:teuthology.orchestra.run.smithi160.stdout:VERSION_ID="8" 2023-12-25T19:50:30.039 INFO:teuthology.orchestra.run.smithi160.stdout:PLATFORM_ID="platform:el8" 2023-12-25T19:50:30.039 INFO:teuthology.orchestra.run.smithi160.stdout:PRETTY_NAME="CentOS Stream 8" 2023-12-25T19:50:30.039 INFO:teuthology.orchestra.run.smithi160.stdout:ANSI_COLOR="0;31" 2023-12-25T19:50:30.039 INFO:teuthology.orchestra.run.smithi160.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-12-25T19:50:30.039 INFO:teuthology.orchestra.run.smithi160.stdout:HOME_URL="https://centos.org/" 2023-12-25T19:50:30.039 INFO:teuthology.orchestra.run.smithi160.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-12-25T19:50:30.040 INFO:teuthology.orchestra.run.smithi160.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-12-25T19:50:30.040 INFO:teuthology.orchestra.run.smithi160.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-12-25T19:50:30.040 INFO:teuthology.lock.ops:Updating smithi160.front.sepia.ceph.com on lock server 2023-12-25T19:50:30.064 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-12-25T19:50:30.075 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-12-25T19:50:30.086 INFO:teuthology.task.internal:Checking for old test directory... 2023-12-25T19:50:30.086 DEBUG:teuthology.orchestra.run.smithi050:> test '!' -e /home/ubuntu/cephtest 2023-12-25T19:50:30.089 DEBUG:teuthology.orchestra.run.smithi160:> test '!' -e /home/ubuntu/cephtest 2023-12-25T19:50:30.110 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-12-25T19:50:30.120 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-12-25T19:50:30.120 DEBUG:teuthology.orchestra.run.smithi050:> test -z $(ls -A /var/lib/ceph) 2023-12-25T19:50:30.151 DEBUG:teuthology.orchestra.run.smithi160:> test -z $(ls -A /var/lib/ceph) 2023-12-25T19:50:30.179 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-12-25T19:50:30.231 INFO:teuthology.run_tasks:Running task kernel... 2023-12-25T19:50:30.246 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-12-25T19:50:30.246 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro'}, 'host.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-12-25T19:50:30.246 DEBUG:teuthology.orchestra.run.smithi050:> test -f /run/.containerenv -o -f /.dockerenv 2023-12-25T19:50:30.247 DEBUG:teuthology.orchestra.run.smithi160:> test -f /run/.containerenv -o -f /.dockerenv 2023-12-25T19:50:30.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-25T19:50:30.267 DEBUG:teuthology.orchestra.run.smithi050:> uname -r 2023-12-25T19:50:30.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-25T19:50:30.268 DEBUG:teuthology.orchestra.run.smithi160:> uname -r 2023-12-25T19:50:30.329 INFO:teuthology.orchestra.run.smithi050.stdout:4.18.0-489.el8.x86_64 2023-12-25T19:50:30.330 INFO:teuthology.task.kernel:Running kernel on smithi050: 4.18.0-489.el8.x86_64 2023-12-25T19:50:30.330 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum install -y kernel 2023-12-25T19:50:30.331 INFO:teuthology.orchestra.run.smithi160.stdout:4.18.0-489.el8.x86_64 2023-12-25T19:50:30.331 INFO:teuthology.task.kernel:Running kernel on smithi160: 4.18.0-489.el8.x86_64 2023-12-25T19:50:30.331 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum install -y kernel 2023-12-25T19:50:35.277 INFO:teuthology.orchestra.run.smithi160.stdout:CentOS Stream 8 - AppStream 15 MB/s | 35 MB 00:02 2023-12-25T19:50:35.484 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS Stream 8 - AppStream 14 MB/s | 35 MB 00:02 2023-12-25T19:50:56.125 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS Stream 8 - BaseOS 4.1 MB/s | 57 MB 00:13 2023-12-25T19:50:56.925 INFO:teuthology.orchestra.run.smithi160.stdout:CentOS Stream 8 - BaseOS 3.7 MB/s | 57 MB 00:15 2023-12-25T19:51:05.483 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS Stream 8 - Extras 33 kB/s | 18 kB 00:00 2023-12-25T19:51:05.591 INFO:teuthology.orchestra.run.smithi160.stdout:CentOS Stream 8 - Extras 43 kB/s | 18 kB 00:00 2023-12-25T19:51:05.867 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS Stream 8 - Extras common packages 38 kB/s | 6.9 kB 00:00 2023-12-25T19:51:06.117 INFO:teuthology.orchestra.run.smithi160.stdout:CentOS Stream 8 - Extras common packages 20 kB/s | 6.9 kB 00:00 2023-12-25T19:51:06.177 INFO:teuthology.orchestra.run.smithi050.stdout:Copr repo for python3-asyncssh owned by ceph 24 kB/s | 3.5 kB 00:00 2023-12-25T19:51:06.427 INFO:teuthology.orchestra.run.smithi160.stdout:Copr repo for python3-asyncssh owned by ceph 22 kB/s | 3.5 kB 00:00 2023-12-25T19:51:07.763 INFO:teuthology.orchestra.run.smithi050.stdout:Extra Packages for Enterprise Linux 11 MB/s | 16 MB 00:01 2023-12-25T19:51:07.952 INFO:teuthology.orchestra.run.smithi160.stdout:Extra Packages for Enterprise Linux 12 MB/s | 16 MB 00:01 2023-12-25T19:51:11.844 INFO:teuthology.orchestra.run.smithi160.stdout:lab-extras 404 kB/s | 24 kB 00:00 2023-12-25T19:51:11.944 INFO:teuthology.orchestra.run.smithi050.stdout:lab-extras 187 kB/s | 24 kB 00:00 2023-12-25T19:51:15.560 INFO:teuthology.orchestra.run.smithi160.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2023-12-25T19:51:15.560 INFO:teuthology.orchestra.run.smithi160.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2023-12-25T19:51:15.856 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2023-12-25T19:51:15.858 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-12-25T19:51:15.858 INFO:teuthology.orchestra.run.smithi160.stdout: Package Architecture Version Repository Size 2023-12-25T19:51:15.858 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-12-25T19:51:15.858 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2023-12-25T19:51:15.858 INFO:teuthology.orchestra.run.smithi160.stdout: kernel x86_64 4.18.0-529.el8 baseos 10 M 2023-12-25T19:51:15.858 INFO:teuthology.orchestra.run.smithi160.stdout:Installing dependencies: 2023-12-25T19:51:15.858 INFO:teuthology.orchestra.run.smithi160.stdout: kernel-core x86_64 4.18.0-529.el8 baseos 43 M 2023-12-25T19:51:15.858 INFO:teuthology.orchestra.run.smithi160.stdout: kernel-modules x86_64 4.18.0-529.el8 baseos 36 M 2023-12-25T19:51:15.858 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T19:51:15.858 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2023-12-25T19:51:15.859 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-12-25T19:51:15.859 INFO:teuthology.orchestra.run.smithi160.stdout:Install 3 Packages 2023-12-25T19:51:15.859 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T19:51:15.859 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 89 M 2023-12-25T19:51:15.859 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 96 M 2023-12-25T19:51:15.859 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2023-12-25T19:51:15.950 INFO:teuthology.orchestra.run.smithi050.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2023-12-25T19:51:15.950 INFO:teuthology.orchestra.run.smithi050.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2023-12-25T19:51:16.273 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-12-25T19:51:16.274 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-12-25T19:51:16.274 INFO:teuthology.orchestra.run.smithi050.stdout: Package Architecture Version Repository Size 2023-12-25T19:51:16.274 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-12-25T19:51:16.274 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2023-12-25T19:51:16.274 INFO:teuthology.orchestra.run.smithi050.stdout: kernel x86_64 4.18.0-529.el8 baseos 10 M 2023-12-25T19:51:16.275 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2023-12-25T19:51:16.275 INFO:teuthology.orchestra.run.smithi050.stdout: kernel-core x86_64 4.18.0-529.el8 baseos 43 M 2023-12-25T19:51:16.275 INFO:teuthology.orchestra.run.smithi050.stdout: kernel-modules x86_64 4.18.0-529.el8 baseos 36 M 2023-12-25T19:51:16.275 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-25T19:51:16.275 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2023-12-25T19:51:16.275 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-12-25T19:51:16.275 INFO:teuthology.orchestra.run.smithi050.stdout:Install 3 Packages 2023-12-25T19:51:16.275 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-25T19:51:16.276 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 89 M 2023-12-25T19:51:16.276 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 96 M 2023-12-25T19:51:16.276 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2023-12-25T19:51:17.920 INFO:teuthology.orchestra.run.smithi160.stdout:(1/3): kernel-4.18.0-529.el8.x86_64.rpm 5.4 MB/s | 10 MB 00:01 2023-12-25T19:51:18.504 INFO:teuthology.orchestra.run.smithi050.stdout:(1/3): kernel-4.18.0-529.el8.x86_64.rpm 4.7 MB/s | 10 MB 00:02 2023-12-25T19:51:19.687 INFO:teuthology.orchestra.run.smithi160.stdout:(2/3): kernel-core-4.18.0-529.el8.x86_64.rpm 12 MB/s | 43 MB 00:03 2023-12-25T19:51:19.945 INFO:teuthology.orchestra.run.smithi050.stdout:(2/3): kernel-core-4.18.0-529.el8.x86_64.rpm 12 MB/s | 43 MB 00:03 2023-12-25T19:51:20.746 INFO:teuthology.orchestra.run.smithi050.stdout:(3/3): kernel-modules-4.18.0-529.el8.x86_64.rpm 8.2 MB/s | 36 MB 00:04 2023-12-25T19:51:20.746 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2023-12-25T19:51:20.746 INFO:teuthology.orchestra.run.smithi050.stdout:Total 20 MB/s | 89 MB 00:04 2023-12-25T19:51:21.458 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2023-12-25T19:51:21.918 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2023-12-25T19:51:21.918 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2023-12-25T19:51:23.208 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2023-12-25T19:51:23.229 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2023-12-25T19:51:23.278 INFO:teuthology.orchestra.run.smithi160.stdout:(3/3): kernel-modules-4.18.0-529.el8.x86_64.rpm 5.0 MB/s | 36 MB 00:07 2023-12-25T19:51:23.279 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2023-12-25T19:51:23.279 INFO:teuthology.orchestra.run.smithi160.stdout:Total 12 MB/s | 89 MB 00:07 2023-12-25T19:51:23.957 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2023-12-25T19:51:24.377 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2023-12-25T19:51:24.377 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2023-12-25T19:51:25.611 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2023-12-25T19:51:25.634 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2023-12-25T19:51:26.189 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2023-12-25T19:51:26.785 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : kernel-core-4.18.0-529.el8.x86_64 1/3 2023-12-25T19:51:28.425 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2023-12-25T19:51:28.939 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : kernel-core-4.18.0-529.el8.x86_64 1/3 2023-12-25T19:51:30.573 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: kernel-core-4.18.0-529.el8.x86_64 1/3 2023-12-25T19:51:30.959 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : kernel-modules-4.18.0-529.el8.x86_64 2/3 2023-12-25T19:51:32.522 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: kernel-core-4.18.0-529.el8.x86_64 1/3 2023-12-25T19:51:32.889 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : kernel-modules-4.18.0-529.el8.x86_64 2/3 2023-12-25T19:51:34.635 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: kernel-modules-4.18.0-529.el8.x86_64 2/3 2023-12-25T19:51:34.953 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : kernel-4.18.0-529.el8.x86_64 3/3 2023-12-25T19:51:36.337 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: kernel-modules-4.18.0-529.el8.x86_64 2/3 2023-12-25T19:51:36.628 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : kernel-4.18.0-529.el8.x86_64 3/3 2023-12-25T19:51:58.573 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: kernel-core-4.18.0-529.el8.x86_64 3/3 2023-12-25T19:51:58.597 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: kernel-modules-4.18.0-529.el8.x86_64 3/3 2023-12-25T19:51:58.962 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: kernel-4.18.0-529.el8.x86_64 3/3 2023-12-25T19:51:58.962 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : kernel-4.18.0-529.el8.x86_64 1/3 2023-12-25T19:51:58.962 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : kernel-core-4.18.0-529.el8.x86_64 2/3 2023-12-25T19:51:58.963 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: kernel-core-4.18.0-529.el8.x86_64 3/3 2023-12-25T19:51:58.990 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: kernel-modules-4.18.0-529.el8.x86_64 3/3 2023-12-25T19:51:59.458 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: kernel-4.18.0-529.el8.x86_64 3/3 2023-12-25T19:51:59.458 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : kernel-4.18.0-529.el8.x86_64 1/3 2023-12-25T19:51:59.458 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : kernel-core-4.18.0-529.el8.x86_64 2/3 2023-12-25T19:51:59.898 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : kernel-modules-4.18.0-529.el8.x86_64 3/3 2023-12-25T19:51:59.899 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-25T19:51:59.899 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2023-12-25T19:51:59.899 INFO:teuthology.orchestra.run.smithi050.stdout: kernel-4.18.0-529.el8.x86_64 kernel-core-4.18.0-529.el8.x86_64 2023-12-25T19:51:59.900 INFO:teuthology.orchestra.run.smithi050.stdout: kernel-modules-4.18.0-529.el8.x86_64 2023-12-25T19:51:59.900 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-25T19:51:59.900 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-12-25T19:52:00.043 DEBUG:teuthology.orchestra.run.smithi050:> rpm -q kernel | sort -rV | head -n 1 2023-12-25T19:52:00.265 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : kernel-modules-4.18.0-529.el8.x86_64 3/3 2023-12-25T19:52:00.265 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T19:52:00.265 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2023-12-25T19:52:00.266 INFO:teuthology.orchestra.run.smithi160.stdout: kernel-4.18.0-529.el8.x86_64 kernel-core-4.18.0-529.el8.x86_64 2023-12-25T19:52:00.266 INFO:teuthology.orchestra.run.smithi160.stdout: kernel-modules-4.18.0-529.el8.x86_64 2023-12-25T19:52:00.266 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T19:52:00.266 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2023-12-25T19:52:00.273 INFO:teuthology.orchestra.run.smithi050.stdout:kernel-4.18.0-529.el8.x86_64 2023-12-25T19:52:00.273 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-529.el8.x86_64 2023-12-25T19:52:00.273 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-529.el8.x86_64 2023-12-25T19:52:00.273 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-12-25T19:52:00.273 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-12-25T19:52:00.273 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2023-12-25T19:52:00.273 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi050.front.sepia.ceph.com, path=None, version=distro) 2023-12-25T19:52:00.274 DEBUG:teuthology.orchestra.run.smithi050:> rpm -q kernel | sort -rV | head -n 1 2023-12-25T19:52:00.444 DEBUG:teuthology.orchestra.run.smithi160:> rpm -q kernel | sort -rV | head -n 1 2023-12-25T19:52:00.493 INFO:teuthology.orchestra.run.smithi050.stdout:kernel-4.18.0-529.el8.x86_64 2023-12-25T19:52:00.493 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-529.el8.x86_64 2023-12-25T19:52:00.494 DEBUG:teuthology.orchestra.run.smithi050:> sudo rpm -qi grub2-tools 2023-12-25T19:52:00.529 INFO:teuthology.orchestra.run.smithi050.stdout:Name : grub2-tools 2023-12-25T19:52:00.529 INFO:teuthology.orchestra.run.smithi050.stdout:Epoch : 1 2023-12-25T19:52:00.529 INFO:teuthology.orchestra.run.smithi050.stdout:Version : 2.02 2023-12-25T19:52:00.529 INFO:teuthology.orchestra.run.smithi050.stdout:Release : 129.el8 2023-12-25T19:52:00.529 INFO:teuthology.orchestra.run.smithi050.stdout:Architecture: x86_64 2023-12-25T19:52:00.529 INFO:teuthology.orchestra.run.smithi050.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2023-12-25T19:52:00.529 INFO:teuthology.orchestra.run.smithi050.stdout:Group : System Environment/Base 2023-12-25T19:52:00.529 INFO:teuthology.orchestra.run.smithi050.stdout:Size : 9265586 2023-12-25T19:52:00.529 INFO:teuthology.orchestra.run.smithi050.stdout:License : GPLv3+ 2023-12-25T19:52:00.529 INFO:teuthology.orchestra.run.smithi050.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-12-25T19:52:00.529 INFO:teuthology.orchestra.run.smithi050.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-12-25T19:52:00.530 INFO:teuthology.orchestra.run.smithi050.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-12-25T19:52:00.530 INFO:teuthology.orchestra.run.smithi050.stdout:Build Host : kbuilder.bsys.centos.org 2023-12-25T19:52:00.530 INFO:teuthology.orchestra.run.smithi050.stdout:Relocations : (not relocatable) 2023-12-25T19:52:00.530 INFO:teuthology.orchestra.run.smithi050.stdout:Packager : CentOS BuildSystem 2023-12-25T19:52:00.530 INFO:teuthology.orchestra.run.smithi050.stdout:Vendor : CentOS 2023-12-25T19:52:00.530 INFO:teuthology.orchestra.run.smithi050.stdout:URL : http://www.gnu.org/software/grub/ 2023-12-25T19:52:00.530 INFO:teuthology.orchestra.run.smithi050.stdout:Summary : Support tools for GRUB. 2023-12-25T19:52:00.530 INFO:teuthology.orchestra.run.smithi050.stdout:Description : 2023-12-25T19:52:00.530 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-25T19:52:00.530 INFO:teuthology.orchestra.run.smithi050.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-12-25T19:52:00.530 INFO:teuthology.orchestra.run.smithi050.stdout:customizable bootloader with modular architecture. It supports a rich 2023-12-25T19:52:00.530 INFO:teuthology.orchestra.run.smithi050.stdout:variety of kernel formats, file systems, computer architectures and 2023-12-25T19:52:00.530 INFO:teuthology.orchestra.run.smithi050.stdout:hardware devices. 2023-12-25T19:52:00.530 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-25T19:52:00.530 INFO:teuthology.orchestra.run.smithi050.stdout:This subpackage provides tools for support of all platforms. 2023-12-25T19:52:00.531 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-12-25T19:52:00.531 INFO:teuthology.task.kernel:Updating grub on smithi050 to boot 4.18.0-529.el8.x86_64 2023-12-25T19:52:00.532 DEBUG:teuthology.orchestra.run.smithi050:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-12-25T19:52:00.664 INFO:teuthology.orchestra.run.smithi160.stdout:kernel-4.18.0-529.el8.x86_64 2023-12-25T19:52:00.664 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-529.el8.x86_64 2023-12-25T19:52:00.664 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-529.el8.x86_64 2023-12-25T19:52:00.665 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-12-25T19:52:00.665 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-12-25T19:52:00.665 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2023-12-25T19:52:00.665 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi160.front.sepia.ceph.com, path=None, version=distro) 2023-12-25T19:52:00.665 DEBUG:teuthology.orchestra.run.smithi160:> rpm -q kernel | sort -rV | head -n 1 2023-12-25T19:52:00.875 INFO:teuthology.orchestra.run.smithi160.stdout:kernel-4.18.0-529.el8.x86_64 2023-12-25T19:52:00.875 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-529.el8.x86_64 2023-12-25T19:52:00.875 DEBUG:teuthology.orchestra.run.smithi160:> sudo rpm -qi grub2-tools 2023-12-25T19:52:00.909 INFO:teuthology.orchestra.run.smithi160.stdout:Name : grub2-tools 2023-12-25T19:52:00.909 INFO:teuthology.orchestra.run.smithi160.stdout:Epoch : 1 2023-12-25T19:52:00.909 INFO:teuthology.orchestra.run.smithi160.stdout:Version : 2.02 2023-12-25T19:52:00.909 INFO:teuthology.orchestra.run.smithi160.stdout:Release : 129.el8 2023-12-25T19:52:00.909 INFO:teuthology.orchestra.run.smithi160.stdout:Architecture: x86_64 2023-12-25T19:52:00.909 INFO:teuthology.orchestra.run.smithi160.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2023-12-25T19:52:00.909 INFO:teuthology.orchestra.run.smithi160.stdout:Group : System Environment/Base 2023-12-25T19:52:00.909 INFO:teuthology.orchestra.run.smithi160.stdout:Size : 9265586 2023-12-25T19:52:00.909 INFO:teuthology.orchestra.run.smithi160.stdout:License : GPLv3+ 2023-12-25T19:52:00.910 INFO:teuthology.orchestra.run.smithi160.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-12-25T19:52:00.910 INFO:teuthology.orchestra.run.smithi160.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-12-25T19:52:00.910 INFO:teuthology.orchestra.run.smithi160.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-12-25T19:52:00.910 INFO:teuthology.orchestra.run.smithi160.stdout:Build Host : kbuilder.bsys.centos.org 2023-12-25T19:52:00.910 INFO:teuthology.orchestra.run.smithi160.stdout:Relocations : (not relocatable) 2023-12-25T19:52:00.910 INFO:teuthology.orchestra.run.smithi160.stdout:Packager : CentOS BuildSystem 2023-12-25T19:52:00.910 INFO:teuthology.orchestra.run.smithi160.stdout:Vendor : CentOS 2023-12-25T19:52:00.910 INFO:teuthology.orchestra.run.smithi160.stdout:URL : http://www.gnu.org/software/grub/ 2023-12-25T19:52:00.910 INFO:teuthology.orchestra.run.smithi160.stdout:Summary : Support tools for GRUB. 2023-12-25T19:52:00.910 INFO:teuthology.orchestra.run.smithi160.stdout:Description : 2023-12-25T19:52:00.910 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T19:52:00.910 INFO:teuthology.orchestra.run.smithi160.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-12-25T19:52:00.910 INFO:teuthology.orchestra.run.smithi160.stdout:customizable bootloader with modular architecture. It supports a rich 2023-12-25T19:52:00.910 INFO:teuthology.orchestra.run.smithi160.stdout:variety of kernel formats, file systems, computer architectures and 2023-12-25T19:52:00.911 INFO:teuthology.orchestra.run.smithi160.stdout:hardware devices. 2023-12-25T19:52:00.911 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T19:52:00.911 INFO:teuthology.orchestra.run.smithi160.stdout:This subpackage provides tools for support of all platforms. 2023-12-25T19:52:00.911 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-12-25T19:52:00.912 INFO:teuthology.task.kernel:Updating grub on smithi160 to boot 4.18.0-529.el8.x86_64 2023-12-25T19:52:00.912 DEBUG:teuthology.orchestra.run.smithi160:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-12-25T19:52:01.909 INFO:teuthology.orchestra.run.smithi050.stderr:Generating grub configuration file ... 2023-12-25T19:52:02.267 INFO:teuthology.orchestra.run.smithi160.stderr:Generating grub configuration file ... 2023-12-25T19:52:04.241 INFO:teuthology.orchestra.run.smithi050.stderr:done 2023-12-25T19:52:04.242 DEBUG:teuthology.orchestra.run.smithi050:> mktemp 2023-12-25T19:52:04.268 INFO:teuthology.orchestra.run.smithi050.stdout:/tmp/tmp.QnY05xn3dT 2023-12-25T19:52:04.268 DEBUG:teuthology.orchestra.run.smithi050:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.QnY05xn3dT 2023-12-25T19:52:04.341 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod 0666 /tmp/tmp.QnY05xn3dT 2023-12-25T19:52:04.495 DEBUG:teuthology.orchestra.remote:smithi050:/tmp/tmp.QnY05xn3dT is 6KB 2023-12-25T19:52:04.543 DEBUG:teuthology.orchestra.run.smithi050:> rm -fr /tmp/tmp.QnY05xn3dT 2023-12-25T19:52:04.561 DEBUG:teuthology.orchestra.run.smithi050:> sudo /bin/ls /boot/loader/entries || true 2023-12-25T19:52:04.632 INFO:teuthology.orchestra.run.smithi050.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2023-12-25T19:52:04.632 INFO:teuthology.orchestra.run.smithi050.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2023-12-25T19:52:04.632 INFO:teuthology.orchestra.run.smithi050.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2023-12-25T19:52:04.632 INFO:teuthology.orchestra.run.smithi050.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-529.el8.x86_64.conf 2023-12-25T19:52:04.634 DEBUG:teuthology.orchestra.run.smithi050:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-529.el8.x86_64 2023-12-25T19:52:04.757 INFO:teuthology.orchestra.run.smithi160.stderr:done 2023-12-25T19:52:04.759 DEBUG:teuthology.orchestra.run.smithi160:> mktemp 2023-12-25T19:52:04.774 DEBUG:teuthology.orchestra.run.smithi050:> sudo shutdown -r now 2023-12-25T19:52:04.799 INFO:teuthology.orchestra.run.smithi160.stdout:/tmp/tmp.Z8JOU9eZJ6 2023-12-25T19:52:04.799 DEBUG:teuthology.orchestra.run.smithi160:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.Z8JOU9eZJ6 2023-12-25T19:52:04.875 DEBUG:teuthology.orchestra.run.smithi160:> sudo chmod 0666 /tmp/tmp.Z8JOU9eZJ6 2023-12-25T19:52:05.032 DEBUG:teuthology.orchestra.remote:smithi160:/tmp/tmp.Z8JOU9eZJ6 is 6KB 2023-12-25T19:52:05.080 DEBUG:teuthology.orchestra.run.smithi160:> rm -fr /tmp/tmp.Z8JOU9eZJ6 2023-12-25T19:52:05.098 DEBUG:teuthology.orchestra.run.smithi160:> sudo /bin/ls /boot/loader/entries || true 2023-12-25T19:52:05.171 INFO:teuthology.orchestra.run.smithi160.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2023-12-25T19:52:05.171 INFO:teuthology.orchestra.run.smithi160.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2023-12-25T19:52:05.171 INFO:teuthology.orchestra.run.smithi160.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2023-12-25T19:52:05.171 INFO:teuthology.orchestra.run.smithi160.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-529.el8.x86_64.conf 2023-12-25T19:52:05.172 DEBUG:teuthology.orchestra.run.smithi160:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-529.el8.x86_64 2023-12-25T19:52:05.335 DEBUG:teuthology.orchestra.run.smithi160:> sudo shutdown -r now 2023-12-25T19:52:34.805 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2023-12-25T19:52:34.806 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi050.front.sepia.ceph.com' 2023-12-25T19:52:34.807 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi050.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-25T19:52:35.339 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2023-12-25T19:52:35.339 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi160.front.sepia.ceph.com' 2023-12-25T19:52:35.340 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-25T19:53:09.910 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.160 2023-12-25T19:53:18.917 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi160.front.sepia.ceph.com' 2023-12-25T19:53:18.918 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-25T19:53:34.822 DEBUG:teuthology.orchestra.remote:timed out 2023-12-25T19:53:37.297 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.160 2023-12-25T19:53:43.829 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi050.front.sepia.ceph.com' 2023-12-25T19:53:43.830 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi050.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-25T19:53:44.233 DEBUG:teuthology.orchestra.run.smithi050:> true 2023-12-25T19:53:44.509 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi050.front.sepia.ceph.com' 2023-12-25T19:53:44.509 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "4.18.0-529.el8.x86_64"... 2023-12-25T19:53:44.510 DEBUG:teuthology.orchestra.run.smithi050:> uname -r 2023-12-25T19:53:44.526 INFO:teuthology.orchestra.run.smithi050.stdout:4.18.0-529.el8.x86_64 2023-12-25T19:53:44.527 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-529.el8.x86_64 vs 4.18.0-529.el8.x86_64 2023-12-25T19:53:44.527 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-12-25T19:53:44.527 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-12-25T19:53:45.528 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2023-12-25T19:53:45.528 DEBUG:teuthology.orchestra.run.smithi050:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-12-25T19:53:45.726 INFO:teuthology.orchestra.run.smithi050.stdout:ttyS1 2023-12-25T19:53:45.743 DEBUG:teuthology.parallel:result is None 2023-12-25T19:53:49.298 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi160.front.sepia.ceph.com' 2023-12-25T19:53:49.299 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-25T19:53:51.496 DEBUG:teuthology.orchestra.run.smithi160:> true 2023-12-25T19:53:51.828 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi160.front.sepia.ceph.com' 2023-12-25T19:53:51.829 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "4.18.0-529.el8.x86_64"... 2023-12-25T19:53:51.829 DEBUG:teuthology.orchestra.run.smithi160:> uname -r 2023-12-25T19:53:51.849 INFO:teuthology.orchestra.run.smithi160.stdout:4.18.0-529.el8.x86_64 2023-12-25T19:53:51.850 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-529.el8.x86_64 vs 4.18.0-529.el8.x86_64 2023-12-25T19:53:51.850 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-12-25T19:53:51.850 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-12-25T19:53:52.851 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2023-12-25T19:53:52.851 DEBUG:teuthology.orchestra.run.smithi160:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-12-25T19:53:52.960 INFO:teuthology.orchestra.run.smithi160.stdout:ttyS1 2023-12-25T19:53:52.980 DEBUG:teuthology.parallel:result is None 2023-12-25T19:53:52.980 INFO:teuthology.run_tasks:Running task internal.base... 2023-12-25T19:53:52.990 INFO:teuthology.task.internal:Creating test directory... 2023-12-25T19:53:52.991 DEBUG:teuthology.orchestra.run.smithi050:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-12-25T19:53:52.995 DEBUG:teuthology.orchestra.run.smithi160:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-12-25T19:53:53.017 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-12-25T19:53:53.023 INFO:teuthology.run_tasks:Running task internal.archive... 2023-12-25T19:53:53.030 INFO:teuthology.task.internal:Creating archive directory... 2023-12-25T19:53:53.030 DEBUG:teuthology.orchestra.run.smithi050:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-12-25T19:53:53.059 DEBUG:teuthology.orchestra.run.smithi160:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-12-25T19:53:53.110 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-12-25T19:53:53.117 INFO:teuthology.task.internal:Enabling coredump saving... 2023-12-25T19:53:53.117 DEBUG:teuthology.orchestra.run.smithi050:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-12-25T19:53:53.120 DEBUG:teuthology.orchestra.run.smithi160:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-12-25T19:53:53.162 INFO:teuthology.orchestra.run.smithi050.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-25T19:53:53.168 INFO:teuthology.orchestra.run.smithi160.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-25T19:53:53.173 INFO:teuthology.orchestra.run.smithi050.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-25T19:53:53.177 INFO:teuthology.orchestra.run.smithi160.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-25T19:53:53.179 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-12-25T19:53:53.189 INFO:teuthology.task.internal:Configuring sudo... 2023-12-25T19:53:53.191 DEBUG:teuthology.orchestra.run.smithi050:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-12-25T19:53:53.194 DEBUG:teuthology.orchestra.run.smithi160:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-12-25T19:53:53.247 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-12-25T19:53:53.257 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-12-25T19:53:53.257 DEBUG:teuthology.orchestra.run.smithi050:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-12-25T19:53:53.264 DEBUG:teuthology.orchestra.run.smithi160:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-12-25T19:53:53.306 DEBUG:teuthology.orchestra.run.smithi050:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-25T19:53:53.371 DEBUG:teuthology.orchestra.run.smithi050:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-25T19:53:53.425 DEBUG:teuthology.orchestra.run.smithi050:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-25T19:53:53.511 DEBUG:teuthology.orchestra.run.smithi050:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-25T19:53:53.554 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2023-12-25T19:53:53.554 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-12-25T19:53:53.623 DEBUG:teuthology.orchestra.run.smithi160:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-25T19:53:53.664 DEBUG:teuthology.orchestra.run.smithi160:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-25T19:53:53.712 DEBUG:teuthology.orchestra.run.smithi160:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-25T19:53:53.796 DEBUG:teuthology.orchestra.run.smithi160:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-25T19:53:53.839 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2023-12-25T19:53:53.839 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-12-25T19:53:53.907 DEBUG:teuthology.orchestra.run.smithi050:> sudo service rsyslog restart 2023-12-25T19:53:53.909 DEBUG:teuthology.orchestra.run.smithi160:> sudo service rsyslog restart 2023-12-25T19:53:53.978 INFO:teuthology.orchestra.run.smithi050.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-25T19:53:54.028 INFO:teuthology.orchestra.run.smithi160.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-25T19:53:54.229 INFO:teuthology.run_tasks:Running task internal.timer... 2023-12-25T19:53:54.237 INFO:teuthology.task.internal:Starting timer... 2023-12-25T19:53:54.237 INFO:teuthology.run_tasks:Running task pcp... 2023-12-25T19:53:54.247 INFO:teuthology.run_tasks:Running task selinux... 2023-12-25T19:53:54.256 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-12-25T19:53:54.256 DEBUG:teuthology.orchestra.run.smithi050:> sudo service auditd rotate 2023-12-25T19:53:54.356 INFO:teuthology.orchestra.run.smithi050.stdout:Rotating logs: 2023-12-25T19:53:54.357 DEBUG:teuthology.orchestra.run.smithi160:> sudo service auditd rotate 2023-12-25T19:53:54.422 INFO:teuthology.orchestra.run.smithi160.stdout:Rotating logs: 2023-12-25T19:53:54.424 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-12-25T19:53:54.425 DEBUG:teuthology.orchestra.run.smithi050:> /usr/sbin/getenforce 2023-12-25T19:53:54.455 INFO:teuthology.orchestra.run.smithi050.stdout:Permissive 2023-12-25T19:53:54.456 DEBUG:teuthology.orchestra.run.smithi160:> /usr/sbin/getenforce 2023-12-25T19:53:54.476 INFO:teuthology.orchestra.run.smithi160.stdout:Permissive 2023-12-25T19:53:54.476 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi050.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi160.front.sepia.ceph.com': 'permissive'} 2023-12-25T19:53:54.476 DEBUG:teuthology.orchestra.run.smithi050:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-25T19:53:54.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-25T19:53:54.524 DEBUG:teuthology.orchestra.run.smithi160:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-25T19:53:54.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-25T19:53:54.551 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-12-25T19:53:54.551 DEBUG:teuthology.orchestra.run.smithi050:> sudo /usr/sbin/setenforce permissive 2023-12-25T19:53:54.592 DEBUG:teuthology.orchestra.run.smithi160:> sudo /usr/sbin/setenforce permissive 2023-12-25T19:53:54.619 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-12-25T19:53:54.629 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-12-25T19:53:54.649 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-12-25T19:53:54.650 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi050.front.sepia.ceph.com,smithi160.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-12-25T20:01:47.787 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi050.front.sepia.ceph.com'), Remote(name='ubuntu@smithi160.front.sepia.ceph.com')] 2023-12-25T20:01:47.788 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi050.front.sepia.ceph.com' 2023-12-25T20:01:47.789 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi050.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-25T20:01:47.859 DEBUG:teuthology.orchestra.run.smithi050:> true 2023-12-25T20:01:47.887 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi050.front.sepia.ceph.com' 2023-12-25T20:01:47.887 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi160.front.sepia.ceph.com' 2023-12-25T20:01:47.888 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-25T20:01:47.955 DEBUG:teuthology.orchestra.run.smithi160:> true 2023-12-25T20:01:48.025 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi160.front.sepia.ceph.com' 2023-12-25T20:01:48.025 INFO:teuthology.run_tasks:Running task clock... 2023-12-25T20:01:48.036 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-12-25T20:01:48.036 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-12-25T20:01:48.037 DEBUG:teuthology.orchestra.run.smithi050:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-25T20:01:48.039 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-12-25T20:01:48.039 DEBUG:teuthology.orchestra.run.smithi160:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-25T20:01:48.071 INFO:teuthology.orchestra.run.smithi050.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-12-25T20:01:48.086 INFO:teuthology.orchestra.run.smithi050.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-12-25T20:01:48.099 INFO:teuthology.orchestra.run.smithi160.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-12-25T20:01:48.114 INFO:teuthology.orchestra.run.smithi160.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-12-25T20:01:48.149 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: ntpd: command not found 2023-12-25T20:01:48.161 INFO:teuthology.orchestra.run.smithi050.stdout:506 Cannot talk to daemon 2023-12-25T20:01:48.175 INFO:teuthology.orchestra.run.smithi050.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-12-25T20:01:48.182 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: ntpd: command not found 2023-12-25T20:01:48.189 INFO:teuthology.orchestra.run.smithi050.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-12-25T20:01:48.193 INFO:teuthology.orchestra.run.smithi160.stdout:506 Cannot talk to daemon 2023-12-25T20:01:48.207 INFO:teuthology.orchestra.run.smithi160.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-12-25T20:01:48.221 INFO:teuthology.orchestra.run.smithi160.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-12-25T20:01:48.262 INFO:teuthology.orchestra.run.smithi050.stderr:bash: ntpq: command not found 2023-12-25T20:01:48.266 INFO:teuthology.orchestra.run.smithi050.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-25T20:01:48.267 INFO:teuthology.orchestra.run.smithi050.stdout:=============================================================================== 2023-12-25T20:01:48.267 INFO:teuthology.orchestra.run.smithi050.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-25T20:01:48.267 INFO:teuthology.orchestra.run.smithi050.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-25T20:01:48.267 INFO:teuthology.orchestra.run.smithi050.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-25T20:01:48.267 INFO:teuthology.orchestra.run.smithi050.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-25T20:01:48.305 INFO:teuthology.orchestra.run.smithi160.stderr:bash: ntpq: command not found 2023-12-25T20:01:48.318 INFO:teuthology.orchestra.run.smithi160.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-25T20:01:48.318 INFO:teuthology.orchestra.run.smithi160.stdout:=============================================================================== 2023-12-25T20:01:48.319 INFO:teuthology.orchestra.run.smithi160.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-25T20:01:48.319 INFO:teuthology.orchestra.run.smithi160.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-25T20:01:48.319 INFO:teuthology.orchestra.run.smithi160.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-25T20:01:48.319 INFO:teuthology.orchestra.run.smithi160.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-25T20:01:48.319 INFO:teuthology.run_tasks:Running task pexec... 2023-12-25T20:01:48.424 INFO:teuthology.task.pexec:Executing custom commands... 2023-12-25T20:01:48.425 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi050.front.sepia.ceph.com 2023-12-25T20:01:48.425 DEBUG:teuthology.orchestra.run.smithi050:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-12-25T20:01:48.426 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi160.front.sepia.ceph.com 2023-12-25T20:01:48.426 DEBUG:teuthology.orchestra.run.smithi160:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-12-25T20:01:49.304 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:02:00 ago on Mon 25 Dec 2023 07:59:49 PM UTC. 2023-12-25T20:01:49.337 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:49 ago on Mon 25 Dec 2023 08:00:00 PM UTC. 2023-12-25T20:01:51.480 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2023-12-25T20:01:51.481 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-12-25T20:01:51.481 INFO:teuthology.orchestra.run.smithi160.stdout: Package Architecture Version Repository Size 2023-12-25T20:01:51.481 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-12-25T20:01:51.481 INFO:teuthology.orchestra.run.smithi160.stdout:Resetting modules: 2023-12-25T20:01:51.481 INFO:teuthology.orchestra.run.smithi160.stdout: container-tools 2023-12-25T20:01:51.481 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:01:51.481 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2023-12-25T20:01:51.481 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-12-25T20:01:51.482 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:01:51.647 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-12-25T20:01:51.649 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-12-25T20:01:51.649 INFO:teuthology.orchestra.run.smithi050.stdout: Package Architecture Version Repository Size 2023-12-25T20:01:51.649 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-12-25T20:01:51.649 INFO:teuthology.orchestra.run.smithi050.stdout:Resetting modules: 2023-12-25T20:01:51.649 INFO:teuthology.orchestra.run.smithi050.stdout: container-tools 2023-12-25T20:01:51.649 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-25T20:01:51.649 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2023-12-25T20:01:51.649 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-12-25T20:01:51.649 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-25T20:01:51.735 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2023-12-25T20:01:51.898 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-12-25T20:01:52.665 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:02:03 ago on Mon 25 Dec 2023 07:59:49 PM UTC. 2023-12-25T20:01:52.860 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:52 ago on Mon 25 Dec 2023 08:00:00 PM UTC. 2023-12-25T20:01:54.807 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2023-12-25T20:01:54.995 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-12-25T20:01:54.995 INFO:teuthology.orchestra.run.smithi160.stdout: Package Arch Version Repository Size 2023-12-25T20:01:54.995 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-12-25T20:01:54.995 INFO:teuthology.orchestra.run.smithi160.stdout:Installing group/module packages: 2023-12-25T20:01:54.995 INFO:teuthology.orchestra.run.smithi160.stdout: aardvark-dns x86_64 2:1.8.0-1.module_el8+712+4cd1bd69 appstream 1.0 M 2023-12-25T20:01:54.995 INFO:teuthology.orchestra.run.smithi160.stdout: buildah x86_64 1:1.32.0-1.module_el8+712+4cd1bd69 appstream 9.6 M 2023-12-25T20:01:54.995 INFO:teuthology.orchestra.run.smithi160.stdout: cockpit-podman noarch 78-1.module_el8+712+4cd1bd69 appstream 750 k 2023-12-25T20:01:54.995 INFO:teuthology.orchestra.run.smithi160.stdout: crun x86_64 1.9.2-1.module_el8+712+4cd1bd69 appstream 241 k 2023-12-25T20:01:54.996 INFO:teuthology.orchestra.run.smithi160.stdout: netavark x86_64 2:1.8.0-2.module_el8+712+4cd1bd69 appstream 3.9 M 2023-12-25T20:01:54.996 INFO:teuthology.orchestra.run.smithi160.stdout: python3-podman noarch 4.7.0-1.module_el8+712+4cd1bd69 appstream 178 k 2023-12-25T20:01:54.996 INFO:teuthology.orchestra.run.smithi160.stdout: skopeo x86_64 2:1.13.3-1.module_el8+712+4cd1bd69 appstream 8.1 M 2023-12-25T20:01:54.996 INFO:teuthology.orchestra.run.smithi160.stdout: toolbox x86_64 0.0.99.4-6.module_el8+712+4cd1bd69 appstream 2.5 M 2023-12-25T20:01:54.996 INFO:teuthology.orchestra.run.smithi160.stdout: udica noarch 0.2.6-20.module_el8+712+4cd1bd69 appstream 53 k 2023-12-25T20:01:54.996 INFO:teuthology.orchestra.run.smithi160.stdout:Installing dependencies: 2023-12-25T20:01:54.996 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2023-12-25T20:01:54.996 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2023-12-25T20:01:54.996 INFO:teuthology.orchestra.run.smithi160.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2023-12-25T20:01:54.996 INFO:teuthology.orchestra.run.smithi160.stdout:Installing weak dependencies: 2023-12-25T20:01:54.996 INFO:teuthology.orchestra.run.smithi160.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2023-12-25T20:01:54.996 INFO:teuthology.orchestra.run.smithi160.stdout:Installing module profiles: 2023-12-25T20:01:54.996 INFO:teuthology.orchestra.run.smithi160.stdout: container-tools/common 2023-12-25T20:01:54.996 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:01:54.996 INFO:teuthology.orchestra.run.smithi160.stdout:Enabling module streams: 2023-12-25T20:01:54.997 INFO:teuthology.orchestra.run.smithi160.stdout: container-tools rhel8 2023-12-25T20:01:54.997 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:01:54.997 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2023-12-25T20:01:54.997 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-12-25T20:01:54.997 INFO:teuthology.orchestra.run.smithi160.stdout:Install 13 Packages 2023-12-25T20:01:54.997 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:01:54.997 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 27 M 2023-12-25T20:01:54.998 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 89 M 2023-12-25T20:01:54.998 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2023-12-25T20:01:55.142 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-12-25T20:01:55.347 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-12-25T20:01:55.347 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2023-12-25T20:01:55.347 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-12-25T20:01:55.347 INFO:teuthology.orchestra.run.smithi050.stdout:Installing group/module packages: 2023-12-25T20:01:55.347 INFO:teuthology.orchestra.run.smithi050.stdout: aardvark-dns x86_64 2:1.8.0-1.module_el8+712+4cd1bd69 appstream 1.0 M 2023-12-25T20:01:55.347 INFO:teuthology.orchestra.run.smithi050.stdout: buildah x86_64 1:1.32.0-1.module_el8+712+4cd1bd69 appstream 9.6 M 2023-12-25T20:01:55.347 INFO:teuthology.orchestra.run.smithi050.stdout: cockpit-podman noarch 78-1.module_el8+712+4cd1bd69 appstream 750 k 2023-12-25T20:01:55.348 INFO:teuthology.orchestra.run.smithi050.stdout: crun x86_64 1.9.2-1.module_el8+712+4cd1bd69 appstream 241 k 2023-12-25T20:01:55.348 INFO:teuthology.orchestra.run.smithi050.stdout: netavark x86_64 2:1.8.0-2.module_el8+712+4cd1bd69 appstream 3.9 M 2023-12-25T20:01:55.348 INFO:teuthology.orchestra.run.smithi050.stdout: python3-podman noarch 4.7.0-1.module_el8+712+4cd1bd69 appstream 178 k 2023-12-25T20:01:55.348 INFO:teuthology.orchestra.run.smithi050.stdout: skopeo x86_64 2:1.13.3-1.module_el8+712+4cd1bd69 appstream 8.1 M 2023-12-25T20:01:55.348 INFO:teuthology.orchestra.run.smithi050.stdout: toolbox x86_64 0.0.99.4-6.module_el8+712+4cd1bd69 appstream 2.5 M 2023-12-25T20:01:55.348 INFO:teuthology.orchestra.run.smithi050.stdout: udica noarch 0.2.6-20.module_el8+712+4cd1bd69 appstream 53 k 2023-12-25T20:01:55.348 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2023-12-25T20:01:55.348 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2023-12-25T20:01:55.348 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2023-12-25T20:01:55.348 INFO:teuthology.orchestra.run.smithi050.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2023-12-25T20:01:55.348 INFO:teuthology.orchestra.run.smithi050.stdout:Installing weak dependencies: 2023-12-25T20:01:55.348 INFO:teuthology.orchestra.run.smithi050.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2023-12-25T20:01:55.348 INFO:teuthology.orchestra.run.smithi050.stdout:Installing module profiles: 2023-12-25T20:01:55.348 INFO:teuthology.orchestra.run.smithi050.stdout: container-tools/common 2023-12-25T20:01:55.348 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-25T20:01:55.349 INFO:teuthology.orchestra.run.smithi050.stdout:Enabling module streams: 2023-12-25T20:01:55.349 INFO:teuthology.orchestra.run.smithi050.stdout: container-tools rhel8 2023-12-25T20:01:55.349 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-25T20:01:55.349 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2023-12-25T20:01:55.349 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-12-25T20:01:55.349 INFO:teuthology.orchestra.run.smithi050.stdout:Install 13 Packages 2023-12-25T20:01:55.349 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-25T20:01:55.350 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 27 M 2023-12-25T20:01:55.350 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 89 M 2023-12-25T20:01:55.350 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2023-12-25T20:01:55.594 INFO:teuthology.orchestra.run.smithi160.stdout:(1/13): cockpit-podman-78-1.module_el8+712+4cd1 1.3 MB/s | 750 kB 00:00 2023-12-25T20:01:55.628 INFO:teuthology.orchestra.run.smithi160.stdout:(2/13): aardvark-dns-1.8.0-1.module_el8+712+4cd 1.7 MB/s | 1.0 MB 00:00 2023-12-25T20:01:55.695 INFO:teuthology.orchestra.run.smithi160.stdout:(3/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 377 kB/s | 38 kB 00:00 2023-12-25T20:01:55.721 INFO:teuthology.orchestra.run.smithi160.stdout:(4/13): crun-1.9.2-1.module_el8+712+4cd1bd69.x8 2.6 MB/s | 241 kB 00:00 2023-12-25T20:01:55.821 INFO:teuthology.orchestra.run.smithi160.stdout:(5/13): python3-podman-4.7.0-1.module_el8+712+4 1.7 MB/s | 178 kB 00:00 2023-12-25T20:01:55.913 INFO:teuthology.orchestra.run.smithi160.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 270 kB/s | 25 kB 00:00 2023-12-25T20:01:55.932 INFO:teuthology.orchestra.run.smithi050.stdout:(1/13): cockpit-podman-78-1.module_el8+712+4cd1 1.4 MB/s | 750 kB 00:00 2023-12-25T20:01:55.997 INFO:teuthology.orchestra.run.smithi160.stdout:(7/13): netavark-1.8.0-2.module_el8+712+4cd1bd6 13 MB/s | 3.9 MB 00:00 2023-12-25T20:01:56.022 INFO:teuthology.orchestra.run.smithi160.stdout:(8/13): python3-pyxdg-0.25-16.el8.noarch.rpm 865 kB/s | 94 kB 00:00 2023-12-25T20:01:56.032 INFO:teuthology.orchestra.run.smithi050.stdout:(2/13): aardvark-dns-1.8.0-1.module_el8+712+4cd 1.6 MB/s | 1.0 MB 00:00 2023-12-25T20:01:56.057 INFO:teuthology.orchestra.run.smithi050.stdout:(3/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 302 kB/s | 38 kB 00:00 2023-12-25T20:01:56.131 INFO:teuthology.orchestra.run.smithi160.stdout:(9/13): buildah-1.32.0-1.module_el8+712+4cd1bd6 8.8 MB/s | 9.6 MB 00:01 2023-12-25T20:01:56.141 INFO:teuthology.orchestra.run.smithi050.stdout:(4/13): crun-1.9.2-1.module_el8+712+4cd1bd69.x8 2.2 MB/s | 241 kB 00:00 2023-12-25T20:01:56.242 INFO:teuthology.orchestra.run.smithi050.stdout:(5/13): python3-podman-4.7.0-1.module_el8+712+4 1.7 MB/s | 178 kB 00:00 2023-12-25T20:01:56.256 INFO:teuthology.orchestra.run.smithi160.stdout:(10/13): udica-0.2.6-20.module_el8+712+4cd1bd69 421 kB/s | 53 kB 00:00 2023-12-25T20:01:56.364 INFO:teuthology.orchestra.run.smithi160.stdout:(11/13): yajl-2.1.0-12.el8.x86_64.rpm 378 kB/s | 41 kB 00:00 2023-12-25T20:01:56.402 INFO:teuthology.orchestra.run.smithi050.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 155 kB/s | 25 kB 00:00 2023-12-25T20:01:56.414 INFO:teuthology.orchestra.run.smithi160.stdout:(12/13): toolbox-0.0.99.4-6.module_el8+712+4cd1 6.5 MB/s | 2.5 MB 00:00 2023-12-25T20:01:56.476 INFO:teuthology.orchestra.run.smithi050.stdout:(7/13): netavark-1.8.0-2.module_el8+712+4cd1bd6 9.4 MB/s | 3.9 MB 00:00 2023-12-25T20:01:56.493 INFO:teuthology.orchestra.run.smithi050.stdout:(8/13): python3-pyxdg-0.25-16.el8.noarch.rpm 1.0 MB/s | 94 kB 00:00 2023-12-25T20:01:56.610 INFO:teuthology.orchestra.run.smithi050.stdout:(9/13): buildah-1.32.0-1.module_el8+712+4cd1bd6 7.9 MB/s | 9.6 MB 00:01 2023-12-25T20:01:56.673 INFO:teuthology.orchestra.run.smithi160.stdout:(13/13): skopeo-1.13.3-1.module_el8+712+4cd1bd6 12 MB/s | 8.1 MB 00:00 2023-12-25T20:01:56.674 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2023-12-25T20:01:56.674 INFO:teuthology.orchestra.run.smithi160.stdout:Total 16 MB/s | 27 MB 00:01 2023-12-25T20:01:56.743 INFO:teuthology.orchestra.run.smithi050.stdout:(10/13): toolbox-0.0.99.4-6.module_el8+712+4cd1 10 MB/s | 2.5 MB 00:00 2023-12-25T20:01:56.760 INFO:teuthology.orchestra.run.smithi050.stdout:(11/13): udica-0.2.6-20.module_el8+712+4cd1bd69 350 kB/s | 53 kB 00:00 2023-12-25T20:01:56.836 INFO:teuthology.orchestra.run.smithi050.stdout:(12/13): yajl-2.1.0-12.el8.x86_64.rpm 446 kB/s | 41 kB 00:00 2023-12-25T20:01:56.941 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2023-12-25T20:01:56.961 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2023-12-25T20:01:56.961 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2023-12-25T20:01:57.211 INFO:teuthology.orchestra.run.smithi050.stdout:(13/13): skopeo-1.13.3-1.module_el8+712+4cd1bd6 11 MB/s | 8.1 MB 00:00 2023-12-25T20:01:57.211 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2023-12-25T20:01:57.211 INFO:teuthology.orchestra.run.smithi050.stdout:Total 14 MB/s | 27 MB 00:01 2023-12-25T20:01:57.215 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2023-12-25T20:01:57.218 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2023-12-25T20:01:57.491 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2023-12-25T20:01:57.517 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2023-12-25T20:01:57.517 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2023-12-25T20:01:57.783 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2023-12-25T20:01:57.786 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2023-12-25T20:01:58.224 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2023-12-25T20:01:58.468 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2023-12-25T20:01:58.512 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2023-12-25T20:01:58.630 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 2/13 2023-12-25T20:01:58.742 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2023-12-25T20:01:58.829 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/13 2023-12-25T20:01:58.942 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 2/13 2023-12-25T20:01:59.048 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2023-12-25T20:01:59.125 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/13 2023-12-25T20:01:59.382 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2023-12-25T20:01:59.611 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 5/13 2023-12-25T20:02:00.099 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 5/13 2023-12-25T20:02:00.623 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2023-12-25T20:02:00.850 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 7/13 2023-12-25T20:02:01.071 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 8/13 2023-12-25T20:02:01.139 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2023-12-25T20:02:01.334 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 9/13 2023-12-25T20:02:01.437 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 7/13 2023-12-25T20:02:01.671 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 8/13 2023-12-25T20:02:01.708 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 10/13 2023-12-25T20:02:01.914 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 9/13 2023-12-25T20:02:02.335 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 10/13 2023-12-25T20:02:02.581 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2023-12-25T20:02:02.801 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 12/13 2023-12-25T20:02:02.954 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2023-12-25T20:02:03.297 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2023-12-25T20:02:03.524 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 12/13 2023-12-25T20:02:03.595 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2023-12-25T20:02:03.595 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 1/13 2023-12-25T20:02:03.595 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2/13 2023-12-25T20:02:03.596 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 3/13 2023-12-25T20:02:03.596 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2023-12-25T20:02:03.596 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 5/13 2023-12-25T20:02:03.596 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2023-12-25T20:02:03.596 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 7/13 2023-12-25T20:02:03.596 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2023-12-25T20:02:03.596 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2023-12-25T20:02:03.596 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 10/13 2023-12-25T20:02:03.596 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2023-12-25T20:02:03.596 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 12/13 2023-12-25T20:02:03.698 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2023-12-25T20:02:04.176 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2023-12-25T20:02:04.176 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 1/13 2023-12-25T20:02:04.176 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2/13 2023-12-25T20:02:04.176 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 3/13 2023-12-25T20:02:04.176 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2023-12-25T20:02:04.176 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 5/13 2023-12-25T20:02:04.176 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2023-12-25T20:02:04.176 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 7/13 2023-12-25T20:02:04.176 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2023-12-25T20:02:04.177 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2023-12-25T20:02:04.177 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 10/13 2023-12-25T20:02:04.177 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2023-12-25T20:02:04.177 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 12/13 2023-12-25T20:02:04.408 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2023-12-25T20:02:04.408 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:02:04.408 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2023-12-25T20:02:04.408 INFO:teuthology.orchestra.run.smithi160.stdout: aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86_64 2023-12-25T20:02:04.408 INFO:teuthology.orchestra.run.smithi160.stdout: buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2023-12-25T20:02:04.408 INFO:teuthology.orchestra.run.smithi160.stdout: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 2023-12-25T20:02:04.408 INFO:teuthology.orchestra.run.smithi160.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2023-12-25T20:02:04.409 INFO:teuthology.orchestra.run.smithi160.stdout: crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 2023-12-25T20:02:04.409 INFO:teuthology.orchestra.run.smithi160.stdout: netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 2023-12-25T20:02:04.409 INFO:teuthology.orchestra.run.smithi160.stdout: python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noarch 2023-12-25T20:02:04.409 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-12-25T20:02:04.409 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-12-25T20:02:04.409 INFO:teuthology.orchestra.run.smithi160.stdout: skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 2023-12-25T20:02:04.409 INFO:teuthology.orchestra.run.smithi160.stdout: toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 2023-12-25T20:02:04.409 INFO:teuthology.orchestra.run.smithi160.stdout: udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 2023-12-25T20:02:04.409 INFO:teuthology.orchestra.run.smithi160.stdout: yajl-2.1.0-12.el8.x86_64 2023-12-25T20:02:04.409 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:02:04.409 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2023-12-25T20:02:04.710 DEBUG:teuthology.parallel:result is None 2023-12-25T20:02:05.013 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2023-12-25T20:02:05.013 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-25T20:02:05.013 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2023-12-25T20:02:05.013 INFO:teuthology.orchestra.run.smithi050.stdout: aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86_64 2023-12-25T20:02:05.013 INFO:teuthology.orchestra.run.smithi050.stdout: buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2023-12-25T20:02:05.013 INFO:teuthology.orchestra.run.smithi050.stdout: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 2023-12-25T20:02:05.013 INFO:teuthology.orchestra.run.smithi050.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2023-12-25T20:02:05.013 INFO:teuthology.orchestra.run.smithi050.stdout: crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 2023-12-25T20:02:05.014 INFO:teuthology.orchestra.run.smithi050.stdout: netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 2023-12-25T20:02:05.014 INFO:teuthology.orchestra.run.smithi050.stdout: python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noarch 2023-12-25T20:02:05.014 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-12-25T20:02:05.014 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-12-25T20:02:05.014 INFO:teuthology.orchestra.run.smithi050.stdout: skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 2023-12-25T20:02:05.014 INFO:teuthology.orchestra.run.smithi050.stdout: toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 2023-12-25T20:02:05.014 INFO:teuthology.orchestra.run.smithi050.stdout: udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 2023-12-25T20:02:05.014 INFO:teuthology.orchestra.run.smithi050.stdout: yajl-2.1.0-12.el8.x86_64 2023-12-25T20:02:05.014 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-25T20:02:05.014 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-12-25T20:02:05.236 DEBUG:teuthology.parallel:result is None 2023-12-25T20:02:05.236 INFO:teuthology.run_tasks:Running task pexec... 2023-12-25T20:02:05.245 INFO:teuthology.task.pexec:Executing custom commands... 2023-12-25T20:02:05.246 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi050.front.sepia.ceph.com 2023-12-25T20:02:05.246 DEBUG:teuthology.orchestra.run.smithi050:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-12-25T20:02:05.247 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi160.front.sepia.ceph.com 2023-12-25T20:02:05.247 DEBUG:teuthology.orchestra.run.smithi160:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-12-25T20:02:05.409 DEBUG:teuthology.parallel:result is None 2023-12-25T20:02:05.422 DEBUG:teuthology.parallel:result is None 2023-12-25T20:02:05.422 INFO:teuthology.run_tasks:Running task install... 2023-12-25T20:02:05.430 DEBUG:teuthology.task.install:project ceph 2023-12-25T20:02:05.430 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'b14a0a106b8cffae0144ff9fb83d9ce1af1a4bd2'}} 2023-12-25T20:02:05.430 DEBUG:teuthology.task.install:config {'exclude_packages': ['ceph-volume'], 'tag': 'v16.2.4', 'flavor': 'default', 'sha1': 'b14a0a106b8cffae0144ff9fb83d9ce1af1a4bd2'} 2023-12-25T20:02:05.430 INFO:teuthology.task.install:Using flavor: default 2023-12-25T20:02:05.436 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-12-25T20:02:05.436 INFO:teuthology.task.install:extra packages: [] 2023-12-25T20:02:05.436 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'b14a0a106b8cffae0144ff9fb83d9ce1af1a4bd2', 'tag': 'v16.2.4', 'wait_for_package': False} 2023-12-25T20:02:05.437 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2023-12-25T20:02:05.437 INFO:teuthology.packaging:ref: None 2023-12-25T20:02:05.437 INFO:teuthology.packaging:tag: v16.2.4 2023-12-25T20:02:05.437 INFO:teuthology.packaging:branch: None 2023-12-25T20:02:05.437 INFO:teuthology.packaging:sha1: b14a0a106b8cffae0144ff9fb83d9ce1af1a4bd2 2023-12-25T20:02:05.636 DEBUG:teuthology.repo_utils:git ls-remote https://github.com/ceph/ceph-ci v16.2.4^{} -> 3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-12-25T20:02:05.637 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-12-25T20:02:05.639 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'b14a0a106b8cffae0144ff9fb83d9ce1af1a4bd2', 'tag': 'v16.2.4', 'wait_for_package': False} 2023-12-25T20:02:05.640 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2023-12-25T20:02:05.640 INFO:teuthology.packaging:ref: None 2023-12-25T20:02:05.640 INFO:teuthology.packaging:tag: v16.2.4 2023-12-25T20:02:05.640 INFO:teuthology.packaging:branch: None 2023-12-25T20:02:05.640 INFO:teuthology.packaging:sha1: b14a0a106b8cffae0144ff9fb83d9ce1af1a4bd2 2023-12-25T20:02:05.640 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-12-25T20:02:05.806 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/ 2023-12-25T20:02:05.806 INFO:teuthology.task.install.rpm:Package version is 16.2.4-0 2023-12-25T20:02:05.822 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/ 2023-12-25T20:02:05.822 INFO:teuthology.task.install.rpm:Package version is 16.2.4-0 2023-12-25T20:02:05.857 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-12-25T20:02:05.857 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2023-12-25T20:02:05.858 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-12-25T20:02:05.869 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-12-25T20:02:05.869 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2023-12-25T20:02:05.869 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-12-25T20:02:05.887 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-12-25T20:02:05.887 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2023-12-25T20:02:05.888 INFO:teuthology.packaging:ref: None 2023-12-25T20:02:05.888 INFO:teuthology.packaging:tag: v16.2.4 2023-12-25T20:02:05.888 INFO:teuthology.packaging:branch: None 2023-12-25T20:02:05.888 INFO:teuthology.packaging:sha1: b14a0a106b8cffae0144ff9fb83d9ce1af1a4bd2 2023-12-25T20:02:05.888 DEBUG:teuthology.orchestra.run.smithi050:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v16.2.4/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-12-25T20:02:05.898 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-12-25T20:02:05.898 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2023-12-25T20:02:05.898 INFO:teuthology.packaging:ref: None 2023-12-25T20:02:05.898 INFO:teuthology.packaging:tag: v16.2.4 2023-12-25T20:02:05.898 INFO:teuthology.packaging:branch: None 2023-12-25T20:02:05.898 INFO:teuthology.packaging:sha1: b14a0a106b8cffae0144ff9fb83d9ce1af1a4bd2 2023-12-25T20:02:05.898 DEBUG:teuthology.orchestra.run.smithi160:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v16.2.4/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-12-25T20:02:05.964 DEBUG:teuthology.orchestra.run.smithi050:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-12-25T20:02:05.967 DEBUG:teuthology.orchestra.run.smithi160:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-12-25T20:02:06.043 DEBUG:teuthology.orchestra.run.smithi050:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-12-25T20:02:06.045 DEBUG:teuthology.orchestra.run.smithi160:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-12-25T20:02:06.111 INFO:teuthology.orchestra.run.smithi050.stdout:check_obsoletes = 1 2023-12-25T20:02:06.111 INFO:teuthology.orchestra.run.smithi160.stdout:check_obsoletes = 1 2023-12-25T20:02:06.112 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum clean all 2023-12-25T20:02:06.114 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum clean all 2023-12-25T20:02:06.386 INFO:teuthology.orchestra.run.smithi160.stdout:76 files removed 2023-12-25T20:02:06.394 INFO:teuthology.orchestra.run.smithi050.stdout:76 files removed 2023-12-25T20:02:06.414 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install ceph-radosgw 2023-12-25T20:02:06.425 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-radosgw 2023-12-25T20:02:06.890 INFO:teuthology.orchestra.run.smithi160.stdout:ceph packages for x86_64 1.1 MB/s | 74 kB 00:00 2023-12-25T20:02:06.946 INFO:teuthology.orchestra.run.smithi050.stdout:ceph packages for x86_64 831 kB/s | 74 kB 00:00 2023-12-25T20:02:07.007 INFO:teuthology.orchestra.run.smithi160.stdout:ceph noarch packages 191 kB/s | 15 kB 00:00 2023-12-25T20:02:07.071 INFO:teuthology.orchestra.run.smithi050.stdout:ceph noarch packages 180 kB/s | 15 kB 00:00 2023-12-25T20:02:07.107 INFO:teuthology.orchestra.run.smithi160.stdout:ceph source packages 24 kB/s | 1.7 kB 00:00 2023-12-25T20:02:07.172 INFO:teuthology.orchestra.run.smithi050.stdout:ceph source packages 25 kB/s | 1.7 kB 00:00 2023-12-25T20:02:09.092 INFO:teuthology.orchestra.run.smithi160.stdout:CentOS Stream 8 - AppStream 18 MB/s | 35 MB 00:01 2023-12-25T20:02:09.275 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS Stream 8 - AppStream 17 MB/s | 35 MB 00:02 2023-12-25T20:02:33.472 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS Stream 8 - BaseOS 3.3 MB/s | 57 MB 00:17 2023-12-25T20:02:33.831 INFO:teuthology.orchestra.run.smithi160.stdout:CentOS Stream 8 - BaseOS 3.1 MB/s | 57 MB 00:18 2023-12-25T20:02:42.241 INFO:teuthology.orchestra.run.smithi160.stdout:CentOS Stream 8 - Extras 76 kB/s | 18 kB 00:00 2023-12-25T20:02:42.672 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS Stream 8 - Extras 81 kB/s | 18 kB 00:00 2023-12-25T20:02:42.808 INFO:teuthology.orchestra.run.smithi160.stdout:CentOS Stream 8 - Extras common packages 18 kB/s | 6.9 kB 00:00 2023-12-25T20:02:43.431 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS Stream 8 - Extras common packages 12 kB/s | 6.9 kB 00:00 2023-12-25T20:02:43.451 INFO:teuthology.orchestra.run.smithi160.stdout:CentOS Stream 8 - PowerTools 14 MB/s | 6.6 MB 00:00 2023-12-25T20:02:44.232 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS Stream 8 - PowerTools 10 MB/s | 6.6 MB 00:00 2023-12-25T20:02:44.927 INFO:teuthology.orchestra.run.smithi160.stdout:Copr repo for python3-asyncssh owned by ceph 22 kB/s | 3.5 kB 00:00 2023-12-25T20:02:45.817 INFO:teuthology.orchestra.run.smithi050.stdout:Copr repo for python3-asyncssh owned by ceph 22 kB/s | 3.5 kB 00:00 2023-12-25T20:02:46.028 INFO:teuthology.orchestra.run.smithi160.stdout:Extra Packages for Enterprise Linux 17 MB/s | 16 MB 00:00 2023-12-25T20:02:47.754 INFO:teuthology.orchestra.run.smithi050.stdout:Extra Packages for Enterprise Linux 9.0 MB/s | 16 MB 00:01 2023-12-25T20:02:49.903 INFO:teuthology.orchestra.run.smithi160.stdout:lab-extras 264 kB/s | 24 kB 00:00 2023-12-25T20:02:51.901 INFO:teuthology.orchestra.run.smithi050.stdout:lab-extras 401 kB/s | 24 kB 00:00 2023-12-25T20:02:54.600 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2023-12-25T20:02:54.602 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-12-25T20:02:54.603 INFO:teuthology.orchestra.run.smithi160.stdout: Package Arch Version Repository Size 2023-12-25T20:02:54.603 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-12-25T20:02:54.603 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2023-12-25T20:02:54.603 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-radosgw x86_64 2:16.2.4-0.el8 ceph 13 M 2023-12-25T20:02:54.603 INFO:teuthology.orchestra.run.smithi160.stdout:Upgrading: 2023-12-25T20:02:54.603 INFO:teuthology.orchestra.run.smithi160.stdout: librados2 x86_64 2:16.2.4-0.el8 ceph 3.7 M 2023-12-25T20:02:54.603 INFO:teuthology.orchestra.run.smithi160.stdout: librbd1 x86_64 2:16.2.4-0.el8 ceph 4.0 M 2023-12-25T20:02:54.603 INFO:teuthology.orchestra.run.smithi160.stdout:Installing dependencies: 2023-12-25T20:02:54.603 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-base x86_64 2:16.2.4-0.el8 ceph 6.0 M 2023-12-25T20:02:54.603 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-common x86_64 2:16.2.4-0.el8 ceph 24 M 2023-12-25T20:02:54.603 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-selinux x86_64 2:16.2.4-0.el8 ceph 25 k 2023-12-25T20:02:54.603 INFO:teuthology.orchestra.run.smithi160.stdout: daxctl-libs x86_64 71.1-7.el8 baseos 42 k 2023-12-25T20:02:54.603 INFO:teuthology.orchestra.run.smithi160.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2023-12-25T20:02:54.604 INFO:teuthology.orchestra.run.smithi160.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-12-25T20:02:54.604 INFO:teuthology.orchestra.run.smithi160.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2023-12-25T20:02:54.604 INFO:teuthology.orchestra.run.smithi160.stdout: libcephfs2 x86_64 2:16.2.4-0.el8 ceph 801 k 2023-12-25T20:02:54.604 INFO:teuthology.orchestra.run.smithi160.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-12-25T20:02:54.604 INFO:teuthology.orchestra.run.smithi160.stdout: libpmemobj x86_64 1.6.1-1.el8 appstream 145 k 2023-12-25T20:02:54.604 INFO:teuthology.orchestra.run.smithi160.stdout: librabbitmq x86_64 0.9.0-4.el8 baseos 47 k 2023-12-25T20:02:54.604 INFO:teuthology.orchestra.run.smithi160.stdout: libradosstriper1 x86_64 2:16.2.4-0.el8 ceph 486 k 2023-12-25T20:02:54.604 INFO:teuthology.orchestra.run.smithi160.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2023-12-25T20:02:54.604 INFO:teuthology.orchestra.run.smithi160.stdout: librgw2 x86_64 2:16.2.4-0.el8 ceph 4.1 M 2023-12-25T20:02:54.604 INFO:teuthology.orchestra.run.smithi160.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-12-25T20:02:54.604 INFO:teuthology.orchestra.run.smithi160.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2023-12-25T20:02:54.604 INFO:teuthology.orchestra.run.smithi160.stdout: ndctl-libs x86_64 71.1-7.el8 baseos 80 k 2023-12-25T20:02:54.604 INFO:teuthology.orchestra.run.smithi160.stdout: python3-ceph-argparse x86_64 2:16.2.4-0.el8 ceph 44 k 2023-12-25T20:02:54.604 INFO:teuthology.orchestra.run.smithi160.stdout: python3-ceph-common x86_64 2:16.2.4-0.el8 ceph 80 k 2023-12-25T20:02:54.605 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cephfs x86_64 2:16.2.4-0.el8 ceph 213 k 2023-12-25T20:02:54.605 INFO:teuthology.orchestra.run.smithi160.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2023-12-25T20:02:54.605 INFO:teuthology.orchestra.run.smithi160.stdout: python3-rados x86_64 2:16.2.4-0.el8 ceph 387 k 2023-12-25T20:02:54.605 INFO:teuthology.orchestra.run.smithi160.stdout: python3-rbd x86_64 2:16.2.4-0.el8 ceph 368 k 2023-12-25T20:02:54.605 INFO:teuthology.orchestra.run.smithi160.stdout: python3-rgw x86_64 2:16.2.4-0.el8 ceph 114 k 2023-12-25T20:02:54.605 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:02:54.605 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2023-12-25T20:02:54.605 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-12-25T20:02:54.605 INFO:teuthology.orchestra.run.smithi160.stdout:Install 25 Packages 2023-12-25T20:02:54.606 INFO:teuthology.orchestra.run.smithi160.stdout:Upgrade 2 Packages 2023-12-25T20:02:54.606 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:02:54.606 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 59 M 2023-12-25T20:02:54.606 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2023-12-25T20:02:55.104 INFO:teuthology.orchestra.run.smithi160.stdout:(1/27): ceph-base-16.2.4-0.el8.x86_64.rpm 50 MB/s | 6.0 MB 00:00 2023-12-25T20:02:55.130 INFO:teuthology.orchestra.run.smithi160.stdout:(2/27): ceph-selinux-16.2.4-0.el8.x86_64.rpm 997 kB/s | 25 kB 00:00 2023-12-25T20:02:55.196 INFO:teuthology.orchestra.run.smithi160.stdout:(3/27): libcephfs2-16.2.4-0.el8.x86_64.rpm 12 MB/s | 801 kB 00:00 2023-12-25T20:02:55.230 INFO:teuthology.orchestra.run.smithi160.stdout:(4/27): libradosstriper1-16.2.4-0.el8.x86_64.rp 14 MB/s | 486 kB 00:00 2023-12-25T20:02:55.322 INFO:teuthology.orchestra.run.smithi160.stdout:(5/27): librgw2-16.2.4-0.el8.x86_64.rpm 45 MB/s | 4.1 MB 00:00 2023-12-25T20:02:55.347 INFO:teuthology.orchestra.run.smithi160.stdout:(6/27): python3-ceph-argparse-16.2.4-0.el8.x86_ 1.8 MB/s | 44 kB 00:00 2023-12-25T20:02:55.373 INFO:teuthology.orchestra.run.smithi160.stdout:(7/27): python3-ceph-common-16.2.4-0.el8.x86_64 3.1 MB/s | 80 kB 00:00 2023-12-25T20:02:55.398 INFO:teuthology.orchestra.run.smithi160.stdout:(8/27): python3-cephfs-16.2.4-0.el8.x86_64.rpm 8.4 MB/s | 213 kB 00:00 2023-12-25T20:02:55.431 INFO:teuthology.orchestra.run.smithi160.stdout:(9/27): python3-rados-16.2.4-0.el8.x86_64.rpm 11 MB/s | 387 kB 00:00 2023-12-25T20:02:55.457 INFO:teuthology.orchestra.run.smithi160.stdout:(10/27): python3-rbd-16.2.4-0.el8.x86_64.rpm 14 MB/s | 368 kB 00:00 2023-12-25T20:02:55.482 INFO:teuthology.orchestra.run.smithi160.stdout:(11/27): python3-rgw-16.2.4-0.el8.x86_64.rpm 4.5 MB/s | 114 kB 00:00 2023-12-25T20:02:55.641 INFO:teuthology.orchestra.run.smithi160.stdout:(12/27): ceph-radosgw-16.2.4-0.el8.x86_64.rpm 19 MB/s | 13 MB 00:00 2023-12-25T20:02:55.910 INFO:teuthology.orchestra.run.smithi160.stdout:(13/27): ceph-common-16.2.4-0.el8.x86_64.rpm 26 MB/s | 24 MB 00:00 2023-12-25T20:02:55.950 INFO:teuthology.orchestra.run.smithi160.stdout:(14/27): libpmemobj-1.6.1-1.el8.x86_64.rpm 311 kB/s | 145 kB 00:00 2023-12-25T20:02:56.050 INFO:teuthology.orchestra.run.smithi160.stdout:(15/27): python3-prettytable-0.7.2-14.el8.noarc 440 kB/s | 44 kB 00:00 2023-12-25T20:02:56.192 INFO:teuthology.orchestra.run.smithi160.stdout:(16/27): daxctl-libs-71.1-7.el8.x86_64.rpm 297 kB/s | 42 kB 00:00 2023-12-25T20:02:56.271 INFO:teuthology.orchestra.run.smithi160.stdout:(17/27): librabbitmq-0.9.0-4.el8.x86_64.rpm 596 kB/s | 47 kB 00:00 2023-12-25T20:02:56.334 INFO:teuthology.orchestra.run.smithi160.stdout:(18/27): ndctl-libs-71.1-7.el8.x86_64.rpm 1.2 MB/s | 80 kB 00:00 2023-12-25T20:02:56.368 INFO:teuthology.orchestra.run.smithi160.stdout:(19/27): lttng-ust-2.8.1-11.el8.x86_64.rpm 567 kB/s | 259 kB 00:00 2023-12-25T20:02:56.401 INFO:teuthology.orchestra.run.smithi160.stdout:(20/27): librdkafka-1.6.1-1.el8.x86_64.rpm 890 kB/s | 677 kB 00:00 2023-12-25T20:02:56.864 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-12-25T20:02:56.867 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-12-25T20:02:56.867 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2023-12-25T20:02:56.867 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-12-25T20:02:56.867 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2023-12-25T20:02:56.867 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-radosgw x86_64 2:16.2.4-0.el8 ceph 13 M 2023-12-25T20:02:56.867 INFO:teuthology.orchestra.run.smithi050.stdout:Upgrading: 2023-12-25T20:02:56.867 INFO:teuthology.orchestra.run.smithi050.stdout: librados2 x86_64 2:16.2.4-0.el8 ceph 3.7 M 2023-12-25T20:02:56.867 INFO:teuthology.orchestra.run.smithi050.stdout: librbd1 x86_64 2:16.2.4-0.el8 ceph 4.0 M 2023-12-25T20:02:56.867 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2023-12-25T20:02:56.867 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-base x86_64 2:16.2.4-0.el8 ceph 6.0 M 2023-12-25T20:02:56.867 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-common x86_64 2:16.2.4-0.el8 ceph 24 M 2023-12-25T20:02:56.867 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-selinux x86_64 2:16.2.4-0.el8 ceph 25 k 2023-12-25T20:02:56.868 INFO:teuthology.orchestra.run.smithi050.stdout: daxctl-libs x86_64 71.1-7.el8 baseos 42 k 2023-12-25T20:02:56.868 INFO:teuthology.orchestra.run.smithi050.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2023-12-25T20:02:56.868 INFO:teuthology.orchestra.run.smithi050.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-12-25T20:02:56.868 INFO:teuthology.orchestra.run.smithi050.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2023-12-25T20:02:56.868 INFO:teuthology.orchestra.run.smithi050.stdout: libcephfs2 x86_64 2:16.2.4-0.el8 ceph 801 k 2023-12-25T20:02:56.868 INFO:teuthology.orchestra.run.smithi050.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-12-25T20:02:56.868 INFO:teuthology.orchestra.run.smithi050.stdout: libpmemobj x86_64 1.6.1-1.el8 appstream 145 k 2023-12-25T20:02:56.868 INFO:teuthology.orchestra.run.smithi050.stdout: librabbitmq x86_64 0.9.0-4.el8 baseos 47 k 2023-12-25T20:02:56.868 INFO:teuthology.orchestra.run.smithi050.stdout: libradosstriper1 x86_64 2:16.2.4-0.el8 ceph 486 k 2023-12-25T20:02:56.868 INFO:teuthology.orchestra.run.smithi050.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2023-12-25T20:02:56.868 INFO:teuthology.orchestra.run.smithi050.stdout: librgw2 x86_64 2:16.2.4-0.el8 ceph 4.1 M 2023-12-25T20:02:56.868 INFO:teuthology.orchestra.run.smithi050.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-12-25T20:02:56.868 INFO:teuthology.orchestra.run.smithi050.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2023-12-25T20:02:56.868 INFO:teuthology.orchestra.run.smithi050.stdout: ndctl-libs x86_64 71.1-7.el8 baseos 80 k 2023-12-25T20:02:56.869 INFO:teuthology.orchestra.run.smithi050.stdout: python3-ceph-argparse x86_64 2:16.2.4-0.el8 ceph 44 k 2023-12-25T20:02:56.869 INFO:teuthology.orchestra.run.smithi050.stdout: python3-ceph-common x86_64 2:16.2.4-0.el8 ceph 80 k 2023-12-25T20:02:56.869 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cephfs x86_64 2:16.2.4-0.el8 ceph 213 k 2023-12-25T20:02:56.869 INFO:teuthology.orchestra.run.smithi050.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2023-12-25T20:02:56.869 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rados x86_64 2:16.2.4-0.el8 ceph 387 k 2023-12-25T20:02:56.869 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rbd x86_64 2:16.2.4-0.el8 ceph 368 k 2023-12-25T20:02:56.869 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rgw x86_64 2:16.2.4-0.el8 ceph 114 k 2023-12-25T20:02:56.869 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-25T20:02:56.869 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2023-12-25T20:02:56.869 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-12-25T20:02:56.869 INFO:teuthology.orchestra.run.smithi050.stdout:Install 25 Packages 2023-12-25T20:02:56.869 INFO:teuthology.orchestra.run.smithi050.stdout:Upgrade 2 Packages 2023-12-25T20:02:56.870 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-25T20:02:56.870 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 59 M 2023-12-25T20:02:56.870 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2023-12-25T20:02:57.060 INFO:teuthology.orchestra.run.smithi160.stdout:(21/27): fmt-6.2.1-1.el8.x86_64.rpm 160 kB/s | 116 kB 00:00 2023-12-25T20:02:57.085 INFO:teuthology.orchestra.run.smithi160.stdout:(22/27): leveldb-1.22-1.el8.x86_64.rpm 265 kB/s | 181 kB 00:00 2023-12-25T20:02:57.111 INFO:teuthology.orchestra.run.smithi050.stdout:(1/27): ceph-base-16.2.4-0.el8.x86_64.rpm 47 MB/s | 6.0 MB 00:00 2023-12-25T20:02:57.136 INFO:teuthology.orchestra.run.smithi050.stdout:(2/27): ceph-selinux-16.2.4-0.el8.x86_64.rpm 990 kB/s | 25 kB 00:00 2023-12-25T20:02:57.185 INFO:teuthology.orchestra.run.smithi160.stdout:(23/27): gperftools-libs-2.7-9.el8.x86_64.rpm 375 kB/s | 306 kB 00:00 2023-12-25T20:02:57.204 INFO:teuthology.orchestra.run.smithi050.stdout:(3/27): libcephfs2-16.2.4-0.el8.x86_64.rpm 12 MB/s | 801 kB 00:00 2023-12-25T20:02:57.211 INFO:teuthology.orchestra.run.smithi160.stdout:(24/27): liboath-2.6.2-3.el8.x86_64.rpm 391 kB/s | 59 kB 00:00 2023-12-25T20:02:57.244 INFO:teuthology.orchestra.run.smithi160.stdout:(25/27): libunwind-1.3.1-3.el8.x86_64.rpm 472 kB/s | 75 kB 00:00 2023-12-25T20:02:57.245 INFO:teuthology.orchestra.run.smithi050.stdout:(4/27): libradosstriper1-16.2.4-0.el8.x86_64.rp 12 MB/s | 486 kB 00:00 2023-12-25T20:02:57.338 INFO:teuthology.orchestra.run.smithi160.stdout:(26/27): librados2-16.2.4-0.el8.x86_64.rpm 25 MB/s | 3.7 MB 00:00 2023-12-25T20:02:57.345 INFO:teuthology.orchestra.run.smithi050.stdout:(5/27): librgw2-16.2.4-0.el8.x86_64.rpm 41 MB/s | 4.1 MB 00:00 2023-12-25T20:02:57.371 INFO:teuthology.orchestra.run.smithi050.stdout:(6/27): python3-ceph-argparse-16.2.4-0.el8.x86_ 1.8 MB/s | 44 kB 00:00 2023-12-25T20:02:57.395 INFO:teuthology.orchestra.run.smithi160.stdout:(27/27): librbd1-16.2.4-0.el8.x86_64.rpm 22 MB/s | 4.0 MB 00:00 2023-12-25T20:02:57.396 INFO:teuthology.orchestra.run.smithi050.stdout:(7/27): python3-ceph-common-16.2.4-0.el8.x86_64 3.1 MB/s | 80 kB 00:00 2023-12-25T20:02:57.397 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2023-12-25T20:02:57.397 INFO:teuthology.orchestra.run.smithi160.stdout:Total 21 MB/s | 59 MB 00:02 2023-12-25T20:02:57.421 INFO:teuthology.orchestra.run.smithi050.stdout:(8/27): python3-cephfs-16.2.4-0.el8.x86_64.rpm 8.3 MB/s | 213 kB 00:00 2023-12-25T20:02:57.455 INFO:teuthology.orchestra.run.smithi050.stdout:(9/27): python3-rados-16.2.4-0.el8.x86_64.rpm 11 MB/s | 387 kB 00:00 2023-12-25T20:02:57.467 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2023-12-25T20:02:57.514 INFO:teuthology.orchestra.run.smithi050.stdout:(10/27): python3-rbd-16.2.4-0.el8.x86_64.rpm 6.2 MB/s | 368 kB 00:00 2023-12-25T20:02:57.516 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2023-12-25T20:02:57.516 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2023-12-25T20:02:57.539 INFO:teuthology.orchestra.run.smithi050.stdout:(11/27): python3-rgw-16.2.4-0.el8.x86_64.rpm 4.4 MB/s | 114 kB 00:00 2023-12-25T20:02:57.698 INFO:teuthology.orchestra.run.smithi050.stdout:(12/27): ceph-radosgw-16.2.4-0.el8.x86_64.rpm 18 MB/s | 13 MB 00:00 2023-12-25T20:02:57.857 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2023-12-25T20:02:57.860 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2023-12-25T20:02:57.965 INFO:teuthology.orchestra.run.smithi050.stdout:(13/27): ceph-common-16.2.4-0.el8.x86_64.rpm 24 MB/s | 24 MB 00:00 2023-12-25T20:02:57.998 INFO:teuthology.orchestra.run.smithi050.stdout:(14/27): libpmemobj-1.6.1-1.el8.x86_64.rpm 316 kB/s | 145 kB 00:00 2023-12-25T20:02:58.090 INFO:teuthology.orchestra.run.smithi050.stdout:(15/27): python3-prettytable-0.7.2-14.el8.noarc 479 kB/s | 44 kB 00:00 2023-12-25T20:02:58.224 INFO:teuthology.orchestra.run.smithi050.stdout:(16/27): daxctl-libs-71.1-7.el8.x86_64.rpm 316 kB/s | 42 kB 00:00 2023-12-25T20:02:58.274 INFO:teuthology.orchestra.run.smithi050.stdout:(17/27): librabbitmq-0.9.0-4.el8.x86_64.rpm 940 kB/s | 47 kB 00:00 2023-12-25T20:02:58.358 INFO:teuthology.orchestra.run.smithi050.stdout:(18/27): ndctl-libs-71.1-7.el8.x86_64.rpm 961 kB/s | 80 kB 00:00 2023-12-25T20:02:58.458 INFO:teuthology.orchestra.run.smithi050.stdout:(19/27): librdkafka-1.6.1-1.el8.x86_64.rpm 890 kB/s | 677 kB 00:00 2023-12-25T20:02:58.483 INFO:teuthology.orchestra.run.smithi050.stdout:(20/27): lttng-ust-2.8.1-11.el8.x86_64.rpm 501 kB/s | 259 kB 00:00 2023-12-25T20:02:58.534 INFO:teuthology.orchestra.run.smithi050.stdout:(21/27): fmt-6.2.1-1.el8.x86_64.rpm 662 kB/s | 116 kB 00:00 2023-12-25T20:02:58.575 INFO:teuthology.orchestra.run.smithi050.stdout:(22/27): liboath-2.6.2-3.el8.x86_64.rpm 1.4 MB/s | 59 kB 00:00 2023-12-25T20:02:58.634 INFO:teuthology.orchestra.run.smithi050.stdout:(23/27): libunwind-1.3.1-3.el8.x86_64.rpm 1.3 MB/s | 75 kB 00:00 2023-12-25T20:02:58.660 INFO:teuthology.orchestra.run.smithi050.stdout:(24/27): gperftools-libs-2.7-9.el8.x86_64.rpm 1.5 MB/s | 306 kB 00:00 2023-12-25T20:02:58.685 INFO:teuthology.orchestra.run.smithi050.stdout:(25/27): leveldb-1.22-1.el8.x86_64.rpm 924 kB/s | 181 kB 00:00 2023-12-25T20:02:58.777 INFO:teuthology.orchestra.run.smithi050.stdout:(26/27): librados2-16.2.4-0.el8.x86_64.rpm 26 MB/s | 3.7 MB 00:00 2023-12-25T20:02:58.852 INFO:teuthology.orchestra.run.smithi050.stdout:(27/27): librbd1-16.2.4-0.el8.x86_64.rpm 21 MB/s | 4.0 MB 00:00 2023-12-25T20:02:58.857 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2023-12-25T20:02:58.857 INFO:teuthology.orchestra.run.smithi050.stdout:Total 30 MB/s | 59 MB 00:01 2023-12-25T20:02:58.931 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2023-12-25T20:02:58.984 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2023-12-25T20:02:58.985 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2023-12-25T20:02:59.011 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2023-12-25T20:02:59.247 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/29 2023-12-25T20:02:59.346 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2023-12-25T20:02:59.349 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2023-12-25T20:02:59.468 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/29 2023-12-25T20:02:59.654 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/29 2023-12-25T20:03:00.060 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/29 2023-12-25T20:03:00.271 INFO:teuthology.orchestra.run.smithi160.stdout: Upgrading : librados2-2:16.2.4-0.el8.x86_64 4/29 2023-12-25T20:03:00.378 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: librados2-2:16.2.4-0.el8.x86_64 4/29 2023-12-25T20:03:00.557 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2023-12-25T20:03:00.670 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-rados-2:16.2.4-0.el8.x86_64 5/29 2023-12-25T20:03:00.786 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/29 2023-12-25T20:03:00.921 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2023-12-25T20:03:01.002 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/29 2023-12-25T20:03:01.090 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2023-12-25T20:03:01.191 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/29 2023-12-25T20:03:01.298 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : librabbitmq-0.9.0-4.el8.x86_64 7/29 2023-12-25T20:03:01.629 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/29 2023-12-25T20:03:01.824 INFO:teuthology.orchestra.run.smithi050.stdout: Upgrading : librados2-2:16.2.4-0.el8.x86_64 4/29 2023-12-25T20:03:01.865 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 8/29 2023-12-25T20:03:01.904 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: librados2-2:16.2.4-0.el8.x86_64 4/29 2023-12-25T20:03:02.029 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : librgw2-2:16.2.4-0.el8.x86_64 9/29 2023-12-25T20:03:02.105 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: librgw2-2:16.2.4-0.el8.x86_64 9/29 2023-12-25T20:03:02.161 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-rados-2:16.2.4-0.el8.x86_64 5/29 2023-12-25T20:03:02.275 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/29 2023-12-25T20:03:02.341 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2023-12-25T20:03:02.407 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2023-12-25T20:03:02.453 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : daxctl-libs-71.1-7.el8.x86_64 11/29 2023-12-25T20:03:02.520 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: daxctl-libs-71.1-7.el8.x86_64 11/29 2023-12-25T20:03:02.603 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : librabbitmq-0.9.0-4.el8.x86_64 7/29 2023-12-25T20:03:02.676 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 12/29 2023-12-25T20:03:02.871 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-cephfs-2:16.2.4-0.el8.x86_64 13/29 2023-12-25T20:03:03.028 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ndctl-libs-71.1-7.el8.x86_64 14/29 2023-12-25T20:03:03.108 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ndctl-libs-71.1-7.el8.x86_64 14/29 2023-12-25T20:03:03.173 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 8/29 2023-12-25T20:03:03.270 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : libpmemobj-1.6.1-1.el8.x86_64 15/29 2023-12-25T20:03:03.315 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : librgw2-2:16.2.4-0.el8.x86_64 9/29 2023-12-25T20:03:03.389 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: librgw2-2:16.2.4-0.el8.x86_64 9/29 2023-12-25T20:03:03.571 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/29 2023-12-25T20:03:03.709 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: libpmemobj-1.6.1-1.el8.x86_64 15/29 2023-12-25T20:03:03.773 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : daxctl-libs-71.1-7.el8.x86_64 11/29 2023-12-25T20:03:03.837 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: daxctl-libs-71.1-7.el8.x86_64 11/29 2023-12-25T20:03:03.928 INFO:teuthology.orchestra.run.smithi160.stdout: Upgrading : librbd1-2:16.2.4-0.el8.x86_64 16/29 2023-12-25T20:03:04.027 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: librbd1-2:16.2.4-0.el8.x86_64 16/29 2023-12-25T20:03:04.039 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 12/29 2023-12-25T20:03:04.193 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-cephfs-2:16.2.4-0.el8.x86_64 13/29 2023-12-25T20:03:04.232 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-rbd-2:16.2.4-0.el8.x86_64 17/29 2023-12-25T20:03:04.439 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-rgw-2:16.2.4-0.el8.x86_64 18/29 2023-12-25T20:03:04.612 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2023-12-25T20:03:04.615 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ndctl-libs-71.1-7.el8.x86_64 14/29 2023-12-25T20:03:04.682 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2023-12-25T20:03:04.690 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ndctl-libs-71.1-7.el8.x86_64 14/29 2023-12-25T20:03:04.865 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libpmemobj-1.6.1-1.el8.x86_64 15/29 2023-12-25T20:03:04.871 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 20/29 2023-12-25T20:03:05.095 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 21/29 2023-12-25T20:03:05.260 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 22/29 2023-12-25T20:03:05.322 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: libpmemobj-1.6.1-1.el8.x86_64 15/29 2023-12-25T20:03:05.487 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-ceph-common-2:16.2.4-0.el8.x86_64 23/29 2023-12-25T20:03:05.523 INFO:teuthology.orchestra.run.smithi050.stdout: Upgrading : librbd1-2:16.2.4-0.el8.x86_64 16/29 2023-12-25T20:03:05.610 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: librbd1-2:16.2.4-0.el8.x86_64 16/29 2023-12-25T20:03:05.813 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-rbd-2:16.2.4-0.el8.x86_64 17/29 2023-12-25T20:03:06.046 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-rgw-2:16.2.4-0.el8.x86_64 18/29 2023-12-25T20:03:06.182 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2023-12-25T20:03:06.253 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2023-12-25T20:03:06.452 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 20/29 2023-12-25T20:03:06.624 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 21/29 2023-12-25T20:03:06.821 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 22/29 2023-12-25T20:03:07.065 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-ceph-common-2:16.2.4-0.el8.x86_64 23/29 2023-12-25T20:03:08.062 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2023-12-25T20:03:08.312 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-common-2:16.2.4-0.el8.x86_64 24/29 2023-12-25T20:03:09.026 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2023-12-25T20:03:09.320 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-base-2:16.2.4-0.el8.x86_64 25/29 2023-12-25T20:03:09.426 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-base-2:16.2.4-0.el8.x86_64 25/29 2023-12-25T20:03:09.552 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2023-12-25T20:03:09.762 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2023-12-25T20:03:10.048 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-common-2:16.2.4-0.el8.x86_64 24/29 2023-12-25T20:03:10.785 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2023-12-25T20:03:11.116 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-base-2:16.2.4-0.el8.x86_64 25/29 2023-12-25T20:03:11.227 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-base-2:16.2.4-0.el8.x86_64 25/29 2023-12-25T20:03:11.398 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2023-12-25T20:03:28.090 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2023-12-25T20:03:28.090 INFO:teuthology.orchestra.run.smithi160.stdout:skipping the directory /sys 2023-12-25T20:03:28.090 INFO:teuthology.orchestra.run.smithi160.stdout:skipping the directory /proc 2023-12-25T20:03:28.091 INFO:teuthology.orchestra.run.smithi160.stdout:skipping the directory /dev 2023-12-25T20:03:28.091 INFO:teuthology.orchestra.run.smithi160.stdout:skipping the directory /run 2023-12-25T20:03:28.091 INFO:teuthology.orchestra.run.smithi160.stdout:skipping the directory /mnt 2023-12-25T20:03:28.091 INFO:teuthology.orchestra.run.smithi160.stdout:skipping the directory /var/tmp 2023-12-25T20:03:28.091 INFO:teuthology.orchestra.run.smithi160.stdout:skipping the directory /home 2023-12-25T20:03:28.091 INFO:teuthology.orchestra.run.smithi160.stdout:skipping the directory /tmp 2023-12-25T20:03:28.091 INFO:teuthology.orchestra.run.smithi160.stdout:skipping the directory /dev 2023-12-25T20:03:28.091 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:03:29.408 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2023-12-25T20:03:29.513 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2023-12-25T20:03:29.513 INFO:teuthology.orchestra.run.smithi160.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 28/29 2023-12-25T20:03:29.735 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 28/29 2023-12-25T20:03:29.735 INFO:teuthology.orchestra.run.smithi160.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 29/29 2023-12-25T20:03:30.468 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 29/29 2023-12-25T20:03:30.469 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-base-2:16.2.4-0.el8.x86_64 1/29 2023-12-25T20:03:30.469 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-common-2:16.2.4-0.el8.x86_64 2/29 2023-12-25T20:03:30.469 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-radosgw-2:16.2.4-0.el8.x86_64 3/29 2023-12-25T20:03:30.469 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-selinux-2:16.2.4-0.el8.x86_64 4/29 2023-12-25T20:03:30.469 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : libcephfs2-2:16.2.4-0.el8.x86_64 5/29 2023-12-25T20:03:30.469 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : libradosstriper1-2:16.2.4-0.el8.x86_64 6/29 2023-12-25T20:03:30.469 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : librgw2-2:16.2.4-0.el8.x86_64 7/29 2023-12-25T20:03:30.469 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 8/29 2023-12-25T20:03:30.469 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-ceph-common-2:16.2.4-0.el8.x86_64 9/29 2023-12-25T20:03:30.469 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-cephfs-2:16.2.4-0.el8.x86_64 10/29 2023-12-25T20:03:30.469 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-rados-2:16.2.4-0.el8.x86_64 11/29 2023-12-25T20:03:30.469 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-rbd-2:16.2.4-0.el8.x86_64 12/29 2023-12-25T20:03:30.469 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-rgw-2:16.2.4-0.el8.x86_64 13/29 2023-12-25T20:03:30.470 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : libpmemobj-1.6.1-1.el8.x86_64 14/29 2023-12-25T20:03:30.470 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 15/29 2023-12-25T20:03:30.470 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 16/29 2023-12-25T20:03:30.470 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 17/29 2023-12-25T20:03:30.470 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : daxctl-libs-71.1-7.el8.x86_64 18/29 2023-12-25T20:03:30.470 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : librabbitmq-0.9.0-4.el8.x86_64 19/29 2023-12-25T20:03:30.470 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ndctl-libs-71.1-7.el8.x86_64 20/29 2023-12-25T20:03:30.470 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 21/29 2023-12-25T20:03:30.470 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 22/29 2023-12-25T20:03:30.470 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : leveldb-1.22-1.el8.x86_64 23/29 2023-12-25T20:03:30.470 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 24/29 2023-12-25T20:03:30.470 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 25/29 2023-12-25T20:03:30.471 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : librados2-2:16.2.4-0.el8.x86_64 26/29 2023-12-25T20:03:30.471 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 27/29 2023-12-25T20:03:30.471 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : librbd1-2:16.2.4-0.el8.x86_64 28/29 2023-12-25T20:03:30.657 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2023-12-25T20:03:30.657 INFO:teuthology.orchestra.run.smithi050.stdout:skipping the directory /sys 2023-12-25T20:03:30.657 INFO:teuthology.orchestra.run.smithi050.stdout:skipping the directory /proc 2023-12-25T20:03:30.657 INFO:teuthology.orchestra.run.smithi050.stdout:skipping the directory /dev 2023-12-25T20:03:30.657 INFO:teuthology.orchestra.run.smithi050.stdout:skipping the directory /run 2023-12-25T20:03:30.657 INFO:teuthology.orchestra.run.smithi050.stdout:skipping the directory /mnt 2023-12-25T20:03:30.657 INFO:teuthology.orchestra.run.smithi050.stdout:skipping the directory /var/tmp 2023-12-25T20:03:30.657 INFO:teuthology.orchestra.run.smithi050.stdout:skipping the directory /home 2023-12-25T20:03:30.657 INFO:teuthology.orchestra.run.smithi050.stdout:skipping the directory /tmp 2023-12-25T20:03:30.657 INFO:teuthology.orchestra.run.smithi050.stdout:skipping the directory /dev 2023-12-25T20:03:30.657 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-25T20:03:31.743 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 29/29 2023-12-25T20:03:31.743 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:03:31.743 INFO:teuthology.orchestra.run.smithi160.stdout:Upgraded: 2023-12-25T20:03:31.743 INFO:teuthology.orchestra.run.smithi160.stdout: librados2-2:16.2.4-0.el8.x86_64 librbd1-2:16.2.4-0.el8.x86_64 2023-12-25T20:03:31.743 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2023-12-25T20:03:31.743 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-base-2:16.2.4-0.el8.x86_64 2023-12-25T20:03:31.743 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-common-2:16.2.4-0.el8.x86_64 2023-12-25T20:03:31.743 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-radosgw-2:16.2.4-0.el8.x86_64 2023-12-25T20:03:31.744 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-selinux-2:16.2.4-0.el8.x86_64 2023-12-25T20:03:31.744 INFO:teuthology.orchestra.run.smithi160.stdout: daxctl-libs-71.1-7.el8.x86_64 2023-12-25T20:03:31.744 INFO:teuthology.orchestra.run.smithi160.stdout: fmt-6.2.1-1.el8.x86_64 2023-12-25T20:03:31.744 INFO:teuthology.orchestra.run.smithi160.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-12-25T20:03:31.744 INFO:teuthology.orchestra.run.smithi160.stdout: leveldb-1.22-1.el8.x86_64 2023-12-25T20:03:31.744 INFO:teuthology.orchestra.run.smithi160.stdout: libcephfs2-2:16.2.4-0.el8.x86_64 2023-12-25T20:03:31.744 INFO:teuthology.orchestra.run.smithi160.stdout: liboath-2.6.2-3.el8.x86_64 2023-12-25T20:03:31.744 INFO:teuthology.orchestra.run.smithi160.stdout: libpmemobj-1.6.1-1.el8.x86_64 2023-12-25T20:03:31.744 INFO:teuthology.orchestra.run.smithi160.stdout: librabbitmq-0.9.0-4.el8.x86_64 2023-12-25T20:03:31.744 INFO:teuthology.orchestra.run.smithi160.stdout: libradosstriper1-2:16.2.4-0.el8.x86_64 2023-12-25T20:03:31.744 INFO:teuthology.orchestra.run.smithi160.stdout: librdkafka-1.6.1-1.el8.x86_64 2023-12-25T20:03:31.744 INFO:teuthology.orchestra.run.smithi160.stdout: librgw2-2:16.2.4-0.el8.x86_64 2023-12-25T20:03:31.744 INFO:teuthology.orchestra.run.smithi160.stdout: libunwind-1.3.1-3.el8.x86_64 2023-12-25T20:03:31.744 INFO:teuthology.orchestra.run.smithi160.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-12-25T20:03:31.744 INFO:teuthology.orchestra.run.smithi160.stdout: ndctl-libs-71.1-7.el8.x86_64 2023-12-25T20:03:31.745 INFO:teuthology.orchestra.run.smithi160.stdout: python3-ceph-argparse-2:16.2.4-0.el8.x86_64 2023-12-25T20:03:31.745 INFO:teuthology.orchestra.run.smithi160.stdout: python3-ceph-common-2:16.2.4-0.el8.x86_64 2023-12-25T20:03:31.745 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cephfs-2:16.2.4-0.el8.x86_64 2023-12-25T20:03:31.745 INFO:teuthology.orchestra.run.smithi160.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-12-25T20:03:31.745 INFO:teuthology.orchestra.run.smithi160.stdout: python3-rados-2:16.2.4-0.el8.x86_64 2023-12-25T20:03:31.745 INFO:teuthology.orchestra.run.smithi160.stdout: python3-rbd-2:16.2.4-0.el8.x86_64 2023-12-25T20:03:31.745 INFO:teuthology.orchestra.run.smithi160.stdout: python3-rgw-2:16.2.4-0.el8.x86_64 2023-12-25T20:03:31.745 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:03:31.745 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2023-12-25T20:03:31.949 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install ceph-test 2023-12-25T20:03:32.088 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2023-12-25T20:03:32.170 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2023-12-25T20:03:32.171 INFO:teuthology.orchestra.run.smithi050.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 28/29 2023-12-25T20:03:32.398 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 28/29 2023-12-25T20:03:32.398 INFO:teuthology.orchestra.run.smithi050.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 29/29 2023-12-25T20:03:32.810 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:00:43 ago on Mon 25 Dec 2023 08:02:49 PM UTC. 2023-12-25T20:03:33.161 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 29/29 2023-12-25T20:03:33.162 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-base-2:16.2.4-0.el8.x86_64 1/29 2023-12-25T20:03:33.162 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-common-2:16.2.4-0.el8.x86_64 2/29 2023-12-25T20:03:33.162 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-radosgw-2:16.2.4-0.el8.x86_64 3/29 2023-12-25T20:03:33.162 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-selinux-2:16.2.4-0.el8.x86_64 4/29 2023-12-25T20:03:33.162 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libcephfs2-2:16.2.4-0.el8.x86_64 5/29 2023-12-25T20:03:33.162 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libradosstriper1-2:16.2.4-0.el8.x86_64 6/29 2023-12-25T20:03:33.162 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librgw2-2:16.2.4-0.el8.x86_64 7/29 2023-12-25T20:03:33.162 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 8/29 2023-12-25T20:03:33.162 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-ceph-common-2:16.2.4-0.el8.x86_64 9/29 2023-12-25T20:03:33.162 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-cephfs-2:16.2.4-0.el8.x86_64 10/29 2023-12-25T20:03:33.162 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-rados-2:16.2.4-0.el8.x86_64 11/29 2023-12-25T20:03:33.162 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-rbd-2:16.2.4-0.el8.x86_64 12/29 2023-12-25T20:03:33.163 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-rgw-2:16.2.4-0.el8.x86_64 13/29 2023-12-25T20:03:33.163 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libpmemobj-1.6.1-1.el8.x86_64 14/29 2023-12-25T20:03:33.163 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 15/29 2023-12-25T20:03:33.163 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 16/29 2023-12-25T20:03:33.163 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 17/29 2023-12-25T20:03:33.163 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : daxctl-libs-71.1-7.el8.x86_64 18/29 2023-12-25T20:03:33.163 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librabbitmq-0.9.0-4.el8.x86_64 19/29 2023-12-25T20:03:33.163 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ndctl-libs-71.1-7.el8.x86_64 20/29 2023-12-25T20:03:33.163 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 21/29 2023-12-25T20:03:33.163 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 22/29 2023-12-25T20:03:33.164 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : leveldb-1.22-1.el8.x86_64 23/29 2023-12-25T20:03:33.164 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 24/29 2023-12-25T20:03:33.164 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 25/29 2023-12-25T20:03:33.164 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librados2-2:16.2.4-0.el8.x86_64 26/29 2023-12-25T20:03:33.164 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 27/29 2023-12-25T20:03:33.164 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librbd1-2:16.2.4-0.el8.x86_64 28/29 2023-12-25T20:03:34.514 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2023-12-25T20:03:34.515 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-12-25T20:03:34.515 INFO:teuthology.orchestra.run.smithi160.stdout: Package Architecture Version Repository Size 2023-12-25T20:03:34.515 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-12-25T20:03:34.515 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2023-12-25T20:03:34.515 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-test x86_64 2:16.2.4-0.el8 ceph 52 M 2023-12-25T20:03:34.515 INFO:teuthology.orchestra.run.smithi160.stdout:Installing dependencies: 2023-12-25T20:03:34.515 INFO:teuthology.orchestra.run.smithi160.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2023-12-25T20:03:34.516 INFO:teuthology.orchestra.run.smithi160.stdout: libcephsqlite x86_64 2:16.2.4-0.el8 ceph 169 k 2023-12-25T20:03:34.516 INFO:teuthology.orchestra.run.smithi160.stdout: oniguruma x86_64 6.8.2-2.el8 appstream 187 k 2023-12-25T20:03:34.516 INFO:teuthology.orchestra.run.smithi160.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2023-12-25T20:03:34.516 INFO:teuthology.orchestra.run.smithi160.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2023-12-25T20:03:34.516 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:03:34.516 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2023-12-25T20:03:34.516 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-12-25T20:03:34.516 INFO:teuthology.orchestra.run.smithi160.stdout:Install 6 Packages 2023-12-25T20:03:34.516 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:03:34.517 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 53 M 2023-12-25T20:03:34.517 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 229 M 2023-12-25T20:03:34.517 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2023-12-25T20:03:34.533 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 29/29 2023-12-25T20:03:34.534 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-25T20:03:34.534 INFO:teuthology.orchestra.run.smithi050.stdout:Upgraded: 2023-12-25T20:03:34.534 INFO:teuthology.orchestra.run.smithi050.stdout: librados2-2:16.2.4-0.el8.x86_64 librbd1-2:16.2.4-0.el8.x86_64 2023-12-25T20:03:34.534 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2023-12-25T20:03:34.534 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-base-2:16.2.4-0.el8.x86_64 2023-12-25T20:03:34.534 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-common-2:16.2.4-0.el8.x86_64 2023-12-25T20:03:34.534 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-radosgw-2:16.2.4-0.el8.x86_64 2023-12-25T20:03:34.534 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-selinux-2:16.2.4-0.el8.x86_64 2023-12-25T20:03:34.534 INFO:teuthology.orchestra.run.smithi050.stdout: daxctl-libs-71.1-7.el8.x86_64 2023-12-25T20:03:34.534 INFO:teuthology.orchestra.run.smithi050.stdout: fmt-6.2.1-1.el8.x86_64 2023-12-25T20:03:34.534 INFO:teuthology.orchestra.run.smithi050.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-12-25T20:03:34.534 INFO:teuthology.orchestra.run.smithi050.stdout: leveldb-1.22-1.el8.x86_64 2023-12-25T20:03:34.534 INFO:teuthology.orchestra.run.smithi050.stdout: libcephfs2-2:16.2.4-0.el8.x86_64 2023-12-25T20:03:34.535 INFO:teuthology.orchestra.run.smithi050.stdout: liboath-2.6.2-3.el8.x86_64 2023-12-25T20:03:34.535 INFO:teuthology.orchestra.run.smithi050.stdout: libpmemobj-1.6.1-1.el8.x86_64 2023-12-25T20:03:34.535 INFO:teuthology.orchestra.run.smithi050.stdout: librabbitmq-0.9.0-4.el8.x86_64 2023-12-25T20:03:34.535 INFO:teuthology.orchestra.run.smithi050.stdout: libradosstriper1-2:16.2.4-0.el8.x86_64 2023-12-25T20:03:34.535 INFO:teuthology.orchestra.run.smithi050.stdout: librdkafka-1.6.1-1.el8.x86_64 2023-12-25T20:03:34.535 INFO:teuthology.orchestra.run.smithi050.stdout: librgw2-2:16.2.4-0.el8.x86_64 2023-12-25T20:03:34.535 INFO:teuthology.orchestra.run.smithi050.stdout: libunwind-1.3.1-3.el8.x86_64 2023-12-25T20:03:34.535 INFO:teuthology.orchestra.run.smithi050.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-12-25T20:03:34.535 INFO:teuthology.orchestra.run.smithi050.stdout: ndctl-libs-71.1-7.el8.x86_64 2023-12-25T20:03:34.535 INFO:teuthology.orchestra.run.smithi050.stdout: python3-ceph-argparse-2:16.2.4-0.el8.x86_64 2023-12-25T20:03:34.535 INFO:teuthology.orchestra.run.smithi050.stdout: python3-ceph-common-2:16.2.4-0.el8.x86_64 2023-12-25T20:03:34.535 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cephfs-2:16.2.4-0.el8.x86_64 2023-12-25T20:03:34.535 INFO:teuthology.orchestra.run.smithi050.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-12-25T20:03:34.536 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rados-2:16.2.4-0.el8.x86_64 2023-12-25T20:03:34.536 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rbd-2:16.2.4-0.el8.x86_64 2023-12-25T20:03:34.536 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rgw-2:16.2.4-0.el8.x86_64 2023-12-25T20:03:34.536 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-25T20:03:34.536 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-12-25T20:03:34.660 INFO:teuthology.orchestra.run.smithi160.stdout:(1/6): libcephsqlite-16.2.4-0.el8.x86_64.rpm 1.6 MB/s | 169 kB 00:00 2023-12-25T20:03:34.770 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-test 2023-12-25T20:03:35.293 INFO:teuthology.orchestra.run.smithi160.stdout:(2/6): ceph-test-16.2.4-0.el8.x86_64.rpm 71 MB/s | 52 MB 00:00 2023-12-25T20:03:35.318 INFO:teuthology.orchestra.run.smithi160.stdout:(3/6): jq-1.6-8.el8.x86_64.rpm 267 kB/s | 203 kB 00:00 2023-12-25T20:03:35.344 INFO:teuthology.orchestra.run.smithi160.stdout:(4/6): oniguruma-6.8.2-2.el8.x86_64.rpm 274 kB/s | 187 kB 00:00 2023-12-25T20:03:35.435 INFO:teuthology.orchestra.run.smithi160.stdout:(5/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 771 kB/s | 71 kB 00:00 2023-12-25T20:03:35.667 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:00:44 ago on Mon 25 Dec 2023 08:02:51 PM UTC. 2023-12-25T20:03:35.736 INFO:teuthology.orchestra.run.smithi160.stdout:(6/6): socat-1.7.4.1-1.el8.x86_64.rpm 731 kB/s | 323 kB 00:00 2023-12-25T20:03:35.737 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2023-12-25T20:03:35.737 INFO:teuthology.orchestra.run.smithi160.stdout:Total 43 MB/s | 53 MB 00:01 2023-12-25T20:03:35.780 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2023-12-25T20:03:35.801 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2023-12-25T20:03:35.802 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2023-12-25T20:03:36.104 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2023-12-25T20:03:36.107 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2023-12-25T20:03:36.577 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2023-12-25T20:03:36.834 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2023-12-25T20:03:37.059 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2023-12-25T20:03:37.232 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/6 2023-12-25T20:03:37.307 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/6 2023-12-25T20:03:37.490 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-12-25T20:03:37.491 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-12-25T20:03:37.491 INFO:teuthology.orchestra.run.smithi050.stdout: Package Architecture Version Repository Size 2023-12-25T20:03:37.491 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-12-25T20:03:37.491 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2023-12-25T20:03:37.492 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-test x86_64 2:16.2.4-0.el8 ceph 52 M 2023-12-25T20:03:37.492 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2023-12-25T20:03:37.492 INFO:teuthology.orchestra.run.smithi050.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2023-12-25T20:03:37.492 INFO:teuthology.orchestra.run.smithi050.stdout: libcephsqlite x86_64 2:16.2.4-0.el8 ceph 169 k 2023-12-25T20:03:37.492 INFO:teuthology.orchestra.run.smithi050.stdout: oniguruma x86_64 6.8.2-2.el8 appstream 187 k 2023-12-25T20:03:37.492 INFO:teuthology.orchestra.run.smithi050.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2023-12-25T20:03:37.492 INFO:teuthology.orchestra.run.smithi050.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2023-12-25T20:03:37.492 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-25T20:03:37.492 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2023-12-25T20:03:37.492 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-12-25T20:03:37.492 INFO:teuthology.orchestra.run.smithi050.stdout:Install 6 Packages 2023-12-25T20:03:37.492 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-25T20:03:37.493 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 53 M 2023-12-25T20:03:37.493 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 229 M 2023-12-25T20:03:37.493 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2023-12-25T20:03:37.522 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2023-12-25T20:03:37.603 INFO:teuthology.orchestra.run.smithi050.stdout:(1/6): libcephsqlite-16.2.4-0.el8.x86_64.rpm 2.5 MB/s | 169 kB 00:00 2023-12-25T20:03:37.673 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2023-12-25T20:03:38.287 INFO:teuthology.orchestra.run.smithi050.stdout:(2/6): ceph-test-16.2.4-0.el8.x86_64.rpm 69 MB/s | 52 MB 00:00 2023-12-25T20:03:38.311 INFO:teuthology.orchestra.run.smithi050.stdout:(3/6): jq-1.6-8.el8.x86_64.rpm 262 kB/s | 203 kB 00:00 2023-12-25T20:03:38.337 INFO:teuthology.orchestra.run.smithi050.stdout:(4/6): oniguruma-6.8.2-2.el8.x86_64.rpm 255 kB/s | 187 kB 00:00 2023-12-25T20:03:38.429 INFO:teuthology.orchestra.run.smithi050.stdout:(5/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 770 kB/s | 71 kB 00:00 2023-12-25T20:03:38.774 INFO:teuthology.orchestra.run.smithi050.stdout:(6/6): socat-1.7.4.1-1.el8.x86_64.rpm 663 kB/s | 323 kB 00:00 2023-12-25T20:03:38.775 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2023-12-25T20:03:38.776 INFO:teuthology.orchestra.run.smithi050.stdout:Total 41 MB/s | 53 MB 00:01 2023-12-25T20:03:38.821 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2023-12-25T20:03:38.843 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2023-12-25T20:03:38.844 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2023-12-25T20:03:39.168 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2023-12-25T20:03:39.171 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2023-12-25T20:03:39.688 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2023-12-25T20:03:39.891 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2023-12-25T20:03:40.073 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2023-12-25T20:03:40.277 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/6 2023-12-25T20:03:40.360 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/6 2023-12-25T20:03:40.553 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2023-12-25T20:03:40.694 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2023-12-25T20:03:42.574 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2023-12-25T20:03:42.838 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-test-2:16.2.4-0.el8.x86_64 6/6 2023-12-25T20:03:43.131 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-test-2:16.2.4-0.el8.x86_64 6/6 2023-12-25T20:03:43.131 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-test-2:16.2.4-0.el8.x86_64 1/6 2023-12-25T20:03:43.131 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : libcephsqlite-2:16.2.4-0.el8.x86_64 2/6 2023-12-25T20:03:43.131 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2023-12-25T20:03:43.131 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/6 2023-12-25T20:03:43.131 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2023-12-25T20:03:43.858 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2023-12-25T20:03:43.858 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:03:43.858 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2023-12-25T20:03:43.858 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-test-2:16.2.4-0.el8.x86_64 jq-1.6-8.el8.x86_64 2023-12-25T20:03:43.858 INFO:teuthology.orchestra.run.smithi160.stdout: libcephsqlite-2:16.2.4-0.el8.x86_64 oniguruma-6.8.2-2.el8.x86_64 2023-12-25T20:03:43.858 INFO:teuthology.orchestra.run.smithi160.stdout: socat-1.7.4.1-1.el8.x86_64 xmlstarlet-1.6.1-20.el8.x86_64 2023-12-25T20:03:43.858 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:03:43.858 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2023-12-25T20:03:44.076 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install ceph 2023-12-25T20:03:44.934 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:00:55 ago on Mon 25 Dec 2023 08:02:49 PM UTC. 2023-12-25T20:03:45.930 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2023-12-25T20:03:46.200 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-test-2:16.2.4-0.el8.x86_64 6/6 2023-12-25T20:03:46.501 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-test-2:16.2.4-0.el8.x86_64 6/6 2023-12-25T20:03:46.501 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-test-2:16.2.4-0.el8.x86_64 1/6 2023-12-25T20:03:46.501 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libcephsqlite-2:16.2.4-0.el8.x86_64 2/6 2023-12-25T20:03:46.501 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2023-12-25T20:03:46.501 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/6 2023-12-25T20:03:46.501 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2023-12-25T20:03:46.632 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2023-12-25T20:03:46.635 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-12-25T20:03:46.636 INFO:teuthology.orchestra.run.smithi160.stdout: Package Arch Version Repository Size 2023-12-25T20:03:46.636 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-12-25T20:03:46.636 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2023-12-25T20:03:46.636 INFO:teuthology.orchestra.run.smithi160.stdout: ceph x86_64 2:16.2.4-0.el8 ceph 6.3 k 2023-12-25T20:03:46.636 INFO:teuthology.orchestra.run.smithi160.stdout:Installing dependencies: 2023-12-25T20:03:46.636 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mds x86_64 2:16.2.4-0.el8 ceph 2.4 M 2023-12-25T20:03:46.636 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr x86_64 2:16.2.4-0.el8 ceph 1.6 M 2023-12-25T20:03:46.636 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr-modules-core noarch 2:16.2.4-0.el8 ceph-noarch 220 k 2023-12-25T20:03:46.636 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mon x86_64 2:16.2.4-0.el8 ceph 5.0 M 2023-12-25T20:03:46.636 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-osd x86_64 2:16.2.4-0.el8 ceph 18 M 2023-12-25T20:03:46.636 INFO:teuthology.orchestra.run.smithi160.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-12-25T20:03:46.636 INFO:teuthology.orchestra.run.smithi160.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-12-25T20:03:46.637 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2023-12-25T20:03:46.637 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-12-25T20:03:46.637 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-12-25T20:03:46.637 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cryptography x86_64 3.2.1-6.el8 baseos 641 k 2023-12-25T20:03:46.637 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-12-25T20:03:46.637 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-12-25T20:03:46.637 INFO:teuthology.orchestra.run.smithi160.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-12-25T20:03:46.637 INFO:teuthology.orchestra.run.smithi160.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2023-12-25T20:03:46.637 INFO:teuthology.orchestra.run.smithi160.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2023-12-25T20:03:46.637 INFO:teuthology.orchestra.run.smithi160.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-12-25T20:03:46.637 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-12-25T20:03:46.637 INFO:teuthology.orchestra.run.smithi160.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-12-25T20:03:46.637 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2023-12-25T20:03:46.637 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2023-12-25T20:03:46.638 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2023-12-25T20:03:46.638 INFO:teuthology.orchestra.run.smithi160.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-12-25T20:03:46.638 INFO:teuthology.orchestra.run.smithi160.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-12-25T20:03:46.638 INFO:teuthology.orchestra.run.smithi160.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-12-25T20:03:46.638 INFO:teuthology.orchestra.run.smithi160.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-12-25T20:03:46.638 INFO:teuthology.orchestra.run.smithi160.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-12-25T20:03:46.638 INFO:teuthology.orchestra.run.smithi160.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-12-25T20:03:46.638 INFO:teuthology.orchestra.run.smithi160.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-12-25T20:03:46.638 INFO:teuthology.orchestra.run.smithi160.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2023-12-25T20:03:46.638 INFO:teuthology.orchestra.run.smithi160.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-12-25T20:03:46.638 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:03:46.638 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2023-12-25T20:03:46.638 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-12-25T20:03:46.638 INFO:teuthology.orchestra.run.smithi160.stdout:Install 32 Packages 2023-12-25T20:03:46.638 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:03:46.639 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 31 M 2023-12-25T20:03:46.639 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 111 M 2023-12-25T20:03:46.639 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2023-12-25T20:03:47.145 INFO:teuthology.orchestra.run.smithi160.stdout:(1/32): ceph-16.2.4-0.el8.x86_64.rpm 110 kB/s | 6.3 kB 00:00 2023-12-25T20:03:47.195 INFO:teuthology.orchestra.run.smithi160.stdout:(2/32): ceph-mgr-16.2.4-0.el8.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2023-12-25T20:03:47.241 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2023-12-25T20:03:47.241 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-25T20:03:47.241 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2023-12-25T20:03:47.241 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-test-2:16.2.4-0.el8.x86_64 jq-1.6-8.el8.x86_64 2023-12-25T20:03:47.241 INFO:teuthology.orchestra.run.smithi050.stdout: libcephsqlite-2:16.2.4-0.el8.x86_64 oniguruma-6.8.2-2.el8.x86_64 2023-12-25T20:03:47.241 INFO:teuthology.orchestra.run.smithi050.stdout: socat-1.7.4.1-1.el8.x86_64 xmlstarlet-1.6.1-20.el8.x86_64 2023-12-25T20:03:47.242 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-25T20:03:47.242 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-12-25T20:03:47.245 INFO:teuthology.orchestra.run.smithi160.stdout:(3/32): ceph-mds-16.2.4-0.el8.x86_64.rpm 15 MB/s | 2.4 MB 00:00 2023-12-25T20:03:47.346 INFO:teuthology.orchestra.run.smithi160.stdout:(4/32): ceph-mon-16.2.4-0.el8.x86_64.rpm 25 MB/s | 5.0 MB 00:00 2023-12-25T20:03:47.371 INFO:teuthology.orchestra.run.smithi160.stdout:(5/32): ceph-mgr-modules-core-16.2.4-0.el8.noar 1.7 MB/s | 220 kB 00:00 2023-12-25T20:03:47.468 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph 2023-12-25T20:03:47.588 INFO:teuthology.orchestra.run.smithi160.stdout:(6/32): ceph-osd-16.2.4-0.el8.x86_64.rpm 46 MB/s | 18 MB 00:00 2023-12-25T20:03:47.730 INFO:teuthology.orchestra.run.smithi160.stdout:(7/32): python3-mako-1.0.6-14.el8.noarch.rpm 409 kB/s | 157 kB 00:00 2023-12-25T20:03:47.755 INFO:teuthology.orchestra.run.smithi160.stdout:(8/32): python3-markupsafe-0.23-19.el8.x86_64.r 101 kB/s | 39 kB 00:00 2023-12-25T20:03:47.822 INFO:teuthology.orchestra.run.smithi160.stdout:(9/32): python3-pytz-2017.2-11.el8.noarch.rpm 681 kB/s | 57 kB 00:00 2023-12-25T20:03:47.981 INFO:teuthology.orchestra.run.smithi160.stdout:(10/32): python3-pyOpenSSL-19.0.0-1.el8.noarch. 264 kB/s | 103 kB 00:00 2023-12-25T20:03:48.057 INFO:teuthology.orchestra.run.smithi160.stdout:(11/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 1.0 MB/s | 249 kB 00:00 2023-12-25T20:03:48.081 INFO:teuthology.orchestra.run.smithi160.stdout:(12/32): python3-werkzeug-0.12.2-4.el8.noarch.r 1.4 MB/s | 457 kB 00:00 2023-12-25T20:03:48.290 INFO:teuthology.orchestra.run.smithi160.stdout:(13/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 209 kB/s | 44 kB 00:00 2023-12-25T20:03:48.324 INFO:teuthology.orchestra.run.smithi160.stdout:(14/32): python3-cryptography-3.2.1-6.el8.x86_6 1.8 MB/s | 641 kB 00:00 2023-12-25T20:03:48.359 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:00:57 ago on Mon 25 Dec 2023 08:02:51 PM UTC. 2023-12-25T20:03:48.466 INFO:teuthology.orchestra.run.smithi160.stdout:(15/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 1.0 MB/s | 185 kB 00:00 2023-12-25T20:03:48.491 INFO:teuthology.orchestra.run.smithi160.stdout:(16/32): python3-pycparser-2.14-14.el8.noarch.r 251 kB/s | 109 kB 00:00 2023-12-25T20:03:48.583 INFO:teuthology.orchestra.run.smithi160.stdout:(17/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 667 kB/s | 173 kB 00:00 2023-12-25T20:03:48.608 INFO:teuthology.orchestra.run.smithi160.stdout:(18/32): python3-cherrypy-18.4.0-1.el8.noarch.r 2.6 MB/s | 384 kB 00:00 2023-12-25T20:03:48.633 INFO:teuthology.orchestra.run.smithi160.stdout:(19/32): python3-jaraco-6.2-6.el8.noarch.rpm 78 kB/s | 11 kB 00:00 2023-12-25T20:03:48.659 INFO:teuthology.orchestra.run.smithi160.stdout:(20/32): python3-jaraco-functools-2.0-4.el8.noa 233 kB/s | 18 kB 00:00 2023-12-25T20:03:48.684 INFO:teuthology.orchestra.run.smithi160.stdout:(21/32): python3-logutils-0.3.5-11.el8.noarch.r 646 kB/s | 49 kB 00:00 2023-12-25T20:03:48.768 INFO:teuthology.orchestra.run.smithi160.stdout:(22/32): python3-more-itertools-7.2.0-3.el8.noa 439 kB/s | 59 kB 00:00 2023-12-25T20:03:48.993 INFO:teuthology.orchestra.run.smithi160.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 50 kB/s | 16 kB 00:00 2023-12-25T20:03:49.018 INFO:teuthology.orchestra.run.smithi160.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 75 kB/s | 19 kB 00:00 2023-12-25T20:03:49.043 INFO:teuthology.orchestra.run.smithi160.stdout:(25/32): python3-pecan-1.3.2-9.el8.noarch.rpm 736 kB/s | 283 kB 00:00 2023-12-25T20:03:49.100 INFO:teuthology.orchestra.run.smithi160.stdout:(26/32): python3-singledispatch-3.4.0.3-18.el8. 222 kB/s | 24 kB 00:00 2023-12-25T20:03:49.127 INFO:teuthology.orchestra.run.smithi160.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 268 kB/s | 29 kB 00:00 2023-12-25T20:03:49.152 INFO:teuthology.orchestra.run.smithi160.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 253 kB/s | 27 kB 00:00 2023-12-25T20:03:49.195 INFO:teuthology.orchestra.run.smithi160.stdout:(29/32): python3-waitress-1.4.3-1.el8.noarch.rp 2.7 MB/s | 255 kB 00:00 2023-12-25T20:03:49.236 INFO:teuthology.orchestra.run.smithi160.stdout:(30/32): python3-webtest-2.0.33-1.el8.noarch.rp 1.0 MB/s | 85 kB 00:00 2023-12-25T20:03:49.270 INFO:teuthology.orchestra.run.smithi160.stdout:(31/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 2.1 MB/s | 251 kB 00:00 2023-12-25T20:03:49.295 INFO:teuthology.orchestra.run.smithi160.stdout:(32/32): python3-zc-lockfile-2.0-2.el8.noarch.r 229 kB/s | 23 kB 00:00 2023-12-25T20:03:49.296 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2023-12-25T20:03:49.296 INFO:teuthology.orchestra.run.smithi160.stdout:Total 12 MB/s | 31 MB 00:02 2023-12-25T20:03:49.384 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2023-12-25T20:03:49.421 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2023-12-25T20:03:49.421 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2023-12-25T20:03:49.733 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2023-12-25T20:03:49.736 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2023-12-25T20:03:50.153 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-12-25T20:03:50.156 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-12-25T20:03:50.156 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2023-12-25T20:03:50.156 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-12-25T20:03:50.156 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2023-12-25T20:03:50.157 INFO:teuthology.orchestra.run.smithi050.stdout: ceph x86_64 2:16.2.4-0.el8 ceph 6.3 k 2023-12-25T20:03:50.157 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2023-12-25T20:03:50.157 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mds x86_64 2:16.2.4-0.el8 ceph 2.4 M 2023-12-25T20:03:50.157 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr x86_64 2:16.2.4-0.el8 ceph 1.6 M 2023-12-25T20:03:50.157 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-modules-core noarch 2:16.2.4-0.el8 ceph-noarch 220 k 2023-12-25T20:03:50.157 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mon x86_64 2:16.2.4-0.el8 ceph 5.0 M 2023-12-25T20:03:50.157 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-osd x86_64 2:16.2.4-0.el8 ceph 18 M 2023-12-25T20:03:50.157 INFO:teuthology.orchestra.run.smithi050.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-12-25T20:03:50.157 INFO:teuthology.orchestra.run.smithi050.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-12-25T20:03:50.157 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2023-12-25T20:03:50.157 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-12-25T20:03:50.157 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-12-25T20:03:50.157 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cryptography x86_64 3.2.1-6.el8 baseos 641 k 2023-12-25T20:03:50.158 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-12-25T20:03:50.158 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-12-25T20:03:50.158 INFO:teuthology.orchestra.run.smithi050.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-12-25T20:03:50.158 INFO:teuthology.orchestra.run.smithi050.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2023-12-25T20:03:50.158 INFO:teuthology.orchestra.run.smithi050.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2023-12-25T20:03:50.158 INFO:teuthology.orchestra.run.smithi050.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-12-25T20:03:50.158 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-12-25T20:03:50.158 INFO:teuthology.orchestra.run.smithi050.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-12-25T20:03:50.158 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2023-12-25T20:03:50.158 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2023-12-25T20:03:50.158 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2023-12-25T20:03:50.158 INFO:teuthology.orchestra.run.smithi050.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-12-25T20:03:50.158 INFO:teuthology.orchestra.run.smithi050.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-12-25T20:03:50.158 INFO:teuthology.orchestra.run.smithi050.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-12-25T20:03:50.159 INFO:teuthology.orchestra.run.smithi050.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-12-25T20:03:50.159 INFO:teuthology.orchestra.run.smithi050.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-12-25T20:03:50.159 INFO:teuthology.orchestra.run.smithi050.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-12-25T20:03:50.159 INFO:teuthology.orchestra.run.smithi050.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-12-25T20:03:50.159 INFO:teuthology.orchestra.run.smithi050.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2023-12-25T20:03:50.159 INFO:teuthology.orchestra.run.smithi050.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-12-25T20:03:50.159 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-25T20:03:50.159 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2023-12-25T20:03:50.159 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-12-25T20:03:50.159 INFO:teuthology.orchestra.run.smithi050.stdout:Install 32 Packages 2023-12-25T20:03:50.159 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-25T20:03:50.160 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 31 M 2023-12-25T20:03:50.160 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 111 M 2023-12-25T20:03:50.160 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2023-12-25T20:03:50.498 INFO:teuthology.orchestra.run.smithi050.stdout:(1/32): ceph-16.2.4-0.el8.x86_64.rpm 57 kB/s | 6.3 kB 00:00 2023-12-25T20:03:50.556 INFO:teuthology.orchestra.run.smithi050.stdout:(2/32): ceph-mds-16.2.4-0.el8.x86_64.rpm 14 MB/s | 2.4 MB 00:00 2023-12-25T20:03:50.590 INFO:teuthology.orchestra.run.smithi050.stdout:(3/32): ceph-mgr-16.2.4-0.el8.x86_64.rpm 7.7 MB/s | 1.6 MB 00:00 2023-12-25T20:03:50.640 INFO:teuthology.orchestra.run.smithi050.stdout:(4/32): ceph-mgr-modules-core-16.2.4-0.el8.noar 4.3 MB/s | 220 kB 00:00 2023-12-25T20:03:50.724 INFO:teuthology.orchestra.run.smithi050.stdout:(5/32): ceph-mon-16.2.4-0.el8.x86_64.rpm 22 MB/s | 5.0 MB 00:00 2023-12-25T20:03:50.896 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2023-12-25T20:03:50.991 INFO:teuthology.orchestra.run.smithi050.stdout:(6/32): ceph-osd-16.2.4-0.el8.x86_64.rpm 42 MB/s | 18 MB 00:00 2023-12-25T20:03:51.075 INFO:teuthology.orchestra.run.smithi050.stdout:(7/32): python3-mako-1.0.6-14.el8.noarch.rpm 362 kB/s | 157 kB 00:00 2023-12-25T20:03:51.101 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2023-12-25T20:03:51.300 INFO:teuthology.orchestra.run.smithi050.stdout:(8/32): python3-markupsafe-0.23-19.el8.x86_64.r 67 kB/s | 39 kB 00:00 2023-12-25T20:03:51.317 INFO:teuthology.orchestra.run.smithi050.stdout:(9/32): python3-pytz-2017.2-11.el8.noarch.rpm 236 kB/s | 57 kB 00:00 2023-12-25T20:03:51.362 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2023-12-25T20:03:51.375 INFO:teuthology.orchestra.run.smithi050.stdout:(10/32): python3-pyOpenSSL-19.0.0-1.el8.noarch. 269 kB/s | 103 kB 00:00 2023-12-25T20:03:51.542 INFO:teuthology.orchestra.run.smithi050.stdout:(11/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 1.1 MB/s | 249 kB 00:00 2023-12-25T20:03:51.603 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2023-12-25T20:03:51.609 INFO:teuthology.orchestra.run.smithi050.stdout:(12/32): python3-werkzeug-0.12.2-4.el8.noarch.r 1.4 MB/s | 457 kB 00:00 2023-12-25T20:03:51.635 INFO:teuthology.orchestra.run.smithi050.stdout:(13/32): python3-pycparser-2.14-14.el8.noarch.r 1.2 MB/s | 109 kB 00:00 2023-12-25T20:03:51.660 INFO:teuthology.orchestra.run.smithi050.stdout:(14/32): python3-cryptography-3.2.1-6.el8.x86_6 2.2 MB/s | 641 kB 00:00 2023-12-25T20:03:51.802 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2023-12-25T20:03:51.876 INFO:teuthology.orchestra.run.smithi050.stdout:(15/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 164 kB/s | 44 kB 00:00 2023-12-25T20:03:51.902 INFO:teuthology.orchestra.run.smithi050.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 693 kB/s | 185 kB 00:00 2023-12-25T20:03:51.927 INFO:teuthology.orchestra.run.smithi050.stdout:(17/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 671 kB/s | 173 kB 00:00 2023-12-25T20:03:51.944 INFO:teuthology.orchestra.run.smithi050.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 267 kB/s | 11 kB 00:00 2023-12-25T20:03:51.970 INFO:teuthology.orchestra.run.smithi050.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 420 kB/s | 18 kB 00:00 2023-12-25T20:03:51.986 INFO:teuthology.orchestra.run.smithi050.stdout:(20/32): python3-logutils-0.3.5-11.el8.noarch.r 1.1 MB/s | 49 kB 00:00 2023-12-25T20:03:51.991 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2023-12-25T20:03:52.003 INFO:teuthology.orchestra.run.smithi050.stdout:(21/32): python3-more-itertools-7.2.0-3.el8.noa 1.7 MB/s | 59 kB 00:00 2023-12-25T20:03:52.029 INFO:teuthology.orchestra.run.smithi050.stdout:(22/32): python3-cherrypy-18.4.0-1.el8.noarch.r 2.5 MB/s | 384 kB 00:00 2023-12-25T20:03:52.054 INFO:teuthology.orchestra.run.smithi050.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 310 kB/s | 16 kB 00:00 2023-12-25T20:03:52.079 INFO:teuthology.orchestra.run.smithi050.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 373 kB/s | 19 kB 00:00 2023-12-25T20:03:52.105 INFO:teuthology.orchestra.run.smithi050.stdout:(25/32): python3-singledispatch-3.4.0.3-18.el8. 476 kB/s | 24 kB 00:00 2023-12-25T20:03:52.146 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2023-12-25T20:03:52.164 INFO:teuthology.orchestra.run.smithi050.stdout:(26/32): python3-tempora-1.14.1-5.el8.noarch.rp 351 kB/s | 29 kB 00:00 2023-12-25T20:03:52.189 INFO:teuthology.orchestra.run.smithi050.stdout:(27/32): python3-trustme-0.6.0-4.el8.noarch.rpm 330 kB/s | 27 kB 00:00 2023-12-25T20:03:52.216 INFO:teuthology.orchestra.run.smithi050.stdout:(28/32): python3-pecan-1.3.2-9.el8.noarch.rpm 1.2 MB/s | 283 kB 00:00 2023-12-25T20:03:52.241 INFO:teuthology.orchestra.run.smithi050.stdout:(29/32): python3-waitress-1.4.3-1.el8.noarch.rp 3.2 MB/s | 255 kB 00:00 2023-12-25T20:03:52.266 INFO:teuthology.orchestra.run.smithi050.stdout:(30/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 3.2 MB/s | 251 kB 00:00 2023-12-25T20:03:52.291 INFO:teuthology.orchestra.run.smithi050.stdout:(31/32): python3-zc-lockfile-2.0-2.el8.noarch.r 460 kB/s | 23 kB 00:00 2023-12-25T20:03:52.317 INFO:teuthology.orchestra.run.smithi050.stdout:(32/32): python3-webtest-2.0.33-1.el8.noarch.rp 850 kB/s | 85 kB 00:00 2023-12-25T20:03:52.321 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2023-12-25T20:03:52.321 INFO:teuthology.orchestra.run.smithi050.stdout:Total 14 MB/s | 31 MB 00:02 2023-12-25T20:03:52.340 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2023-12-25T20:03:52.409 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2023-12-25T20:03:52.452 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2023-12-25T20:03:52.452 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2023-12-25T20:03:52.509 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2023-12-25T20:03:52.742 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2023-12-25T20:03:52.768 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2023-12-25T20:03:52.772 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2023-12-25T20:03:52.963 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2023-12-25T20:03:53.224 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2023-12-25T20:03:53.483 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2023-12-25T20:03:53.820 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2023-12-25T20:03:53.981 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2023-12-25T20:03:54.073 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-cryptography-3.2.1-6.el8.x86_64 14/32 2023-12-25T20:03:54.218 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2023-12-25T20:03:54.278 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2023-12-25T20:03:54.468 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2023-12-25T20:03:54.545 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2023-12-25T20:03:54.721 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-12-25T20:03:54.723 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2023-12-25T20:03:54.907 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2023-12-25T20:03:55.029 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2023-12-25T20:03:55.080 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2023-12-25T20:03:55.244 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2023-12-25T20:03:55.259 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2023-12-25T20:03:55.436 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2023-12-25T20:03:55.448 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2023-12-25T20:03:55.631 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2023-12-25T20:03:55.682 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2023-12-25T20:03:55.818 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2023-12-25T20:03:55.941 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2023-12-25T20:03:56.055 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2023-12-25T20:03:56.158 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2023-12-25T20:03:56.325 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2023-12-25T20:03:56.420 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2023-12-25T20:03:56.618 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2023-12-25T20:03:56.798 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2023-12-25T20:03:57.037 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2023-12-25T20:03:57.112 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2023-12-25T20:03:57.296 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-cryptography-3.2.1-6.el8.x86_64 14/32 2023-12-25T20:03:57.476 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2023-12-25T20:03:57.551 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 27/32 2023-12-25T20:03:57.741 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2023-12-25T20:03:57.771 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2023-12-25T20:03:57.935 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-12-25T20:03:58.289 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2023-12-25T20:03:58.507 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2023-12-25T20:03:58.696 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2023-12-25T20:03:58.922 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2023-12-25T20:03:59.215 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2023-12-25T20:03:59.299 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2023-12-25T20:03:59.448 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2023-12-25T20:03:59.527 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2023-12-25T20:03:59.705 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2023-12-25T20:03:59.960 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2023-12-25T20:04:00.080 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2023-12-25T20:04:00.170 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2023-12-25T20:04:00.390 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2023-12-25T20:04:00.393 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2023-12-25T20:04:00.595 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2023-12-25T20:04:00.613 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2023-12-25T20:04:00.744 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-2:16.2.4-0.el8.x86_64 32/32 2023-12-25T20:04:00.856 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 27/32 2023-12-25T20:04:01.100 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2023-12-25T20:04:01.141 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-2:16.2.4-0.el8.x86_64 32/32 2023-12-25T20:04:01.142 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-2:16.2.4-0.el8.x86_64 1/32 2023-12-25T20:04:01.142 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-mds-2:16.2.4-0.el8.x86_64 2/32 2023-12-25T20:04:01.142 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-mgr-2:16.2.4-0.el8.x86_64 3/32 2023-12-25T20:04:01.142 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-mon-2:16.2.4-0.el8.x86_64 4/32 2023-12-25T20:04:01.142 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-osd-2:16.2.4-0.el8.x86_64 5/32 2023-12-25T20:04:01.142 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 6/32 2023-12-25T20:04:01.142 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2023-12-25T20:04:01.142 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2023-12-25T20:04:01.142 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2023-12-25T20:04:01.142 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2023-12-25T20:04:01.142 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2023-12-25T20:04:01.142 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2023-12-25T20:04:01.142 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-cryptography-3.2.1-6.el8.x86_64 13/32 2023-12-25T20:04:01.143 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2023-12-25T20:04:01.143 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2023-12-25T20:04:01.143 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2023-12-25T20:04:01.143 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-12-25T20:04:01.143 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2023-12-25T20:04:01.143 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2023-12-25T20:04:01.143 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2023-12-25T20:04:01.144 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2023-12-25T20:04:01.144 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2023-12-25T20:04:01.144 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2023-12-25T20:04:01.144 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2023-12-25T20:04:01.144 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2023-12-25T20:04:01.144 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2023-12-25T20:04:01.144 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2023-12-25T20:04:01.144 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2023-12-25T20:04:01.144 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2023-12-25T20:04:01.144 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2023-12-25T20:04:01.144 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2023-12-25T20:04:02.253 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2023-12-25T20:04:02.253 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:04:02.254 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2023-12-25T20:04:02.254 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-2:16.2.4-0.el8.x86_64 2023-12-25T20:04:02.254 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mds-2:16.2.4-0.el8.x86_64 2023-12-25T20:04:02.254 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr-2:16.2.4-0.el8.x86_64 2023-12-25T20:04:02.254 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 2023-12-25T20:04:02.254 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mon-2:16.2.4-0.el8.x86_64 2023-12-25T20:04:02.254 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-osd-2:16.2.4-0.el8.x86_64 2023-12-25T20:04:02.254 INFO:teuthology.orchestra.run.smithi160.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-12-25T20:04:02.254 INFO:teuthology.orchestra.run.smithi160.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-12-25T20:04:02.254 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cffi-1.11.5-6.el8.x86_64 2023-12-25T20:04:02.254 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-12-25T20:04:02.254 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-12-25T20:04:02.254 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cryptography-3.2.1-6.el8.x86_64 2023-12-25T20:04:02.254 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jaraco-6.2-6.el8.noarch 2023-12-25T20:04:02.255 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-12-25T20:04:02.255 INFO:teuthology.orchestra.run.smithi160.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-12-25T20:04:02.255 INFO:teuthology.orchestra.run.smithi160.stdout: python3-mako-1.0.6-14.el8.noarch 2023-12-25T20:04:02.255 INFO:teuthology.orchestra.run.smithi160.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-12-25T20:04:02.255 INFO:teuthology.orchestra.run.smithi160.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-12-25T20:04:02.255 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-12-25T20:04:02.255 INFO:teuthology.orchestra.run.smithi160.stdout: python3-portend-2.6-1.el8.noarch 2023-12-25T20:04:02.255 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-12-25T20:04:02.255 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pycparser-2.14-14.el8.noarch 2023-12-25T20:04:02.255 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pytz-2017.2-11.el8.noarch 2023-12-25T20:04:02.255 INFO:teuthology.orchestra.run.smithi160.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-12-25T20:04:02.255 INFO:teuthology.orchestra.run.smithi160.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-12-25T20:04:02.255 INFO:teuthology.orchestra.run.smithi160.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-12-25T20:04:02.255 INFO:teuthology.orchestra.run.smithi160.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-12-25T20:04:02.255 INFO:teuthology.orchestra.run.smithi160.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-12-25T20:04:02.256 INFO:teuthology.orchestra.run.smithi160.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-12-25T20:04:02.256 INFO:teuthology.orchestra.run.smithi160.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-12-25T20:04:02.256 INFO:teuthology.orchestra.run.smithi160.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-12-25T20:04:02.256 INFO:teuthology.orchestra.run.smithi160.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-12-25T20:04:02.256 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:04:02.256 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2023-12-25T20:04:02.539 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install ceph-base 2023-12-25T20:04:02.738 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2023-12-25T20:04:02.917 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2023-12-25T20:04:03.379 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2023-12-25T20:04:03.403 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:01:14 ago on Mon 25 Dec 2023 08:02:49 PM UTC. 2023-12-25T20:04:03.617 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2023-12-25T20:04:03.853 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2023-12-25T20:04:04.033 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2023-12-25T20:04:04.051 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2023-12-25T20:04:04.216 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-2:16.2.4-0.el8.x86_64 32/32 2023-12-25T20:04:04.650 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-2:16.2.4-0.el8.x86_64 32/32 2023-12-25T20:04:04.650 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-2:16.2.4-0.el8.x86_64 1/32 2023-12-25T20:04:04.650 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mds-2:16.2.4-0.el8.x86_64 2/32 2023-12-25T20:04:04.650 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mgr-2:16.2.4-0.el8.x86_64 3/32 2023-12-25T20:04:04.650 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mon-2:16.2.4-0.el8.x86_64 4/32 2023-12-25T20:04:04.650 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-osd-2:16.2.4-0.el8.x86_64 5/32 2023-12-25T20:04:04.650 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 6/32 2023-12-25T20:04:04.650 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2023-12-25T20:04:04.650 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2023-12-25T20:04:04.650 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2023-12-25T20:04:04.651 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2023-12-25T20:04:04.651 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2023-12-25T20:04:04.651 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2023-12-25T20:04:04.651 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-cryptography-3.2.1-6.el8.x86_64 13/32 2023-12-25T20:04:04.651 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2023-12-25T20:04:04.651 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2023-12-25T20:04:04.651 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2023-12-25T20:04:04.651 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-12-25T20:04:04.651 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2023-12-25T20:04:04.651 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2023-12-25T20:04:04.651 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2023-12-25T20:04:04.651 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2023-12-25T20:04:04.651 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2023-12-25T20:04:04.651 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2023-12-25T20:04:04.651 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2023-12-25T20:04:04.652 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2023-12-25T20:04:04.652 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2023-12-25T20:04:04.652 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2023-12-25T20:04:04.652 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2023-12-25T20:04:04.652 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2023-12-25T20:04:04.652 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2023-12-25T20:04:04.652 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2023-12-25T20:04:04.868 INFO:teuthology.orchestra.run.smithi160.stdout:Package ceph-base-2:16.2.4-0.el8.x86_64 is already installed. 2023-12-25T20:04:05.102 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2023-12-25T20:04:05.103 INFO:teuthology.orchestra.run.smithi160.stdout:Nothing to do. 2023-12-25T20:04:05.103 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2023-12-25T20:04:05.218 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install cephadm 2023-12-25T20:04:05.947 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2023-12-25T20:04:05.947 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-25T20:04:05.947 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2023-12-25T20:04:05.947 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-2:16.2.4-0.el8.x86_64 2023-12-25T20:04:05.947 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mds-2:16.2.4-0.el8.x86_64 2023-12-25T20:04:05.947 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-2:16.2.4-0.el8.x86_64 2023-12-25T20:04:05.948 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 2023-12-25T20:04:05.948 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mon-2:16.2.4-0.el8.x86_64 2023-12-25T20:04:05.948 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-osd-2:16.2.4-0.el8.x86_64 2023-12-25T20:04:05.948 INFO:teuthology.orchestra.run.smithi050.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-12-25T20:04:05.948 INFO:teuthology.orchestra.run.smithi050.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-12-25T20:04:05.948 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cffi-1.11.5-6.el8.x86_64 2023-12-25T20:04:05.948 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-12-25T20:04:05.948 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-12-25T20:04:05.948 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cryptography-3.2.1-6.el8.x86_64 2023-12-25T20:04:05.948 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco-6.2-6.el8.noarch 2023-12-25T20:04:05.948 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-12-25T20:04:05.948 INFO:teuthology.orchestra.run.smithi050.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-12-25T20:04:05.948 INFO:teuthology.orchestra.run.smithi050.stdout: python3-mako-1.0.6-14.el8.noarch 2023-12-25T20:04:05.948 INFO:teuthology.orchestra.run.smithi050.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-12-25T20:04:05.949 INFO:teuthology.orchestra.run.smithi050.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-12-25T20:04:05.949 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-12-25T20:04:05.949 INFO:teuthology.orchestra.run.smithi050.stdout: python3-portend-2.6-1.el8.noarch 2023-12-25T20:04:05.949 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-12-25T20:04:05.949 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pycparser-2.14-14.el8.noarch 2023-12-25T20:04:05.949 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pytz-2017.2-11.el8.noarch 2023-12-25T20:04:05.949 INFO:teuthology.orchestra.run.smithi050.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-12-25T20:04:05.949 INFO:teuthology.orchestra.run.smithi050.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-12-25T20:04:05.949 INFO:teuthology.orchestra.run.smithi050.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-12-25T20:04:05.949 INFO:teuthology.orchestra.run.smithi050.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-12-25T20:04:05.949 INFO:teuthology.orchestra.run.smithi050.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-12-25T20:04:05.949 INFO:teuthology.orchestra.run.smithi050.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-12-25T20:04:05.949 INFO:teuthology.orchestra.run.smithi050.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-12-25T20:04:05.949 INFO:teuthology.orchestra.run.smithi050.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-12-25T20:04:05.949 INFO:teuthology.orchestra.run.smithi050.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-12-25T20:04:05.950 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-25T20:04:05.950 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-12-25T20:04:06.080 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:01:17 ago on Mon 25 Dec 2023 08:02:49 PM UTC. 2023-12-25T20:04:06.889 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-base 2023-12-25T20:04:07.792 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:16 ago on Mon 25 Dec 2023 08:02:51 PM UTC. 2023-12-25T20:04:07.793 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2023-12-25T20:04:07.794 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-12-25T20:04:07.794 INFO:teuthology.orchestra.run.smithi160.stdout: Package Architecture Version Repository Size 2023-12-25T20:04:07.794 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-12-25T20:04:07.794 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2023-12-25T20:04:07.794 INFO:teuthology.orchestra.run.smithi160.stdout: cephadm noarch 2:16.2.4-0.el8 ceph-noarch 71 k 2023-12-25T20:04:07.794 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:04:07.794 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2023-12-25T20:04:07.795 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-12-25T20:04:07.795 INFO:teuthology.orchestra.run.smithi160.stdout:Install 1 Package 2023-12-25T20:04:07.795 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:04:07.795 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 71 k 2023-12-25T20:04:07.795 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 290 k 2023-12-25T20:04:07.795 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2023-12-25T20:04:07.848 INFO:teuthology.orchestra.run.smithi160.stdout:cephadm-16.2.4-0.el8.noarch.rpm 1.3 MB/s | 71 kB 00:00 2023-12-25T20:04:07.848 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2023-12-25T20:04:07.849 INFO:teuthology.orchestra.run.smithi160.stdout:Total 1.3 MB/s | 71 kB 00:00 2023-12-25T20:04:07.849 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2023-12-25T20:04:07.856 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2023-12-25T20:04:07.856 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2023-12-25T20:04:07.857 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2023-12-25T20:04:07.861 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2023-12-25T20:04:07.902 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2023-12-25T20:04:08.379 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2023-12-25T20:04:09.099 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : cephadm-2:16.2.4-0.el8.noarch 1/1 2023-12-25T20:04:09.338 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2023-12-25T20:04:09.350 INFO:teuthology.orchestra.run.smithi050.stdout:Package ceph-base-2:16.2.4-0.el8.x86_64 is already installed. 2023-12-25T20:04:09.595 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-12-25T20:04:09.596 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2023-12-25T20:04:09.596 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-12-25T20:04:09.685 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install cephadm 2023-12-25T20:04:10.039 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : cephadm-2:16.2.4-0.el8.noarch 1/1 2023-12-25T20:04:10.039 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:04:10.039 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2023-12-25T20:04:10.039 INFO:teuthology.orchestra.run.smithi160.stdout: cephadm-2:16.2.4-0.el8.noarch 2023-12-25T20:04:10.040 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:04:10.040 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2023-12-25T20:04:10.246 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install ceph-immutable-object-cache 2023-12-25T20:04:10.580 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:19 ago on Mon 25 Dec 2023 08:02:51 PM UTC. 2023-12-25T20:04:11.112 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:01:22 ago on Mon 25 Dec 2023 08:02:49 PM UTC. 2023-12-25T20:04:12.387 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-12-25T20:04:12.388 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-12-25T20:04:12.388 INFO:teuthology.orchestra.run.smithi050.stdout: Package Architecture Version Repository Size 2023-12-25T20:04:12.388 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-12-25T20:04:12.388 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2023-12-25T20:04:12.388 INFO:teuthology.orchestra.run.smithi050.stdout: cephadm noarch 2:16.2.4-0.el8 ceph-noarch 71 k 2023-12-25T20:04:12.388 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-25T20:04:12.388 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2023-12-25T20:04:12.388 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-12-25T20:04:12.388 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2023-12-25T20:04:12.388 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-25T20:04:12.389 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 71 k 2023-12-25T20:04:12.389 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 290 k 2023-12-25T20:04:12.389 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2023-12-25T20:04:12.753 INFO:teuthology.orchestra.run.smithi050.stdout:cephadm-16.2.4-0.el8.noarch.rpm 196 kB/s | 71 kB 00:00 2023-12-25T20:04:12.754 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2023-12-25T20:04:12.754 INFO:teuthology.orchestra.run.smithi050.stdout:Total 195 kB/s | 71 kB 00:00 2023-12-25T20:04:12.754 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2023-12-25T20:04:12.762 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2023-12-25T20:04:12.762 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2023-12-25T20:04:12.765 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2023-12-25T20:04:12.768 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2023-12-25T20:04:12.805 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2023-12-25T20:04:12.806 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-12-25T20:04:12.806 INFO:teuthology.orchestra.run.smithi160.stdout: Package Arch Version Repo Size 2023-12-25T20:04:12.806 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-12-25T20:04:12.806 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2023-12-25T20:04:12.806 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-immutable-object-cache x86_64 2:16.2.4-0.el8 ceph 163 k 2023-12-25T20:04:12.806 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:04:12.807 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2023-12-25T20:04:12.807 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-12-25T20:04:12.807 INFO:teuthology.orchestra.run.smithi160.stdout:Install 1 Package 2023-12-25T20:04:12.807 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:04:12.807 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 163 k 2023-12-25T20:04:12.807 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 587 k 2023-12-25T20:04:12.807 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2023-12-25T20:04:12.821 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2023-12-25T20:04:12.852 INFO:teuthology.orchestra.run.smithi160.stdout:ceph-immutable-object-cache-16.2.4-0.el8.x86_64 3.5 MB/s | 163 kB 00:00 2023-12-25T20:04:12.853 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2023-12-25T20:04:12.853 INFO:teuthology.orchestra.run.smithi160.stdout:Total 3.4 MB/s | 163 kB 00:00 2023-12-25T20:04:12.853 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2023-12-25T20:04:12.864 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2023-12-25T20:04:12.864 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2023-12-25T20:04:12.910 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2023-12-25T20:04:12.913 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2023-12-25T20:04:13.025 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2023-12-25T20:04:13.243 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2023-12-25T20:04:13.382 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2023-12-25T20:04:13.537 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : cephadm-2:16.2.4-0.el8.noarch 1/1 2023-12-25T20:04:13.640 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2023-12-25T20:04:13.784 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2023-12-25T20:04:14.335 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2023-12-25T20:04:14.335 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:04:14.335 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2023-12-25T20:04:14.335 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 2023-12-25T20:04:14.335 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:04:14.335 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2023-12-25T20:04:14.501 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : cephadm-2:16.2.4-0.el8.noarch 1/1 2023-12-25T20:04:14.501 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-25T20:04:14.501 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2023-12-25T20:04:14.502 INFO:teuthology.orchestra.run.smithi050.stdout: cephadm-2:16.2.4-0.el8.noarch 2023-12-25T20:04:14.502 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-25T20:04:14.502 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-12-25T20:04:14.709 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-immutable-object-cache 2023-12-25T20:04:15.608 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:24 ago on Mon 25 Dec 2023 08:02:51 PM UTC. 2023-12-25T20:04:15.917 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install ceph-mgr 2023-12-25T20:04:16.781 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:01:27 ago on Mon 25 Dec 2023 08:02:49 PM UTC. 2023-12-25T20:04:17.397 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-12-25T20:04:17.398 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-12-25T20:04:17.398 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repo Size 2023-12-25T20:04:17.398 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-12-25T20:04:17.398 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2023-12-25T20:04:17.398 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-immutable-object-cache x86_64 2:16.2.4-0.el8 ceph 163 k 2023-12-25T20:04:17.398 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-25T20:04:17.398 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2023-12-25T20:04:17.398 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-12-25T20:04:17.399 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2023-12-25T20:04:17.399 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-25T20:04:17.399 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 163 k 2023-12-25T20:04:17.399 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 587 k 2023-12-25T20:04:17.399 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2023-12-25T20:04:17.444 INFO:teuthology.orchestra.run.smithi050.stdout:ceph-immutable-object-cache-16.2.4-0.el8.x86_64 3.6 MB/s | 163 kB 00:00 2023-12-25T20:04:17.444 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2023-12-25T20:04:17.444 INFO:teuthology.orchestra.run.smithi050.stdout:Total 3.5 MB/s | 163 kB 00:00 2023-12-25T20:04:17.445 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2023-12-25T20:04:17.456 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2023-12-25T20:04:17.456 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2023-12-25T20:04:17.508 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2023-12-25T20:04:17.511 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2023-12-25T20:04:18.228 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2023-12-25T20:04:18.244 INFO:teuthology.orchestra.run.smithi160.stdout:Package ceph-mgr-2:16.2.4-0.el8.x86_64 is already installed. 2023-12-25T20:04:18.474 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2023-12-25T20:04:18.475 INFO:teuthology.orchestra.run.smithi160.stdout:Nothing to do. 2023-12-25T20:04:18.475 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2023-12-25T20:04:18.586 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install ceph-mgr-dashboard 2023-12-25T20:04:19.008 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2023-12-25T20:04:19.443 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:01:30 ago on Mon 25 Dec 2023 08:02:49 PM UTC. 2023-12-25T20:04:19.447 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2023-12-25T20:04:20.239 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2023-12-25T20:04:20.239 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-25T20:04:20.239 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2023-12-25T20:04:20.239 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 2023-12-25T20:04:20.239 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-25T20:04:20.239 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-12-25T20:04:20.475 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-mgr 2023-12-25T20:04:21.138 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2023-12-25T20:04:21.139 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-12-25T20:04:21.139 INFO:teuthology.orchestra.run.smithi160.stdout: Package Arch Version Repository Size 2023-12-25T20:04:21.139 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-12-25T20:04:21.139 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2023-12-25T20:04:21.139 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr-dashboard noarch 2:16.2.4-0.el8 ceph-noarch 2.7 M 2023-12-25T20:04:21.139 INFO:teuthology.orchestra.run.smithi160.stdout:Installing dependencies: 2023-12-25T20:04:21.139 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-grafana-dashboards noarch 2:16.2.4-0.el8 ceph-noarch 24 k 2023-12-25T20:04:21.139 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-prometheus-alerts noarch 2:16.2.4-0.el8 ceph-noarch 8.9 k 2023-12-25T20:04:21.139 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2023-12-25T20:04:21.140 INFO:teuthology.orchestra.run.smithi160.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-12-25T20:04:21.140 INFO:teuthology.orchestra.run.smithi160.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-12-25T20:04:21.140 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:04:21.140 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2023-12-25T20:04:21.140 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-12-25T20:04:21.140 INFO:teuthology.orchestra.run.smithi160.stdout:Install 6 Packages 2023-12-25T20:04:21.140 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:04:21.141 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 3.0 M 2023-12-25T20:04:21.141 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 15 M 2023-12-25T20:04:21.141 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2023-12-25T20:04:21.291 INFO:teuthology.orchestra.run.smithi160.stdout:(1/6): ceph-grafana-dashboards-16.2.4-0.el8.noa 312 kB/s | 24 kB 00:00 2023-12-25T20:04:21.314 INFO:teuthology.orchestra.run.smithi160.stdout:(2/6): ceph-prometheus-alerts-16.2.4-0.el8.noar 89 kB/s | 8.9 kB 00:00 2023-12-25T20:04:21.367 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:30 ago on Mon 25 Dec 2023 08:02:51 PM UTC. 2023-12-25T20:04:21.381 INFO:teuthology.orchestra.run.smithi160.stdout:(3/6): ceph-mgr-dashboard-16.2.4-0.el8.noarch.r 16 MB/s | 2.7 MB 00:00 2023-12-25T20:04:21.506 INFO:teuthology.orchestra.run.smithi160.stdout:(4/6): python3-jwt-1.6.1-2.el8.noarch.rpm 198 kB/s | 43 kB 00:00 2023-12-25T20:04:22.256 INFO:teuthology.orchestra.run.smithi160.stdout:(5/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 36 kB/s | 34 kB 00:00 2023-12-25T20:04:22.781 INFO:teuthology.orchestra.run.smithi160.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 140 kB/s | 196 kB 00:01 2023-12-25T20:04:22.785 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2023-12-25T20:04:22.785 INFO:teuthology.orchestra.run.smithi160.stdout:Total 1.8 MB/s | 3.0 MB 00:01 2023-12-25T20:04:22.797 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2023-12-25T20:04:22.808 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2023-12-25T20:04:22.808 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2023-12-25T20:04:22.918 INFO:teuthology.orchestra.run.smithi050.stdout:Package ceph-mgr-2:16.2.4-0.el8.x86_64 is already installed. 2023-12-25T20:04:22.956 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2023-12-25T20:04:22.959 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2023-12-25T20:04:23.166 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-12-25T20:04:23.167 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2023-12-25T20:04:23.167 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-12-25T20:04:23.262 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2023-12-25T20:04:23.266 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-mgr-dashboard 2023-12-25T20:04:23.497 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2023-12-25T20:04:23.734 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2023-12-25T20:04:23.976 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2023-12-25T20:04:24.106 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 4/6 2023-12-25T20:04:24.159 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:33 ago on Mon 25 Dec 2023 08:02:51 PM UTC. 2023-12-25T20:04:25.120 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 5/6 2023-12-25T20:04:25.416 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2023-12-25T20:04:25.690 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2023-12-25T20:04:25.690 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 1/6 2023-12-25T20:04:25.690 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2/6 2023-12-25T20:04:25.690 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 3/6 2023-12-25T20:04:25.691 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2023-12-25T20:04:25.691 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2023-12-25T20:04:25.978 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-12-25T20:04:25.979 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-12-25T20:04:25.979 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2023-12-25T20:04:25.979 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-12-25T20:04:25.979 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2023-12-25T20:04:25.979 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-dashboard noarch 2:16.2.4-0.el8 ceph-noarch 2.7 M 2023-12-25T20:04:25.979 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2023-12-25T20:04:25.980 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-grafana-dashboards noarch 2:16.2.4-0.el8 ceph-noarch 24 k 2023-12-25T20:04:25.980 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-prometheus-alerts noarch 2:16.2.4-0.el8 ceph-noarch 8.9 k 2023-12-25T20:04:25.980 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2023-12-25T20:04:25.980 INFO:teuthology.orchestra.run.smithi050.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-12-25T20:04:25.980 INFO:teuthology.orchestra.run.smithi050.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-12-25T20:04:25.980 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-25T20:04:25.980 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2023-12-25T20:04:25.980 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-12-25T20:04:25.980 INFO:teuthology.orchestra.run.smithi050.stdout:Install 6 Packages 2023-12-25T20:04:25.980 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-25T20:04:25.981 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 3.0 M 2023-12-25T20:04:25.981 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 15 M 2023-12-25T20:04:25.981 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2023-12-25T20:04:26.281 INFO:teuthology.orchestra.run.smithi050.stdout:(1/6): ceph-grafana-dashboards-16.2.4-0.el8.noa 514 kB/s | 24 kB 00:00 2023-12-25T20:04:26.306 INFO:teuthology.orchestra.run.smithi050.stdout:(2/6): ceph-prometheus-alerts-16.2.4-0.el8.noar 125 kB/s | 8.9 kB 00:00 2023-12-25T20:04:26.373 INFO:teuthology.orchestra.run.smithi050.stdout:(3/6): ceph-mgr-dashboard-16.2.4-0.el8.noarch.r 19 MB/s | 2.7 MB 00:00 2023-12-25T20:04:26.390 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2023-12-25T20:04:26.390 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:04:26.391 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2023-12-25T20:04:26.391 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 2023-12-25T20:04:26.391 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2023-12-25T20:04:26.391 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 2023-12-25T20:04:26.391 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-12-25T20:04:26.391 INFO:teuthology.orchestra.run.smithi160.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-12-25T20:04:26.391 INFO:teuthology.orchestra.run.smithi160.stdout: python3-routes-2.4.1-12.el8.noarch 2023-12-25T20:04:26.391 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:04:26.391 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2023-12-25T20:04:26.448 INFO:teuthology.orchestra.run.smithi050.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 237 kB/s | 34 kB 00:00 2023-12-25T20:04:26.482 INFO:teuthology.orchestra.run.smithi050.stdout:(5/6): python3-jwt-1.6.1-2.el8.noarch.rpm 212 kB/s | 43 kB 00:00 2023-12-25T20:04:26.548 INFO:teuthology.orchestra.run.smithi050.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 1.1 MB/s | 196 kB 00:00 2023-12-25T20:04:26.552 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2023-12-25T20:04:26.552 INFO:teuthology.orchestra.run.smithi050.stdout:Total 5.2 MB/s | 3.0 MB 00:00 2023-12-25T20:04:26.562 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2023-12-25T20:04:26.572 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2023-12-25T20:04:26.573 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2023-12-25T20:04:26.620 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install ceph-mgr-diskprediction-local 2023-12-25T20:04:26.722 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2023-12-25T20:04:26.725 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2023-12-25T20:04:27.026 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2023-12-25T20:04:27.262 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2023-12-25T20:04:27.477 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:01:38 ago on Mon 25 Dec 2023 08:02:49 PM UTC. 2023-12-25T20:04:27.540 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2023-12-25T20:04:27.798 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2023-12-25T20:04:27.911 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 4/6 2023-12-25T20:04:28.992 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 5/6 2023-12-25T20:04:29.170 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2023-12-25T20:04:29.171 INFO:teuthology.orchestra.run.smithi160.stdout:============================================================================================ 2023-12-25T20:04:29.171 INFO:teuthology.orchestra.run.smithi160.stdout: Package Arch Version Repository Size 2023-12-25T20:04:29.171 INFO:teuthology.orchestra.run.smithi160.stdout:============================================================================================ 2023-12-25T20:04:29.171 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2023-12-25T20:04:29.171 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.4-0.el8 ceph-noarch 8.6 M 2023-12-25T20:04:29.171 INFO:teuthology.orchestra.run.smithi160.stdout:Installing dependencies: 2023-12-25T20:04:29.171 INFO:teuthology.orchestra.run.smithi160.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2023-12-25T20:04:29.172 INFO:teuthology.orchestra.run.smithi160.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2023-12-25T20:04:29.172 INFO:teuthology.orchestra.run.smithi160.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2023-12-25T20:04:29.172 INFO:teuthology.orchestra.run.smithi160.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2023-12-25T20:04:29.172 INFO:teuthology.orchestra.run.smithi160.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2023-12-25T20:04:29.172 INFO:teuthology.orchestra.run.smithi160.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2023-12-25T20:04:29.172 INFO:teuthology.orchestra.run.smithi160.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2023-12-25T20:04:29.172 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:04:29.172 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2023-12-25T20:04:29.172 INFO:teuthology.orchestra.run.smithi160.stdout:============================================================================================ 2023-12-25T20:04:29.172 INFO:teuthology.orchestra.run.smithi160.stdout:Install 8 Packages 2023-12-25T20:04:29.173 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:04:29.173 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 37 M 2023-12-25T20:04:29.173 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 210 M 2023-12-25T20:04:29.173 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2023-12-25T20:04:29.455 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2023-12-25T20:04:29.547 INFO:teuthology.orchestra.run.smithi160.stdout:(1/8): ceph-mgr-diskprediction-local-16.2.4-0.e 61 MB/s | 8.6 MB 00:00 2023-12-25T20:04:29.840 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2023-12-25T20:04:29.841 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 1/6 2023-12-25T20:04:29.841 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2/6 2023-12-25T20:04:29.841 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 3/6 2023-12-25T20:04:29.841 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2023-12-25T20:04:29.841 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2023-12-25T20:04:30.281 INFO:teuthology.orchestra.run.smithi160.stdout:(2/8): openblas-0.3.15-4.el8.x86_64.rpm 5.4 MB/s | 4.7 MB 00:00 2023-12-25T20:04:30.356 INFO:teuthology.orchestra.run.smithi160.stdout:(3/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 5.1 MB/s | 4.9 MB 00:00 2023-12-25T20:04:30.381 INFO:teuthology.orchestra.run.smithi160.stdout:(4/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 2.2 MB/s | 225 kB 00:00 2023-12-25T20:04:30.431 INFO:teuthology.orchestra.run.smithi160.stdout:(5/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 4.2 MB/s | 3.7 MB 00:00 2023-12-25T20:04:30.584 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2023-12-25T20:04:30.584 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-25T20:04:30.585 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2023-12-25T20:04:30.585 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 2023-12-25T20:04:30.585 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2023-12-25T20:04:30.585 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 2023-12-25T20:04:30.585 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-12-25T20:04:30.585 INFO:teuthology.orchestra.run.smithi050.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-12-25T20:04:30.585 INFO:teuthology.orchestra.run.smithi050.stdout: python3-routes-2.4.1-12.el8.noarch 2023-12-25T20:04:30.585 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-25T20:04:30.585 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-12-25T20:04:30.607 INFO:teuthology.orchestra.run.smithi160.stdout:(6/8): libquadmath-8.5.0-21.el8.x86_64.rpm 985 kB/s | 172 kB 00:00 2023-12-25T20:04:30.720 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-mgr-diskprediction-local 2023-12-25T20:04:31.299 INFO:teuthology.orchestra.run.smithi160.stdout:(7/8): python3-scipy-1.0.0-21.module_el8+762+77 15 MB/s | 14 MB 00:00 2023-12-25T20:04:31.613 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:40 ago on Mon 25 Dec 2023 08:02:51 PM UTC. 2023-12-25T20:04:32.279 INFO:teuthology.orchestra.run.smithi160.stdout:(8/8): libgfortran-8.5.0-21.el8.x86_64.rpm 340 kB/s | 645 kB 00:01 2023-12-25T20:04:32.279 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2023-12-25T20:04:32.280 INFO:teuthology.orchestra.run.smithi160.stdout:Total 12 MB/s | 37 MB 00:03 2023-12-25T20:04:32.517 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2023-12-25T20:04:32.536 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2023-12-25T20:04:32.536 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2023-12-25T20:04:32.867 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2023-12-25T20:04:32.870 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2023-12-25T20:04:33.454 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-12-25T20:04:33.455 INFO:teuthology.orchestra.run.smithi050.stdout:============================================================================================ 2023-12-25T20:04:33.456 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2023-12-25T20:04:33.456 INFO:teuthology.orchestra.run.smithi050.stdout:============================================================================================ 2023-12-25T20:04:33.456 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2023-12-25T20:04:33.456 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.4-0.el8 ceph-noarch 8.6 M 2023-12-25T20:04:33.456 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2023-12-25T20:04:33.456 INFO:teuthology.orchestra.run.smithi050.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2023-12-25T20:04:33.456 INFO:teuthology.orchestra.run.smithi050.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2023-12-25T20:04:33.457 INFO:teuthology.orchestra.run.smithi050.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2023-12-25T20:04:33.457 INFO:teuthology.orchestra.run.smithi050.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2023-12-25T20:04:33.457 INFO:teuthology.orchestra.run.smithi050.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2023-12-25T20:04:33.457 INFO:teuthology.orchestra.run.smithi050.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2023-12-25T20:04:33.457 INFO:teuthology.orchestra.run.smithi050.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2023-12-25T20:04:33.457 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-25T20:04:33.457 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2023-12-25T20:04:33.457 INFO:teuthology.orchestra.run.smithi050.stdout:============================================================================================ 2023-12-25T20:04:33.458 INFO:teuthology.orchestra.run.smithi050.stdout:Install 8 Packages 2023-12-25T20:04:33.458 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-25T20:04:33.459 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 37 M 2023-12-25T20:04:33.459 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 210 M 2023-12-25T20:04:33.459 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2023-12-25T20:04:33.557 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2023-12-25T20:04:33.698 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2023-12-25T20:04:33.738 INFO:teuthology.orchestra.run.smithi050.stdout:(1/8): ceph-mgr-diskprediction-local-16.2.4-0.e 43 MB/s | 8.6 MB 00:00 2023-12-25T20:04:33.835 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2023-12-25T20:04:34.006 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2023-12-25T20:04:34.479 INFO:teuthology.orchestra.run.smithi050.stdout:(2/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 5.2 MB/s | 4.9 MB 00:00 2023-12-25T20:04:34.555 INFO:teuthology.orchestra.run.smithi050.stdout:(3/8): openblas-0.3.15-4.el8.x86_64.rpm 4.6 MB/s | 4.7 MB 00:01 2023-12-25T20:04:34.581 INFO:teuthology.orchestra.run.smithi050.stdout:(4/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 2.2 MB/s | 225 kB 00:00 2023-12-25T20:04:34.589 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2023-12-25T20:04:34.664 INFO:teuthology.orchestra.run.smithi050.stdout:(5/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 4.0 MB/s | 3.7 MB 00:00 2023-12-25T20:04:34.789 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2023-12-25T20:04:34.848 INFO:teuthology.orchestra.run.smithi050.stdout:(6/8): libquadmath-8.5.0-21.el8.x86_64.rpm 941 kB/s | 172 kB 00:00 2023-12-25T20:04:35.407 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2023-12-25T20:04:35.572 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-12-25T20:04:35.874 INFO:teuthology.orchestra.run.smithi050.stdout:(7/8): python3-scipy-1.0.0-21.module_el8+762+77 11 MB/s | 14 MB 00:01 2023-12-25T20:04:36.058 INFO:teuthology.orchestra.run.smithi050.stdout:(8/8): libgfortran-8.5.0-21.el8.x86_64.rpm 437 kB/s | 645 kB 00:01 2023-12-25T20:04:36.058 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2023-12-25T20:04:36.058 INFO:teuthology.orchestra.run.smithi050.stdout:Total 14 MB/s | 37 MB 00:02 2023-12-25T20:04:36.295 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-12-25T20:04:36.300 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2023-12-25T20:04:36.320 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2023-12-25T20:04:36.320 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2023-12-25T20:04:36.634 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2023-12-25T20:04:36.678 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2023-12-25T20:04:36.681 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2023-12-25T20:04:36.898 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-12-25T20:04:37.363 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2023-12-25T20:04:37.469 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2023-12-25T20:04:37.603 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2023-12-25T20:04:37.752 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2023-12-25T20:04:38.367 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2023-12-25T20:04:38.578 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2023-12-25T20:04:38.674 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-12-25T20:04:39.229 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2023-12-25T20:04:39.386 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-12-25T20:04:39.977 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 7/8 2023-12-25T20:04:40.128 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-12-25T20:04:40.195 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2023-12-25T20:04:40.477 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2023-12-25T20:04:40.531 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2023-12-25T20:04:40.531 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 1/8 2023-12-25T20:04:40.531 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2023-12-25T20:04:40.531 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2023-12-25T20:04:40.531 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2023-12-25T20:04:40.531 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2023-12-25T20:04:40.531 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 6/8 2023-12-25T20:04:40.531 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2023-12-25T20:04:40.763 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-12-25T20:04:41.266 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2023-12-25T20:04:41.266 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:04:41.266 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2023-12-25T20:04:41.266 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 2023-12-25T20:04:41.266 INFO:teuthology.orchestra.run.smithi160.stdout: libgfortran-8.5.0-21.el8.x86_64 2023-12-25T20:04:41.266 INFO:teuthology.orchestra.run.smithi160.stdout: libquadmath-8.5.0-21.el8.x86_64 2023-12-25T20:04:41.266 INFO:teuthology.orchestra.run.smithi160.stdout: openblas-0.3.15-4.el8.x86_64 2023-12-25T20:04:41.266 INFO:teuthology.orchestra.run.smithi160.stdout: openblas-threads-0.3.15-4.el8.x86_64 2023-12-25T20:04:41.266 INFO:teuthology.orchestra.run.smithi160.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-12-25T20:04:41.266 INFO:teuthology.orchestra.run.smithi160.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-12-25T20:04:41.266 INFO:teuthology.orchestra.run.smithi160.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2023-12-25T20:04:41.267 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:04:41.267 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2023-12-25T20:04:41.413 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install ceph-mgr-rook 2023-12-25T20:04:42.269 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:01:53 ago on Mon 25 Dec 2023 08:02:49 PM UTC. 2023-12-25T20:04:42.648 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-12-25T20:04:43.971 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2023-12-25T20:04:43.973 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-12-25T20:04:43.973 INFO:teuthology.orchestra.run.smithi160.stdout: Package Arch Version Repository Size 2023-12-25T20:04:43.973 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-12-25T20:04:43.973 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2023-12-25T20:04:43.973 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr-rook noarch 2:16.2.4-0.el8 ceph-noarch 27 k 2023-12-25T20:04:43.973 INFO:teuthology.orchestra.run.smithi160.stdout:Installing dependencies: 2023-12-25T20:04:43.974 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-12-25T20:04:43.974 INFO:teuthology.orchestra.run.smithi160.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-12-25T20:04:43.974 INFO:teuthology.orchestra.run.smithi160.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-12-25T20:04:43.974 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2023-12-25T20:04:43.974 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2023-12-25T20:04:43.974 INFO:teuthology.orchestra.run.smithi160.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-12-25T20:04:43.974 INFO:teuthology.orchestra.run.smithi160.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2023-12-25T20:04:43.974 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2023-12-25T20:04:43.974 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2023-12-25T20:04:43.974 INFO:teuthology.orchestra.run.smithi160.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2023-12-25T20:04:43.974 INFO:teuthology.orchestra.run.smithi160.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-12-25T20:04:43.974 INFO:teuthology.orchestra.run.smithi160.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-12-25T20:04:43.974 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:04:43.974 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2023-12-25T20:04:43.975 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-12-25T20:04:43.975 INFO:teuthology.orchestra.run.smithi160.stdout:Install 13 Packages 2023-12-25T20:04:43.975 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:04:43.975 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 1.9 M 2023-12-25T20:04:43.975 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 24 M 2023-12-25T20:04:43.975 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2023-12-25T20:04:44.039 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 7/8 2023-12-25T20:04:44.253 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2023-12-25T20:04:44.305 INFO:teuthology.orchestra.run.smithi160.stdout:(1/13): ceph-mgr-rook-16.2.4-0.el8.noarch.rpm 709 kB/s | 27 kB 00:00 2023-12-25T20:04:44.547 INFO:teuthology.orchestra.run.smithi160.stdout:(2/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 95 kB/s | 27 kB 00:00 2023-12-25T20:04:44.572 INFO:teuthology.orchestra.run.smithi160.stdout:(3/13): python3-jsonpointer-1.10-11.el8.noarch. 65 kB/s | 20 kB 00:00 2023-12-25T20:04:44.610 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2023-12-25T20:04:44.611 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 1/8 2023-12-25T20:04:44.611 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2023-12-25T20:04:44.611 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2023-12-25T20:04:44.611 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2023-12-25T20:04:44.611 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2023-12-25T20:04:44.611 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 6/8 2023-12-25T20:04:44.611 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2023-12-25T20:04:44.714 INFO:teuthology.orchestra.run.smithi160.stdout:(4/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 309 kB/s | 126 kB 00:00 2023-12-25T20:04:44.739 INFO:teuthology.orchestra.run.smithi160.stdout:(5/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 928 kB/s | 155 kB 00:00 2023-12-25T20:04:44.765 INFO:teuthology.orchestra.run.smithi160.stdout:(6/13): python3-pyasn1-modules-0.3.7-6.el8.noar 572 kB/s | 110 kB 00:00 2023-12-25T20:04:44.898 INFO:teuthology.orchestra.run.smithi160.stdout:(7/13): python3-requests-oauthlib-1.0.0-1.el8.n 232 kB/s | 43 kB 00:00 2023-12-25T20:04:44.924 INFO:teuthology.orchestra.run.smithi160.stdout:(8/13): python3-certifi-2018.10.15-7.el8.noarch 103 kB/s | 16 kB 00:00 2023-12-25T20:04:44.949 INFO:teuthology.orchestra.run.smithi160.stdout:(9/13): python3-cachetools-3.1.1-4.el8.noarch.r 158 kB/s | 33 kB 00:00 2023-12-25T20:04:45.016 INFO:teuthology.orchestra.run.smithi160.stdout:(10/13): python3-rsa-4.9-2.el8.noarch.rpm 964 kB/s | 64 kB 00:00 2023-12-25T20:04:45.041 INFO:teuthology.orchestra.run.smithi160.stdout:(11/13): python3-google-auth-1.1.1-10.el8.noarc 571 kB/s | 81 kB 00:00 2023-12-25T20:04:45.066 INFO:teuthology.orchestra.run.smithi160.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 1.2 MB/s | 61 kB 00:00 2023-12-25T20:04:45.100 INFO:teuthology.orchestra.run.smithi160.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 6.8 MB/s | 1.2 MB 00:00 2023-12-25T20:04:45.102 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2023-12-25T20:04:45.102 INFO:teuthology.orchestra.run.smithi160.stdout:Total 1.7 MB/s | 1.9 MB 00:01 2023-12-25T20:04:45.157 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2023-12-25T20:04:45.172 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2023-12-25T20:04:45.172 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2023-12-25T20:04:45.259 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2023-12-25T20:04:45.262 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2023-12-25T20:04:45.366 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2023-12-25T20:04:45.366 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-25T20:04:45.367 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2023-12-25T20:04:45.367 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 2023-12-25T20:04:45.367 INFO:teuthology.orchestra.run.smithi050.stdout: libgfortran-8.5.0-21.el8.x86_64 2023-12-25T20:04:45.367 INFO:teuthology.orchestra.run.smithi050.stdout: libquadmath-8.5.0-21.el8.x86_64 2023-12-25T20:04:45.367 INFO:teuthology.orchestra.run.smithi050.stdout: openblas-0.3.15-4.el8.x86_64 2023-12-25T20:04:45.367 INFO:teuthology.orchestra.run.smithi050.stdout: openblas-threads-0.3.15-4.el8.x86_64 2023-12-25T20:04:45.367 INFO:teuthology.orchestra.run.smithi050.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-12-25T20:04:45.367 INFO:teuthology.orchestra.run.smithi050.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-12-25T20:04:45.367 INFO:teuthology.orchestra.run.smithi050.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2023-12-25T20:04:45.367 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-25T20:04:45.367 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-12-25T20:04:45.516 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-mgr-rook 2023-12-25T20:04:45.775 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2023-12-25T20:04:46.032 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-12-25T20:04:46.205 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2023-12-25T20:04:46.407 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:55 ago on Mon 25 Dec 2023 08:02:51 PM UTC. 2023-12-25T20:04:46.473 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2023-12-25T20:04:46.752 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2023-12-25T20:04:47.006 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2023-12-25T20:04:47.248 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2023-12-25T20:04:47.564 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2023-12-25T20:04:47.824 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2023-12-25T20:04:48.216 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-12-25T20:04:48.218 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-12-25T20:04:48.218 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2023-12-25T20:04:48.218 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-12-25T20:04:48.218 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2023-12-25T20:04:48.218 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-rook noarch 2:16.2.4-0.el8 ceph-noarch 27 k 2023-12-25T20:04:48.218 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2023-12-25T20:04:48.218 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-12-25T20:04:48.218 INFO:teuthology.orchestra.run.smithi050.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-12-25T20:04:48.218 INFO:teuthology.orchestra.run.smithi050.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-12-25T20:04:48.219 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2023-12-25T20:04:48.219 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2023-12-25T20:04:48.219 INFO:teuthology.orchestra.run.smithi050.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-12-25T20:04:48.219 INFO:teuthology.orchestra.run.smithi050.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2023-12-25T20:04:48.219 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2023-12-25T20:04:48.219 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2023-12-25T20:04:48.219 INFO:teuthology.orchestra.run.smithi050.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2023-12-25T20:04:48.219 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-12-25T20:04:48.219 INFO:teuthology.orchestra.run.smithi050.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-12-25T20:04:48.219 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-25T20:04:48.219 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2023-12-25T20:04:48.219 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-12-25T20:04:48.219 INFO:teuthology.orchestra.run.smithi050.stdout:Install 13 Packages 2023-12-25T20:04:48.219 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-25T20:04:48.220 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 1.9 M 2023-12-25T20:04:48.220 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 24 M 2023-12-25T20:04:48.220 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2023-12-25T20:04:48.832 INFO:teuthology.orchestra.run.smithi050.stdout:(1/13): ceph-mgr-rook-16.2.4-0.el8.noarch.rpm 353 kB/s | 27 kB 00:00 2023-12-25T20:04:48.877 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2023-12-25T20:04:48.998 INFO:teuthology.orchestra.run.smithi050.stdout:(2/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 110 kB/s | 27 kB 00:00 2023-12-25T20:04:49.023 INFO:teuthology.orchestra.run.smithi050.stdout:(3/13): python3-jsonpointer-1.10-11.el8.noarch. 75 kB/s | 20 kB 00:00 2023-12-25T20:04:49.169 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2023-12-25T20:04:49.173 INFO:teuthology.orchestra.run.smithi050.stdout:(4/13): python3-pyasn1-modules-0.3.7-6.el8.noar 628 kB/s | 110 kB 00:00 2023-12-25T20:04:49.199 INFO:teuthology.orchestra.run.smithi050.stdout:(5/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 884 kB/s | 155 kB 00:00 2023-12-25T20:04:49.224 INFO:teuthology.orchestra.run.smithi050.stdout:(6/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 323 kB/s | 126 kB 00:00 2023-12-25T20:04:49.308 INFO:teuthology.orchestra.run.smithi050.stdout:(7/13): python3-requests-oauthlib-1.0.0-1.el8.n 318 kB/s | 43 kB 00:00 2023-12-25T20:04:49.370 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2023-12-25T20:04:49.433 INFO:teuthology.orchestra.run.smithi050.stdout:(8/13): python3-cachetools-3.1.1-4.el8.noarch.r 141 kB/s | 33 kB 00:00 2023-12-25T20:04:49.458 INFO:teuthology.orchestra.run.smithi050.stdout:(9/13): python3-certifi-2018.10.15-7.el8.noarch 70 kB/s | 16 kB 00:00 2023-12-25T20:04:49.542 INFO:teuthology.orchestra.run.smithi050.stdout:(10/13): python3-rsa-4.9-2.el8.noarch.rpm 775 kB/s | 64 kB 00:00 2023-12-25T20:04:49.576 INFO:teuthology.orchestra.run.smithi050.stdout:(11/13): python3-google-auth-1.1.1-10.el8.noarc 305 kB/s | 81 kB 00:00 2023-12-25T20:04:49.585 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2023-12-25T20:04:49.601 INFO:teuthology.orchestra.run.smithi050.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 1.0 MB/s | 61 kB 00:00 2023-12-25T20:04:49.753 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2023-12-25T20:04:50.051 INFO:teuthology.orchestra.run.smithi050.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 2.0 MB/s | 1.2 MB 00:00 2023-12-25T20:04:50.053 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2023-12-25T20:04:50.054 INFO:teuthology.orchestra.run.smithi050.stdout:Total 1.0 MB/s | 1.9 MB 00:01 2023-12-25T20:04:50.056 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2023-12-25T20:04:50.056 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-mgr-rook-2:16.2.4-0.el8.noarch 1/13 2023-12-25T20:04:50.056 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2023-12-25T20:04:50.056 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2023-12-25T20:04:50.056 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2023-12-25T20:04:50.056 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2023-12-25T20:04:50.056 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2023-12-25T20:04:50.056 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2023-12-25T20:04:50.056 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2023-12-25T20:04:50.057 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-12-25T20:04:50.057 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2023-12-25T20:04:50.057 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2023-12-25T20:04:50.057 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2023-12-25T20:04:50.115 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2023-12-25T20:04:50.132 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2023-12-25T20:04:50.132 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2023-12-25T20:04:50.226 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2023-12-25T20:04:50.229 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2023-12-25T20:04:50.782 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2023-12-25T20:04:50.816 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2023-12-25T20:04:50.816 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:04:50.816 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2023-12-25T20:04:50.816 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr-rook-2:16.2.4-0.el8.noarch 2023-12-25T20:04:50.816 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-12-25T20:04:50.816 INFO:teuthology.orchestra.run.smithi160.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-12-25T20:04:50.816 INFO:teuthology.orchestra.run.smithi160.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-12-25T20:04:50.816 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-12-25T20:04:50.816 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-12-25T20:04:50.816 INFO:teuthology.orchestra.run.smithi160.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-12-25T20:04:50.816 INFO:teuthology.orchestra.run.smithi160.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-12-25T20:04:50.816 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-12-25T20:04:50.817 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-12-25T20:04:50.817 INFO:teuthology.orchestra.run.smithi160.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-12-25T20:04:50.817 INFO:teuthology.orchestra.run.smithi160.stdout: python3-rsa-4.9-2.el8.noarch 2023-12-25T20:04:50.817 INFO:teuthology.orchestra.run.smithi160.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-12-25T20:04:50.817 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:04:50.817 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2023-12-25T20:04:50.933 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install ceph-mgr-cephadm 2023-12-25T20:04:51.022 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-12-25T20:04:51.243 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2023-12-25T20:04:51.488 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2023-12-25T20:04:51.725 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2023-12-25T20:04:51.789 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:02:02 ago on Mon 25 Dec 2023 08:02:49 PM UTC. 2023-12-25T20:04:51.990 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2023-12-25T20:04:52.236 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2023-12-25T20:04:52.539 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2023-12-25T20:04:52.792 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2023-12-25T20:04:53.485 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2023-12-25T20:04:53.486 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-12-25T20:04:53.486 INFO:teuthology.orchestra.run.smithi160.stdout: Package Arch Version Repository Size 2023-12-25T20:04:53.486 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-12-25T20:04:53.486 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2023-12-25T20:04:53.486 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr-cephadm noarch 2:16.2.4-0.el8 ceph-noarch 89 k 2023-12-25T20:04:53.486 INFO:teuthology.orchestra.run.smithi160.stdout:Installing dependencies: 2023-12-25T20:04:53.486 INFO:teuthology.orchestra.run.smithi160.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2023-12-25T20:04:53.486 INFO:teuthology.orchestra.run.smithi160.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2023-12-25T20:04:53.486 INFO:teuthology.orchestra.run.smithi160.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2023-12-25T20:04:53.486 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2023-12-25T20:04:53.486 INFO:teuthology.orchestra.run.smithi160.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2023-12-25T20:04:53.487 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:04:53.487 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2023-12-25T20:04:53.487 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-12-25T20:04:53.487 INFO:teuthology.orchestra.run.smithi160.stdout:Install 6 Packages 2023-12-25T20:04:53.487 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:04:53.487 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 5.8 M 2023-12-25T20:04:53.487 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 24 M 2023-12-25T20:04:53.488 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2023-12-25T20:04:53.907 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2023-12-25T20:04:53.915 INFO:teuthology.orchestra.run.smithi160.stdout:(1/6): ceph-mgr-cephadm-16.2.4-0.el8.noarch.rpm 1.9 MB/s | 89 kB 00:00 2023-12-25T20:04:54.006 INFO:teuthology.orchestra.run.smithi160.stdout:(2/6): python3-apipkg-1.5-6.el8.noarch.rpm 231 kB/s | 21 kB 00:00 2023-12-25T20:04:54.173 INFO:teuthology.orchestra.run.smithi160.stdout:(3/6): python3-execnet-1.7.1-1.el8.noarch.rpm 2.1 MB/s | 355 kB 00:00 2023-12-25T20:04:54.191 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2023-12-25T20:04:54.233 INFO:teuthology.orchestra.run.smithi160.stdout:(4/6): python3-remoto-1.2.1-1.el8.noarch.rpm 835 kB/s | 49 kB 00:00 2023-12-25T20:04:54.407 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2023-12-25T20:04:54.626 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2023-12-25T20:04:54.641 INFO:teuthology.orchestra.run.smithi160.stdout:(5/6): python3-jinja2-2.10.1-3.el8.noarch.rpm 697 kB/s | 538 kB 00:00 2023-12-25T20:04:54.809 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2023-12-25T20:04:55.124 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2023-12-25T20:04:55.124 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mgr-rook-2:16.2.4-0.el8.noarch 1/13 2023-12-25T20:04:55.124 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2023-12-25T20:04:55.124 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2023-12-25T20:04:55.124 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2023-12-25T20:04:55.124 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2023-12-25T20:04:55.124 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2023-12-25T20:04:55.124 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2023-12-25T20:04:55.124 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2023-12-25T20:04:55.124 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-12-25T20:04:55.124 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2023-12-25T20:04:55.124 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2023-12-25T20:04:55.124 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2023-12-25T20:04:55.934 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2023-12-25T20:04:55.934 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-25T20:04:55.934 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2023-12-25T20:04:55.934 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-rook-2:16.2.4-0.el8.noarch 2023-12-25T20:04:55.935 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-12-25T20:04:55.935 INFO:teuthology.orchestra.run.smithi050.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-12-25T20:04:55.935 INFO:teuthology.orchestra.run.smithi050.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-12-25T20:04:55.935 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-12-25T20:04:55.935 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-12-25T20:04:55.935 INFO:teuthology.orchestra.run.smithi050.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-12-25T20:04:55.935 INFO:teuthology.orchestra.run.smithi050.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-12-25T20:04:55.935 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-12-25T20:04:55.935 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-12-25T20:04:55.936 INFO:teuthology.orchestra.run.smithi050.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-12-25T20:04:55.936 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rsa-4.9-2.el8.noarch 2023-12-25T20:04:55.936 INFO:teuthology.orchestra.run.smithi050.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-12-25T20:04:55.936 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-25T20:04:55.936 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-12-25T20:04:56.056 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-mgr-cephadm 2023-12-25T20:04:56.332 INFO:teuthology.orchestra.run.smithi160.stdout:(6/6): python3-babel-2.5.1-7.el8.noarch.rpm 1.9 MB/s | 4.8 MB 00:02 2023-12-25T20:04:56.333 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2023-12-25T20:04:56.334 INFO:teuthology.orchestra.run.smithi160.stdout:Total 2.0 MB/s | 5.8 MB 00:02 2023-12-25T20:04:56.387 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2023-12-25T20:04:56.397 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2023-12-25T20:04:56.397 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2023-12-25T20:04:56.503 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2023-12-25T20:04:56.509 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2023-12-25T20:04:56.813 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2023-12-25T20:04:56.951 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:02:05 ago on Mon 25 Dec 2023 08:02:51 PM UTC. 2023-12-25T20:04:57.085 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2023-12-25T20:04:57.371 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2023-12-25T20:04:58.275 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2023-12-25T20:04:58.648 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2023-12-25T20:04:58.771 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-12-25T20:04:58.772 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-12-25T20:04:58.773 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2023-12-25T20:04:58.773 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-12-25T20:04:58.773 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2023-12-25T20:04:58.773 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-cephadm noarch 2:16.2.4-0.el8 ceph-noarch 89 k 2023-12-25T20:04:58.773 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2023-12-25T20:04:58.773 INFO:teuthology.orchestra.run.smithi050.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2023-12-25T20:04:58.773 INFO:teuthology.orchestra.run.smithi050.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2023-12-25T20:04:58.773 INFO:teuthology.orchestra.run.smithi050.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2023-12-25T20:04:58.773 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2023-12-25T20:04:58.773 INFO:teuthology.orchestra.run.smithi050.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2023-12-25T20:04:58.773 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-25T20:04:58.773 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2023-12-25T20:04:58.773 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-12-25T20:04:58.774 INFO:teuthology.orchestra.run.smithi050.stdout:Install 6 Packages 2023-12-25T20:04:58.774 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-25T20:04:58.774 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 5.8 M 2023-12-25T20:04:58.774 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 24 M 2023-12-25T20:04:58.774 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2023-12-25T20:04:58.907 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 5/6 2023-12-25T20:04:59.077 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2023-12-25T20:04:59.192 INFO:teuthology.orchestra.run.smithi050.stdout:(1/6): ceph-mgr-cephadm-16.2.4-0.el8.noarch.rpm 2.1 MB/s | 89 kB 00:00 2023-12-25T20:04:59.402 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2023-12-25T20:04:59.403 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 1/6 2023-12-25T20:04:59.403 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2023-12-25T20:04:59.403 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/6 2023-12-25T20:04:59.403 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2023-12-25T20:04:59.403 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2023-12-25T20:04:59.594 INFO:teuthology.orchestra.run.smithi050.stdout:(2/6): python3-apipkg-1.5-6.el8.noarch.rpm 53 kB/s | 21 kB 00:00 2023-12-25T20:04:59.717 INFO:teuthology.orchestra.run.smithi050.stdout:(3/6): python3-jinja2-2.10.1-3.el8.noarch.rpm 947 kB/s | 538 kB 00:00 2023-12-25T20:05:00.104 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2023-12-25T20:05:00.104 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:05:00.104 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2023-12-25T20:05:00.104 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch python3-apipkg-1.5-6.el8.noarch 2023-12-25T20:05:00.104 INFO:teuthology.orchestra.run.smithi160.stdout: python3-babel-2.5.1-7.el8.noarch python3-execnet-1.7.1-1.el8.noarch 2023-12-25T20:05:00.104 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jinja2-2.10.1-3.el8.noarch python3-remoto-1.2.1-1.el8.noarch 2023-12-25T20:05:00.105 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:05:00.105 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2023-12-25T20:05:00.117 INFO:teuthology.orchestra.run.smithi050.stdout:(4/6): python3-execnet-1.7.1-1.el8.noarch.rpm 680 kB/s | 355 kB 00:00 2023-12-25T20:05:00.176 INFO:teuthology.orchestra.run.smithi050.stdout:(5/6): python3-remoto-1.2.1-1.el8.noarch.rpm 107 kB/s | 49 kB 00:00 2023-12-25T20:05:00.230 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install ceph-fuse 2023-12-25T20:05:00.334 INFO:teuthology.orchestra.run.smithi050.stdout:(6/6): python3-babel-2.5.1-7.el8.noarch.rpm 4.0 MB/s | 4.8 MB 00:01 2023-12-25T20:05:00.337 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2023-12-25T20:05:00.337 INFO:teuthology.orchestra.run.smithi050.stdout:Total 3.7 MB/s | 5.8 MB 00:01 2023-12-25T20:05:00.393 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2023-12-25T20:05:00.404 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2023-12-25T20:05:00.404 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2023-12-25T20:05:00.518 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2023-12-25T20:05:00.521 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2023-12-25T20:05:00.808 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2023-12-25T20:05:01.087 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:02:12 ago on Mon 25 Dec 2023 08:02:49 PM UTC. 2023-12-25T20:05:01.114 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2023-12-25T20:05:01.396 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2023-12-25T20:05:02.631 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2023-12-25T20:05:02.781 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2023-12-25T20:05:02.782 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-12-25T20:05:02.782 INFO:teuthology.orchestra.run.smithi160.stdout: Package Architecture Version Repository Size 2023-12-25T20:05:02.782 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-12-25T20:05:02.782 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2023-12-25T20:05:02.782 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-fuse x86_64 2:16.2.4-0.el8 ceph 885 k 2023-12-25T20:05:02.782 INFO:teuthology.orchestra.run.smithi160.stdout:Installing dependencies: 2023-12-25T20:05:02.782 INFO:teuthology.orchestra.run.smithi160.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2023-12-25T20:05:02.782 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:05:02.782 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2023-12-25T20:05:02.782 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-12-25T20:05:02.783 INFO:teuthology.orchestra.run.smithi160.stdout:Install 2 Packages 2023-12-25T20:05:02.783 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:05:02.783 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 968 k 2023-12-25T20:05:02.783 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 3.2 M 2023-12-25T20:05:02.783 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2023-12-25T20:05:03.004 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2023-12-25T20:05:03.160 INFO:teuthology.orchestra.run.smithi160.stdout:(1/2): ceph-fuse-16.2.4-0.el8.x86_64.rpm 16 MB/s | 885 kB 00:00 2023-12-25T20:05:03.224 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 5/6 2023-12-25T20:05:03.277 INFO:teuthology.orchestra.run.smithi160.stdout:(2/2): fuse-2.9.7-17.el8.x86_64.rpm 487 kB/s | 83 kB 00:00 2023-12-25T20:05:03.277 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2023-12-25T20:05:03.277 INFO:teuthology.orchestra.run.smithi160.stdout:Total 1.9 MB/s | 968 kB 00:00 2023-12-25T20:05:03.287 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2023-12-25T20:05:03.308 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2023-12-25T20:05:03.308 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2023-12-25T20:05:03.393 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2023-12-25T20:05:03.396 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2023-12-25T20:05:03.441 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2023-12-25T20:05:03.587 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2023-12-25T20:05:03.749 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2023-12-25T20:05:03.749 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 1/6 2023-12-25T20:05:03.749 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2023-12-25T20:05:03.749 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/6 2023-12-25T20:05:03.750 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2023-12-25T20:05:03.750 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2023-12-25T20:05:03.893 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2023-12-25T20:05:04.143 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2023-12-25T20:05:04.483 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2023-12-25T20:05:04.483 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-25T20:05:04.483 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2023-12-25T20:05:04.483 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch python3-apipkg-1.5-6.el8.noarch 2023-12-25T20:05:04.484 INFO:teuthology.orchestra.run.smithi050.stdout: python3-babel-2.5.1-7.el8.noarch python3-execnet-1.7.1-1.el8.noarch 2023-12-25T20:05:04.484 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jinja2-2.10.1-3.el8.noarch python3-remoto-1.2.1-1.el8.noarch 2023-12-25T20:05:04.484 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-25T20:05:04.484 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-12-25T20:05:04.532 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2023-12-25T20:05:04.532 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-fuse-2:16.2.4-0.el8.x86_64 1/2 2023-12-25T20:05:04.607 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-fuse 2023-12-25T20:05:05.269 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2023-12-25T20:05:05.269 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:05:05.269 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2023-12-25T20:05:05.269 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-fuse-2:16.2.4-0.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2023-12-25T20:05:05.269 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:05:05.270 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2023-12-25T20:05:05.512 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install librados-devel 2023-12-25T20:05:05.513 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:02:14 ago on Mon 25 Dec 2023 08:02:51 PM UTC. 2023-12-25T20:05:06.378 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:02:17 ago on Mon 25 Dec 2023 08:02:49 PM UTC. 2023-12-25T20:05:07.316 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-12-25T20:05:07.317 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-12-25T20:05:07.317 INFO:teuthology.orchestra.run.smithi050.stdout: Package Architecture Version Repository Size 2023-12-25T20:05:07.317 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-12-25T20:05:07.317 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2023-12-25T20:05:07.317 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-fuse x86_64 2:16.2.4-0.el8 ceph 885 k 2023-12-25T20:05:07.317 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2023-12-25T20:05:07.318 INFO:teuthology.orchestra.run.smithi050.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2023-12-25T20:05:07.318 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-25T20:05:07.318 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2023-12-25T20:05:07.318 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-12-25T20:05:07.318 INFO:teuthology.orchestra.run.smithi050.stdout:Install 2 Packages 2023-12-25T20:05:07.318 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-25T20:05:07.318 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 968 k 2023-12-25T20:05:07.318 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 3.2 M 2023-12-25T20:05:07.318 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2023-12-25T20:05:08.089 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2023-12-25T20:05:08.090 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-12-25T20:05:08.090 INFO:teuthology.orchestra.run.smithi160.stdout: Package Architecture Version Repository Size 2023-12-25T20:05:08.090 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-12-25T20:05:08.090 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2023-12-25T20:05:08.090 INFO:teuthology.orchestra.run.smithi160.stdout: librados-devel x86_64 2:16.2.4-0.el8 ceph 131 k 2023-12-25T20:05:08.090 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:05:08.090 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2023-12-25T20:05:08.090 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-12-25T20:05:08.090 INFO:teuthology.orchestra.run.smithi160.stdout:Install 1 Package 2023-12-25T20:05:08.091 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:05:08.091 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 131 k 2023-12-25T20:05:08.091 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 439 k 2023-12-25T20:05:08.091 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2023-12-25T20:05:08.138 INFO:teuthology.orchestra.run.smithi160.stdout:librados-devel-16.2.4-0.el8.x86_64.rpm 2.8 MB/s | 131 kB 00:00 2023-12-25T20:05:08.138 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2023-12-25T20:05:08.138 INFO:teuthology.orchestra.run.smithi160.stdout:Total 2.7 MB/s | 131 kB 00:00 2023-12-25T20:05:08.138 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2023-12-25T20:05:08.144 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2023-12-25T20:05:08.144 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2023-12-25T20:05:08.194 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2023-12-25T20:05:08.197 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2023-12-25T20:05:08.313 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2023-12-25T20:05:08.712 INFO:teuthology.orchestra.run.smithi050.stdout:(1/2): ceph-fuse-16.2.4-0.el8.x86_64.rpm 4.5 MB/s | 885 kB 00:00 2023-12-25T20:05:08.937 INFO:teuthology.orchestra.run.smithi050.stdout:(2/2): fuse-2.9.7-17.el8.x86_64.rpm 199 kB/s | 83 kB 00:00 2023-12-25T20:05:08.938 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2023-12-25T20:05:08.938 INFO:teuthology.orchestra.run.smithi050.stdout:Total 597 kB/s | 968 kB 00:01 2023-12-25T20:05:08.951 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2023-12-25T20:05:08.974 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2023-12-25T20:05:08.975 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2023-12-25T20:05:09.075 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2023-12-25T20:05:09.078 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2023-12-25T20:05:09.279 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2023-12-25T20:05:09.537 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2023-12-25T20:05:09.565 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2023-12-25T20:05:09.740 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2023-12-25T20:05:10.137 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2023-12-25T20:05:10.137 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-fuse-2:16.2.4-0.el8.x86_64 1/2 2023-12-25T20:05:10.198 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: librados-devel-2:16.2.4-0.el8.x86_64 1/1 2023-12-25T20:05:10.901 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2023-12-25T20:05:10.901 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-25T20:05:10.901 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2023-12-25T20:05:10.901 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-fuse-2:16.2.4-0.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2023-12-25T20:05:10.901 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-25T20:05:10.901 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-12-25T20:05:11.120 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install librados-devel 2023-12-25T20:05:11.610 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2023-12-25T20:05:11.611 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:05:11.611 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2023-12-25T20:05:11.611 INFO:teuthology.orchestra.run.smithi160.stdout: librados-devel-2:16.2.4-0.el8.x86_64 2023-12-25T20:05:11.611 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:05:11.611 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2023-12-25T20:05:11.864 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install libcephfs2 2023-12-25T20:05:12.019 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:02:21 ago on Mon 25 Dec 2023 08:02:51 PM UTC. 2023-12-25T20:05:12.728 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:02:23 ago on Mon 25 Dec 2023 08:02:49 PM UTC. 2023-12-25T20:05:13.837 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-12-25T20:05:13.838 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-12-25T20:05:13.838 INFO:teuthology.orchestra.run.smithi050.stdout: Package Architecture Version Repository Size 2023-12-25T20:05:13.838 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-12-25T20:05:13.838 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2023-12-25T20:05:13.838 INFO:teuthology.orchestra.run.smithi050.stdout: librados-devel x86_64 2:16.2.4-0.el8 ceph 131 k 2023-12-25T20:05:13.838 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-25T20:05:13.838 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2023-12-25T20:05:13.838 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-12-25T20:05:13.838 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2023-12-25T20:05:13.839 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-25T20:05:13.839 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 131 k 2023-12-25T20:05:13.839 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 439 k 2023-12-25T20:05:13.839 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2023-12-25T20:05:14.205 INFO:teuthology.orchestra.run.smithi160.stdout:Package libcephfs2-2:16.2.4-0.el8.x86_64 is already installed. 2023-12-25T20:05:14.387 INFO:teuthology.orchestra.run.smithi050.stdout:librados-devel-16.2.4-0.el8.x86_64.rpm 240 kB/s | 131 kB 00:00 2023-12-25T20:05:14.387 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2023-12-25T20:05:14.388 INFO:teuthology.orchestra.run.smithi050.stdout:Total 239 kB/s | 131 kB 00:00 2023-12-25T20:05:14.388 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2023-12-25T20:05:14.395 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2023-12-25T20:05:14.395 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2023-12-25T20:05:14.431 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2023-12-25T20:05:14.432 INFO:teuthology.orchestra.run.smithi160.stdout:Nothing to do. 2023-12-25T20:05:14.432 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2023-12-25T20:05:14.453 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2023-12-25T20:05:14.456 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2023-12-25T20:05:14.537 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install libcephfs-devel 2023-12-25T20:05:14.570 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2023-12-25T20:05:14.806 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2023-12-25T20:05:15.094 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: librados-devel-2:16.2.4-0.el8.x86_64 1/1 2023-12-25T20:05:15.405 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:02:26 ago on Mon 25 Dec 2023 08:02:49 PM UTC. 2023-12-25T20:05:15.820 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2023-12-25T20:05:15.820 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-25T20:05:15.820 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2023-12-25T20:05:15.820 INFO:teuthology.orchestra.run.smithi050.stdout: librados-devel-2:16.2.4-0.el8.x86_64 2023-12-25T20:05:15.820 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-25T20:05:15.820 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-12-25T20:05:16.034 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install libcephfs2 2023-12-25T20:05:16.922 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:02:25 ago on Mon 25 Dec 2023 08:02:51 PM UTC. 2023-12-25T20:05:17.093 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2023-12-25T20:05:17.093 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-12-25T20:05:17.093 INFO:teuthology.orchestra.run.smithi160.stdout: Package Architecture Version Repository Size 2023-12-25T20:05:17.093 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-12-25T20:05:17.093 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2023-12-25T20:05:17.093 INFO:teuthology.orchestra.run.smithi160.stdout: libcephfs-devel x86_64 2:16.2.4-0.el8 ceph 24 k 2023-12-25T20:05:17.093 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:05:17.094 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2023-12-25T20:05:17.094 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-12-25T20:05:17.094 INFO:teuthology.orchestra.run.smithi160.stdout:Install 1 Package 2023-12-25T20:05:17.094 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:05:17.094 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 24 k 2023-12-25T20:05:17.094 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 94 k 2023-12-25T20:05:17.094 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2023-12-25T20:05:17.137 INFO:teuthology.orchestra.run.smithi160.stdout:libcephfs-devel-16.2.4-0.el8.x86_64.rpm 580 kB/s | 24 kB 00:00 2023-12-25T20:05:17.137 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2023-12-25T20:05:17.137 INFO:teuthology.orchestra.run.smithi160.stdout:Total 562 kB/s | 24 kB 00:00 2023-12-25T20:05:17.138 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2023-12-25T20:05:17.141 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2023-12-25T20:05:17.141 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2023-12-25T20:05:17.143 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2023-12-25T20:05:17.146 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2023-12-25T20:05:17.195 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2023-12-25T20:05:17.415 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2023-12-25T20:05:17.685 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2023-12-25T20:05:18.348 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2023-12-25T20:05:18.348 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:05:18.348 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2023-12-25T20:05:18.349 INFO:teuthology.orchestra.run.smithi160.stdout: libcephfs-devel-2:16.2.4-0.el8.x86_64 2023-12-25T20:05:18.349 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:05:18.349 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2023-12-25T20:05:18.489 INFO:teuthology.orchestra.run.smithi050.stdout:Package libcephfs2-2:16.2.4-0.el8.x86_64 is already installed. 2023-12-25T20:05:18.504 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install librados2 2023-12-25T20:05:18.732 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-12-25T20:05:18.733 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2023-12-25T20:05:18.733 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-12-25T20:05:18.826 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install libcephfs-devel 2023-12-25T20:05:19.360 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:02:30 ago on Mon 25 Dec 2023 08:02:49 PM UTC. 2023-12-25T20:05:19.719 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:02:28 ago on Mon 25 Dec 2023 08:02:51 PM UTC. 2023-12-25T20:05:20.820 INFO:teuthology.orchestra.run.smithi160.stdout:Package librados2-2:16.2.4-0.el8.x86_64 is already installed. 2023-12-25T20:05:21.047 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2023-12-25T20:05:21.048 INFO:teuthology.orchestra.run.smithi160.stdout:Nothing to do. 2023-12-25T20:05:21.048 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2023-12-25T20:05:21.151 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install librbd1 2023-12-25T20:05:21.520 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-12-25T20:05:21.520 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-12-25T20:05:21.520 INFO:teuthology.orchestra.run.smithi050.stdout: Package Architecture Version Repository Size 2023-12-25T20:05:21.520 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-12-25T20:05:21.521 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2023-12-25T20:05:21.521 INFO:teuthology.orchestra.run.smithi050.stdout: libcephfs-devel x86_64 2:16.2.4-0.el8 ceph 24 k 2023-12-25T20:05:21.521 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-25T20:05:21.521 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2023-12-25T20:05:21.521 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-12-25T20:05:21.521 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2023-12-25T20:05:21.521 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-25T20:05:21.521 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 24 k 2023-12-25T20:05:21.522 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 94 k 2023-12-25T20:05:21.522 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2023-12-25T20:05:21.564 INFO:teuthology.orchestra.run.smithi050.stdout:libcephfs-devel-16.2.4-0.el8.x86_64.rpm 571 kB/s | 24 kB 00:00 2023-12-25T20:05:21.565 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2023-12-25T20:05:21.565 INFO:teuthology.orchestra.run.smithi050.stdout:Total 556 kB/s | 24 kB 00:00 2023-12-25T20:05:21.566 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2023-12-25T20:05:21.569 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2023-12-25T20:05:21.569 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2023-12-25T20:05:21.571 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2023-12-25T20:05:21.575 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2023-12-25T20:05:21.623 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2023-12-25T20:05:21.829 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2023-12-25T20:05:22.011 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:02:33 ago on Mon 25 Dec 2023 08:02:49 PM UTC. 2023-12-25T20:05:22.107 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2023-12-25T20:05:22.799 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2023-12-25T20:05:22.799 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-25T20:05:22.799 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2023-12-25T20:05:22.799 INFO:teuthology.orchestra.run.smithi050.stdout: libcephfs-devel-2:16.2.4-0.el8.x86_64 2023-12-25T20:05:22.799 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-25T20:05:22.799 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-12-25T20:05:22.910 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install librados2 2023-12-25T20:05:23.475 INFO:teuthology.orchestra.run.smithi160.stdout:Package librbd1-2:16.2.4-0.el8.x86_64 is already installed. 2023-12-25T20:05:23.704 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2023-12-25T20:05:23.705 INFO:teuthology.orchestra.run.smithi160.stdout:Nothing to do. 2023-12-25T20:05:23.705 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2023-12-25T20:05:23.804 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:02:32 ago on Mon 25 Dec 2023 08:02:51 PM UTC. 2023-12-25T20:05:23.864 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install python3-rados 2023-12-25T20:05:24.723 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:02:35 ago on Mon 25 Dec 2023 08:02:49 PM UTC. 2023-12-25T20:05:25.359 INFO:teuthology.orchestra.run.smithi050.stdout:Package librados2-2:16.2.4-0.el8.x86_64 is already installed. 2023-12-25T20:05:25.602 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-12-25T20:05:25.603 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2023-12-25T20:05:25.603 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-12-25T20:05:25.706 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install librbd1 2023-12-25T20:05:26.200 INFO:teuthology.orchestra.run.smithi160.stdout:Package python3-rados-2:16.2.4-0.el8.x86_64 is already installed. 2023-12-25T20:05:26.426 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2023-12-25T20:05:26.427 INFO:teuthology.orchestra.run.smithi160.stdout:Nothing to do. 2023-12-25T20:05:26.427 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2023-12-25T20:05:26.536 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install python3-rgw 2023-12-25T20:05:26.599 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:02:35 ago on Mon 25 Dec 2023 08:02:51 PM UTC. 2023-12-25T20:05:27.395 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:02:38 ago on Mon 25 Dec 2023 08:02:49 PM UTC. 2023-12-25T20:05:28.174 INFO:teuthology.orchestra.run.smithi050.stdout:Package librbd1-2:16.2.4-0.el8.x86_64 is already installed. 2023-12-25T20:05:28.416 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-12-25T20:05:28.418 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2023-12-25T20:05:28.418 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-12-25T20:05:28.509 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install python3-rados 2023-12-25T20:05:28.841 INFO:teuthology.orchestra.run.smithi160.stdout:Package python3-rgw-2:16.2.4-0.el8.x86_64 is already installed. 2023-12-25T20:05:29.067 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2023-12-25T20:05:29.068 INFO:teuthology.orchestra.run.smithi160.stdout:Nothing to do. 2023-12-25T20:05:29.068 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2023-12-25T20:05:29.173 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install python3-cephfs 2023-12-25T20:05:29.409 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:02:38 ago on Mon 25 Dec 2023 08:02:51 PM UTC. 2023-12-25T20:05:30.037 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:02:41 ago on Mon 25 Dec 2023 08:02:49 PM UTC. 2023-12-25T20:05:30.977 INFO:teuthology.orchestra.run.smithi050.stdout:Package python3-rados-2:16.2.4-0.el8.x86_64 is already installed. 2023-12-25T20:05:31.220 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-12-25T20:05:31.222 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2023-12-25T20:05:31.222 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-12-25T20:05:31.315 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install python3-rgw 2023-12-25T20:05:31.508 INFO:teuthology.orchestra.run.smithi160.stdout:Package python3-cephfs-2:16.2.4-0.el8.x86_64 is already installed. 2023-12-25T20:05:31.732 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2023-12-25T20:05:31.734 INFO:teuthology.orchestra.run.smithi160.stdout:Nothing to do. 2023-12-25T20:05:31.734 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2023-12-25T20:05:31.833 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install python3-rbd 2023-12-25T20:05:32.212 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:02:41 ago on Mon 25 Dec 2023 08:02:51 PM UTC. 2023-12-25T20:05:32.696 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:02:43 ago on Mon 25 Dec 2023 08:02:49 PM UTC. 2023-12-25T20:05:33.759 INFO:teuthology.orchestra.run.smithi050.stdout:Package python3-rgw-2:16.2.4-0.el8.x86_64 is already installed. 2023-12-25T20:05:34.003 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-12-25T20:05:34.004 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2023-12-25T20:05:34.004 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-12-25T20:05:34.108 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install python3-cephfs 2023-12-25T20:05:34.145 INFO:teuthology.orchestra.run.smithi160.stdout:Package python3-rbd-2:16.2.4-0.el8.x86_64 is already installed. 2023-12-25T20:05:34.370 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2023-12-25T20:05:34.371 INFO:teuthology.orchestra.run.smithi160.stdout:Nothing to do. 2023-12-25T20:05:34.371 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2023-12-25T20:05:34.476 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install rbd-fuse 2023-12-25T20:05:35.013 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:02:44 ago on Mon 25 Dec 2023 08:02:51 PM UTC. 2023-12-25T20:05:35.328 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:02:46 ago on Mon 25 Dec 2023 08:02:49 PM UTC. 2023-12-25T20:05:36.582 INFO:teuthology.orchestra.run.smithi050.stdout:Package python3-cephfs-2:16.2.4-0.el8.x86_64 is already installed. 2023-12-25T20:05:36.824 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-12-25T20:05:36.825 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2023-12-25T20:05:36.826 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-12-25T20:05:36.924 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install python3-rbd 2023-12-25T20:05:37.009 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2023-12-25T20:05:37.010 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-12-25T20:05:37.010 INFO:teuthology.orchestra.run.smithi160.stdout: Package Architecture Version Repository Size 2023-12-25T20:05:37.010 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-12-25T20:05:37.010 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2023-12-25T20:05:37.010 INFO:teuthology.orchestra.run.smithi160.stdout: rbd-fuse x86_64 2:16.2.4-0.el8 ceph 89 k 2023-12-25T20:05:37.010 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:05:37.010 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2023-12-25T20:05:37.010 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-12-25T20:05:37.010 INFO:teuthology.orchestra.run.smithi160.stdout:Install 1 Package 2023-12-25T20:05:37.011 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:05:37.011 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 89 k 2023-12-25T20:05:37.011 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 298 k 2023-12-25T20:05:37.011 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2023-12-25T20:05:37.067 INFO:teuthology.orchestra.run.smithi160.stdout:rbd-fuse-16.2.4-0.el8.x86_64.rpm 1.6 MB/s | 89 kB 00:00 2023-12-25T20:05:37.068 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2023-12-25T20:05:37.068 INFO:teuthology.orchestra.run.smithi160.stdout:Total 1.5 MB/s | 89 kB 00:00 2023-12-25T20:05:37.068 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2023-12-25T20:05:37.078 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2023-12-25T20:05:37.078 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2023-12-25T20:05:37.128 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2023-12-25T20:05:37.131 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2023-12-25T20:05:37.262 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2023-12-25T20:05:37.445 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2023-12-25T20:05:37.743 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2023-12-25T20:05:37.820 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:02:46 ago on Mon 25 Dec 2023 08:02:51 PM UTC. 2023-12-25T20:05:38.479 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2023-12-25T20:05:38.479 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:05:38.479 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2023-12-25T20:05:38.479 INFO:teuthology.orchestra.run.smithi160.stdout: rbd-fuse-2:16.2.4-0.el8.x86_64 2023-12-25T20:05:38.479 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:05:38.479 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2023-12-25T20:05:38.746 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install rbd-mirror 2023-12-25T20:05:39.401 INFO:teuthology.orchestra.run.smithi050.stdout:Package python3-rbd-2:16.2.4-0.el8.x86_64 is already installed. 2023-12-25T20:05:39.613 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:02:50 ago on Mon 25 Dec 2023 08:02:49 PM UTC. 2023-12-25T20:05:39.647 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-12-25T20:05:39.648 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2023-12-25T20:05:39.648 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-12-25T20:05:39.755 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install rbd-fuse 2023-12-25T20:05:40.651 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:02:49 ago on Mon 25 Dec 2023 08:02:51 PM UTC. 2023-12-25T20:05:41.314 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2023-12-25T20:05:41.315 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-12-25T20:05:41.315 INFO:teuthology.orchestra.run.smithi160.stdout: Package Architecture Version Repository Size 2023-12-25T20:05:41.315 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-12-25T20:05:41.315 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2023-12-25T20:05:41.315 INFO:teuthology.orchestra.run.smithi160.stdout: rbd-mirror x86_64 2:16.2.4-0.el8 ceph 3.7 M 2023-12-25T20:05:41.315 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:05:41.315 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2023-12-25T20:05:41.316 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-12-25T20:05:41.316 INFO:teuthology.orchestra.run.smithi160.stdout:Install 1 Package 2023-12-25T20:05:41.316 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:05:41.316 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 3.7 M 2023-12-25T20:05:41.316 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 18 M 2023-12-25T20:05:41.316 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2023-12-25T20:05:41.403 INFO:teuthology.orchestra.run.smithi160.stdout:rbd-mirror-16.2.4-0.el8.x86_64.rpm 43 MB/s | 3.7 MB 00:00 2023-12-25T20:05:41.403 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2023-12-25T20:05:41.404 INFO:teuthology.orchestra.run.smithi160.stdout:Total 42 MB/s | 3.7 MB 00:00 2023-12-25T20:05:41.404 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2023-12-25T20:05:41.416 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2023-12-25T20:05:41.416 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2023-12-25T20:05:41.479 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2023-12-25T20:05:41.481 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2023-12-25T20:05:41.953 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2023-12-25T20:05:42.146 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2023-12-25T20:05:42.461 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-12-25T20:05:42.462 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-12-25T20:05:42.462 INFO:teuthology.orchestra.run.smithi050.stdout: Package Architecture Version Repository Size 2023-12-25T20:05:42.462 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-12-25T20:05:42.462 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2023-12-25T20:05:42.462 INFO:teuthology.orchestra.run.smithi050.stdout: rbd-fuse x86_64 2:16.2.4-0.el8 ceph 89 k 2023-12-25T20:05:42.462 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-25T20:05:42.462 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2023-12-25T20:05:42.462 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-12-25T20:05:42.463 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2023-12-25T20:05:42.463 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-25T20:05:42.463 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 89 k 2023-12-25T20:05:42.463 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 298 k 2023-12-25T20:05:42.463 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2023-12-25T20:05:42.547 INFO:teuthology.orchestra.run.smithi050.stdout:rbd-fuse-16.2.4-0.el8.x86_64.rpm 1.0 MB/s | 89 kB 00:00 2023-12-25T20:05:42.548 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2023-12-25T20:05:42.548 INFO:teuthology.orchestra.run.smithi050.stdout:Total 1.0 MB/s | 89 kB 00:00 2023-12-25T20:05:42.548 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2023-12-25T20:05:42.559 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2023-12-25T20:05:42.559 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2023-12-25T20:05:42.575 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2023-12-25T20:05:42.612 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2023-12-25T20:05:42.615 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2023-12-25T20:05:42.727 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2023-12-25T20:05:42.943 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2023-12-25T20:05:43.249 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2023-12-25T20:05:43.257 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2023-12-25T20:05:43.257 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:05:43.257 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2023-12-25T20:05:43.257 INFO:teuthology.orchestra.run.smithi160.stdout: rbd-mirror-2:16.2.4-0.el8.x86_64 2023-12-25T20:05:43.257 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:05:43.257 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2023-12-25T20:05:43.642 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install rbd-nbd 2023-12-25T20:05:43.943 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2023-12-25T20:05:43.943 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-25T20:05:43.943 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2023-12-25T20:05:43.943 INFO:teuthology.orchestra.run.smithi050.stdout: rbd-fuse-2:16.2.4-0.el8.x86_64 2023-12-25T20:05:43.943 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-25T20:05:43.943 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-12-25T20:05:44.171 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install rbd-mirror 2023-12-25T20:05:44.512 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:02:55 ago on Mon 25 Dec 2023 08:02:49 PM UTC. 2023-12-25T20:05:45.071 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:02:54 ago on Mon 25 Dec 2023 08:02:51 PM UTC. 2023-12-25T20:05:46.212 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2023-12-25T20:05:46.212 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-12-25T20:05:46.213 INFO:teuthology.orchestra.run.smithi160.stdout: Package Architecture Version Repository Size 2023-12-25T20:05:46.213 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-12-25T20:05:46.213 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2023-12-25T20:05:46.213 INFO:teuthology.orchestra.run.smithi160.stdout: rbd-nbd x86_64 2:16.2.4-0.el8 ceph 173 k 2023-12-25T20:05:46.213 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:05:46.213 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2023-12-25T20:05:46.213 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-12-25T20:05:46.213 INFO:teuthology.orchestra.run.smithi160.stdout:Install 1 Package 2023-12-25T20:05:46.213 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:05:46.214 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 173 k 2023-12-25T20:05:46.214 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 649 k 2023-12-25T20:05:46.214 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2023-12-25T20:05:46.256 INFO:teuthology.orchestra.run.smithi160.stdout:rbd-nbd-16.2.4-0.el8.x86_64.rpm 4.0 MB/s | 173 kB 00:00 2023-12-25T20:05:46.257 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2023-12-25T20:05:46.257 INFO:teuthology.orchestra.run.smithi160.stdout:Total 3.9 MB/s | 173 kB 00:00 2023-12-25T20:05:46.257 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2023-12-25T20:05:46.268 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2023-12-25T20:05:46.268 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2023-12-25T20:05:46.317 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2023-12-25T20:05:46.320 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2023-12-25T20:05:46.435 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2023-12-25T20:05:46.689 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2023-12-25T20:05:46.872 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-12-25T20:05:46.873 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-12-25T20:05:46.873 INFO:teuthology.orchestra.run.smithi050.stdout: Package Architecture Version Repository Size 2023-12-25T20:05:46.873 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-12-25T20:05:46.873 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2023-12-25T20:05:46.873 INFO:teuthology.orchestra.run.smithi050.stdout: rbd-mirror x86_64 2:16.2.4-0.el8 ceph 3.7 M 2023-12-25T20:05:46.873 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-25T20:05:46.873 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2023-12-25T20:05:46.873 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-12-25T20:05:46.873 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2023-12-25T20:05:46.873 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-25T20:05:46.874 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 3.7 M 2023-12-25T20:05:46.874 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 18 M 2023-12-25T20:05:46.874 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2023-12-25T20:05:46.957 INFO:teuthology.orchestra.run.smithi050.stdout:rbd-mirror-16.2.4-0.el8.x86_64.rpm 45 MB/s | 3.7 MB 00:00 2023-12-25T20:05:46.958 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2023-12-25T20:05:46.958 INFO:teuthology.orchestra.run.smithi050.stdout:Total 44 MB/s | 3.7 MB 00:00 2023-12-25T20:05:46.958 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2023-12-25T20:05:46.968 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2023-12-25T20:05:46.971 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2023-12-25T20:05:46.971 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2023-12-25T20:05:47.040 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2023-12-25T20:05:47.043 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2023-12-25T20:05:47.560 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2023-12-25T20:05:47.676 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2023-12-25T20:05:47.676 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:05:47.676 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2023-12-25T20:05:47.676 INFO:teuthology.orchestra.run.smithi160.stdout: rbd-nbd-2:16.2.4-0.el8.x86_64 2023-12-25T20:05:47.676 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:05:47.676 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2023-12-25T20:05:47.794 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2023-12-25T20:05:47.893 DEBUG:teuthology.parallel:result is None 2023-12-25T20:05:48.237 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2023-12-25T20:05:48.975 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2023-12-25T20:05:48.975 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-25T20:05:48.975 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2023-12-25T20:05:48.976 INFO:teuthology.orchestra.run.smithi050.stdout: rbd-mirror-2:16.2.4-0.el8.x86_64 2023-12-25T20:05:48.976 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-25T20:05:48.976 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-12-25T20:05:49.209 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install rbd-nbd 2023-12-25T20:05:50.114 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:02:59 ago on Mon 25 Dec 2023 08:02:51 PM UTC. 2023-12-25T20:05:51.922 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-12-25T20:05:51.922 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-12-25T20:05:51.922 INFO:teuthology.orchestra.run.smithi050.stdout: Package Architecture Version Repository Size 2023-12-25T20:05:51.922 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-12-25T20:05:51.922 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2023-12-25T20:05:51.923 INFO:teuthology.orchestra.run.smithi050.stdout: rbd-nbd x86_64 2:16.2.4-0.el8 ceph 173 k 2023-12-25T20:05:51.923 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-25T20:05:51.923 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2023-12-25T20:05:51.923 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-12-25T20:05:51.923 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2023-12-25T20:05:51.923 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-25T20:05:51.923 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 173 k 2023-12-25T20:05:51.923 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 649 k 2023-12-25T20:05:51.924 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2023-12-25T20:05:51.968 INFO:teuthology.orchestra.run.smithi050.stdout:rbd-nbd-16.2.4-0.el8.x86_64.rpm 3.9 MB/s | 173 kB 00:00 2023-12-25T20:05:51.968 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2023-12-25T20:05:51.968 INFO:teuthology.orchestra.run.smithi050.stdout:Total 3.8 MB/s | 173 kB 00:00 2023-12-25T20:05:51.969 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2023-12-25T20:05:51.980 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2023-12-25T20:05:51.980 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2023-12-25T20:05:52.033 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2023-12-25T20:05:52.036 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2023-12-25T20:05:52.169 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2023-12-25T20:05:52.393 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2023-12-25T20:05:52.724 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2023-12-25T20:05:53.467 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2023-12-25T20:05:53.467 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-25T20:05:53.467 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2023-12-25T20:05:53.467 INFO:teuthology.orchestra.run.smithi050.stdout: rbd-nbd-2:16.2.4-0.el8.x86_64 2023-12-25T20:05:53.467 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-25T20:05:53.468 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-12-25T20:05:53.708 DEBUG:teuthology.parallel:result is None 2023-12-25T20:05:53.708 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2023-12-25T20:05:53.708 INFO:teuthology.packaging:ref: None 2023-12-25T20:05:53.708 INFO:teuthology.packaging:tag: v16.2.4 2023-12-25T20:05:53.709 INFO:teuthology.packaging:branch: None 2023-12-25T20:05:53.709 INFO:teuthology.packaging:sha1: b14a0a106b8cffae0144ff9fb83d9ce1af1a4bd2 2023-12-25T20:05:53.709 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-12-25T20:05:53.852 DEBUG:teuthology.orchestra.run.smithi050:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-12-25T20:05:53.877 INFO:teuthology.orchestra.run.smithi050.stdout:16.2.4-0.el8 2023-12-25T20:05:53.878 INFO:teuthology.packaging:The installed version of ceph is 16.2.4-0.el8 2023-12-25T20:05:53.878 INFO:teuthology.task.install:The correct ceph version 16.2.4-0 is installed. 2023-12-25T20:05:53.880 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2023-12-25T20:05:53.880 INFO:teuthology.packaging:ref: None 2023-12-25T20:05:53.880 INFO:teuthology.packaging:tag: v16.2.4 2023-12-25T20:05:53.880 INFO:teuthology.packaging:branch: None 2023-12-25T20:05:53.880 INFO:teuthology.packaging:sha1: b14a0a106b8cffae0144ff9fb83d9ce1af1a4bd2 2023-12-25T20:05:53.881 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-12-25T20:05:54.102 DEBUG:teuthology.orchestra.run.smithi160:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-12-25T20:05:54.129 INFO:teuthology.orchestra.run.smithi160.stdout:16.2.4-0.el8 2023-12-25T20:05:54.130 INFO:teuthology.packaging:The installed version of ceph is 16.2.4-0.el8 2023-12-25T20:05:54.130 INFO:teuthology.task.install:The correct ceph version 16.2.4-0 is installed. 2023-12-25T20:05:54.131 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-12-25T20:05:54.132 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2023-12-25T20:05:54.132 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-12-25T20:05:54.161 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2023-12-25T20:05:54.161 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-12-25T20:05:54.203 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-12-25T20:05:54.203 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2023-12-25T20:05:54.203 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/usr/bin/daemon-helper 2023-12-25T20:05:54.232 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-12-25T20:05:54.300 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2023-12-25T20:05:54.301 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd of=/usr/bin/daemon-helper 2023-12-25T20:05:54.331 DEBUG:teuthology.orchestra.run.smithi160:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-12-25T20:05:54.402 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-12-25T20:05:54.402 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2023-12-25T20:05:54.402 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/usr/bin/adjust-ulimits 2023-12-25T20:05:54.431 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-12-25T20:05:54.499 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2023-12-25T20:05:54.500 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd of=/usr/bin/adjust-ulimits 2023-12-25T20:05:54.530 DEBUG:teuthology.orchestra.run.smithi160:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-12-25T20:05:54.600 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-12-25T20:05:54.601 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2023-12-25T20:05:54.601 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/usr/bin/stdin-killer 2023-12-25T20:05:54.628 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-12-25T20:05:54.696 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2023-12-25T20:05:54.696 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd of=/usr/bin/stdin-killer 2023-12-25T20:05:54.726 DEBUG:teuthology.orchestra.run.smithi160:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-12-25T20:05:54.794 INFO:teuthology.run_tasks:Running task print... 2023-12-25T20:05:54.805 INFO:teuthology.task.print:**** done install task... 2023-12-25T20:05:54.805 INFO:teuthology.run_tasks:Running task cephadm... 2023-12-25T20:05:54.893 INFO:tasks.cephadm:Config: {'avoid_pacific_features': True, 'cephadm_branch': 'v16.2.4', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'image': 'docker.io/ceph/ceph:v16.2.4', 'roleless': True, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mon pg warn min per osd': 0}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': 'b14a0a106b8cffae0144ff9fb83d9ce1af1a4bd2'} 2023-12-25T20:05:54.893 INFO:tasks.cephadm:Cluster image is docker.io/ceph/ceph:v16.2.4 2023-12-25T20:05:54.894 INFO:tasks.cephadm:Cluster fsid is 02c8f25c-a361-11ee-95a5-87774f69a715 2023-12-25T20:05:54.894 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2023-12-25T20:05:54.894 INFO:tasks.cephadm:No mon roles; fabricating mons 2023-12-25T20:05:54.894 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi050': '172.21.15.50', 'mon.smithi160': '172.21.15.160'} 2023-12-25T20:05:54.894 INFO:tasks.cephadm:Normalizing hostnames... 2023-12-25T20:05:54.895 DEBUG:teuthology.orchestra.run.smithi050:> sudo hostname $(hostname -s) 2023-12-25T20:05:54.923 DEBUG:teuthology.orchestra.run.smithi160:> sudo hostname $(hostname -s) 2023-12-25T20:05:54.952 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref v16.2.4)... 2023-12-25T20:05:54.953 DEBUG:teuthology.orchestra.run.smithi050:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v16.2.4/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-12-25T20:05:55.305 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-rw-r--. 1 ubuntu ubuntu 292005 Dec 25 20:05 /home/ubuntu/cephtest/cephadm 2023-12-25T20:05:55.305 DEBUG:teuthology.orchestra.run.smithi160:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v16.2.4/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-12-25T20:05:55.597 INFO:teuthology.orchestra.run.smithi160.stdout:-rw-rw-r--. 1 ubuntu ubuntu 292005 Dec 25 20:05 /home/ubuntu/cephtest/cephadm 2023-12-25T20:05:55.598 DEBUG:teuthology.orchestra.run.smithi050:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-12-25T20:05:55.622 DEBUG:teuthology.orchestra.run.smithi160:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-12-25T20:05:55.654 DEBUG:teuthology.orchestra.run.smithi050:> sudo mkdir -p /etc/ceph 2023-12-25T20:05:55.697 DEBUG:teuthology.orchestra.run.smithi160:> sudo mkdir -p /etc/ceph 2023-12-25T20:05:55.734 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod 777 /etc/ceph 2023-12-25T20:05:55.771 DEBUG:teuthology.orchestra.run.smithi160:> sudo chmod 777 /etc/ceph 2023-12-25T20:05:55.804 INFO:tasks.cephadm:Writing seed config... 2023-12-25T20:05:55.805 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2023-12-25T20:05:55.805 INFO:tasks.cephadm: override: [client] debug client = 20 2023-12-25T20:05:55.805 INFO:tasks.cephadm: override: [client] debug ms = 1 2023-12-25T20:05:55.805 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2023-12-25T20:05:55.805 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2023-12-25T20:05:55.806 INFO:tasks.cephadm: override: [global] mon pg warn min per osd = 0 2023-12-25T20:05:55.806 INFO:tasks.cephadm: override: [mds] debug mds = 20 2023-12-25T20:05:55.806 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2023-12-25T20:05:55.806 INFO:tasks.cephadm: override: [mds] debug ms = 1 2023-12-25T20:05:55.806 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2023-12-25T20:05:55.806 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2023-12-25T20:05:55.806 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2023-12-25T20:05:55.806 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2023-12-25T20:05:55.806 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2023-12-25T20:05:55.806 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2023-12-25T20:05:55.806 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2023-12-25T20:05:55.806 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2023-12-25T20:05:55.806 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2023-12-25T20:05:55.806 INFO:tasks.cephadm: override: [mon] debug mon = 20 2023-12-25T20:05:55.806 INFO:tasks.cephadm: override: [mon] debug ms = 1 2023-12-25T20:05:55.806 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2023-12-25T20:05:55.807 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2023-12-25T20:05:55.807 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2023-12-25T20:05:55.807 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2023-12-25T20:05:55.807 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2023-12-25T20:05:55.807 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2023-12-25T20:05:55.807 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2023-12-25T20:05:55.807 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2023-12-25T20:05:55.807 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2023-12-25T20:05:55.807 INFO:tasks.cephadm: override: [osd] debug ms = 1 2023-12-25T20:05:55.807 INFO:tasks.cephadm: override: [osd] debug osd = 20 2023-12-25T20:05:55.807 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2023-12-25T20:05:55.807 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2023-12-25T20:05:55.807 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2023-12-25T20:05:55.807 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2023-12-25T20:05:55.807 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2023-12-25T20:05:55.808 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2023-12-25T20:05:55.808 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2023-12-25T20:05:55.808 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2023-12-25T20:05:55.808 DEBUG:teuthology.orchestra.run.smithi050:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2023-12-25T20:05:55.833 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 02c8f25c-a361-11ee-95a5-87774f69a715 mon pg warn min per osd = 0 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2023-12-25T20:05:55.834 DEBUG:teuthology.orchestra.run.smithi050:mon.smithi050> sudo journalctl -f -n 0 -u ceph-02c8f25c-a361-11ee-95a5-87774f69a715@mon.smithi050.service 2023-12-25T20:05:55.876 INFO:tasks.cephadm:Bootstrapping... 2023-12-25T20:05:55.876 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 -v bootstrap --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.50 && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2023-12-25T20:05:55.908 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:-- Logs begin at Mon 2023-12-25 19:53:27 UTC. -- 2023-12-25T20:05:56.152 INFO:teuthology.orchestra.run.smithi050.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2023-12-25T20:05:57.062 INFO:teuthology.orchestra.run.smithi050.stderr:/bin/podman: 4.6.1 2023-12-25T20:05:57.073 INFO:teuthology.orchestra.run.smithi050.stderr:Verifying podman|docker is present... 2023-12-25T20:05:57.073 INFO:teuthology.orchestra.run.smithi050.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2023-12-25T20:05:57.098 INFO:teuthology.orchestra.run.smithi050.stderr:/bin/podman: 4.6.1 2023-12-25T20:05:57.109 INFO:teuthology.orchestra.run.smithi050.stderr:Verifying lvm2 is present... 2023-12-25T20:05:57.109 INFO:teuthology.orchestra.run.smithi050.stderr:Verifying time synchronization is in place... 2023-12-25T20:05:57.109 INFO:teuthology.orchestra.run.smithi050.stderr:Running command: systemctl is-enabled chrony.service 2023-12-25T20:05:57.117 INFO:teuthology.orchestra.run.smithi050.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2023-12-25T20:05:57.118 INFO:teuthology.orchestra.run.smithi050.stderr:Running command: systemctl is-active chrony.service 2023-12-25T20:05:57.124 INFO:teuthology.orchestra.run.smithi050.stderr:systemctl: inactive 2023-12-25T20:05:57.124 INFO:teuthology.orchestra.run.smithi050.stderr:Running command: systemctl is-enabled chronyd.service 2023-12-25T20:05:57.131 INFO:teuthology.orchestra.run.smithi050.stderr:systemctl: enabled 2023-12-25T20:05:57.132 INFO:teuthology.orchestra.run.smithi050.stderr:Running command: systemctl is-active chronyd.service 2023-12-25T20:05:57.138 INFO:teuthology.orchestra.run.smithi050.stderr:systemctl: active 2023-12-25T20:05:57.139 INFO:teuthology.orchestra.run.smithi050.stderr:Unit chronyd.service is enabled and running 2023-12-25T20:05:57.139 INFO:teuthology.orchestra.run.smithi050.stderr:Repeating the final host check... 2023-12-25T20:05:57.139 INFO:teuthology.orchestra.run.smithi050.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2023-12-25T20:05:57.160 INFO:teuthology.orchestra.run.smithi050.stderr:/bin/podman: 4.6.1 2023-12-25T20:05:57.175 INFO:teuthology.orchestra.run.smithi050.stderr:podman|docker (/bin/podman) is present 2023-12-25T20:05:57.175 INFO:teuthology.orchestra.run.smithi050.stderr:systemctl is present 2023-12-25T20:05:57.175 INFO:teuthology.orchestra.run.smithi050.stderr:lvcreate is present 2023-12-25T20:05:57.175 INFO:teuthology.orchestra.run.smithi050.stderr:Running command: systemctl is-enabled chrony.service 2023-12-25T20:05:57.181 INFO:teuthology.orchestra.run.smithi050.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2023-12-25T20:05:57.182 INFO:teuthology.orchestra.run.smithi050.stderr:Running command: systemctl is-active chrony.service 2023-12-25T20:05:57.187 INFO:teuthology.orchestra.run.smithi050.stderr:systemctl: inactive 2023-12-25T20:05:57.188 INFO:teuthology.orchestra.run.smithi050.stderr:Running command: systemctl is-enabled chronyd.service 2023-12-25T20:05:57.194 INFO:teuthology.orchestra.run.smithi050.stderr:systemctl: enabled 2023-12-25T20:05:57.194 INFO:teuthology.orchestra.run.smithi050.stderr:Running command: systemctl is-active chronyd.service 2023-12-25T20:05:57.200 INFO:teuthology.orchestra.run.smithi050.stderr:systemctl: active 2023-12-25T20:05:57.201 INFO:teuthology.orchestra.run.smithi050.stderr:Unit chronyd.service is enabled and running 2023-12-25T20:05:57.201 INFO:teuthology.orchestra.run.smithi050.stderr:Host looks OK 2023-12-25T20:05:57.201 INFO:teuthology.orchestra.run.smithi050.stderr:Cluster fsid: 02c8f25c-a361-11ee-95a5-87774f69a715 2023-12-25T20:05:57.201 INFO:teuthology.orchestra.run.smithi050.stderr:Acquiring lock 140098145065448 on /run/cephadm/02c8f25c-a361-11ee-95a5-87774f69a715.lock 2023-12-25T20:05:57.202 INFO:teuthology.orchestra.run.smithi050.stderr:Lock 140098145065448 acquired on /run/cephadm/02c8f25c-a361-11ee-95a5-87774f69a715.lock 2023-12-25T20:05:57.202 INFO:teuthology.orchestra.run.smithi050.stderr:Verifying IP 172.21.15.50 port 3300 ... 2023-12-25T20:05:57.202 INFO:teuthology.orchestra.run.smithi050.stderr:Verifying IP 172.21.15.50 port 6789 ... 2023-12-25T20:05:57.202 INFO:teuthology.orchestra.run.smithi050.stderr:Base mon IP is 172.21.15.50, final addrv is [v2:172.21.15.50:3300,v1:172.21.15.50:6789] 2023-12-25T20:05:57.202 INFO:teuthology.orchestra.run.smithi050.stderr:Running command: /sbin/ip route ls 2023-12-25T20:05:57.204 INFO:teuthology.orchestra.run.smithi050.stderr:/sbin/ip: default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.50 metric 100 2023-12-25T20:05:57.204 INFO:teuthology.orchestra.run.smithi050.stderr:/sbin/ip: 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.50 metric 100 2023-12-25T20:05:57.205 INFO:teuthology.orchestra.run.smithi050.stderr:Running command: /sbin/ip -6 route ls 2023-12-25T20:05:57.208 INFO:teuthology.orchestra.run.smithi050.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2023-12-25T20:05:57.208 INFO:teuthology.orchestra.run.smithi050.stderr:/sbin/ip: fe80::/64 dev ens1f0 proto kernel metric 256 pref medium 2023-12-25T20:05:57.208 INFO:teuthology.orchestra.run.smithi050.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 1024 expires 1787sec hoplimit 64 pref medium 2023-12-25T20:05:57.208 INFO:teuthology.orchestra.run.smithi050.stderr:Running command: /sbin/ip -6 addr ls 2023-12-25T20:05:57.211 INFO:teuthology.orchestra.run.smithi050.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2023-12-25T20:05:57.211 INFO:teuthology.orchestra.run.smithi050.stderr:/sbin/ip: inet6 ::1/128 scope host 2023-12-25T20:05:57.211 INFO:teuthology.orchestra.run.smithi050.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2023-12-25T20:05:57.211 INFO:teuthology.orchestra.run.smithi050.stderr:/sbin/ip: 4: ens1f0: mtu 1500 state UP qlen 1000 2023-12-25T20:05:57.211 INFO:teuthology.orchestra.run.smithi050.stderr:/sbin/ip: inet6 fe80::ec4:7aff:febd:1584/64 scope link 2023-12-25T20:05:57.211 INFO:teuthology.orchestra.run.smithi050.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2023-12-25T20:05:57.212 INFO:teuthology.orchestra.run.smithi050.stderr:Mon IP 172.21.15.50 is in CIDR network 172.21.0.0/20 2023-12-25T20:05:57.212 INFO:teuthology.orchestra.run.smithi050.stderr:- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2023-12-25T20:05:57.213 INFO:teuthology.orchestra.run.smithi050.stderr:Pulling container image docker.io/ceph/ceph:v16.2.4... 2023-12-25T20:05:57.213 INFO:teuthology.orchestra.run.smithi050.stderr:Running command: /bin/podman pull docker.io/ceph/ceph:v16.2.4 2023-12-25T20:05:57.309 INFO:teuthology.orchestra.run.smithi050.stderr:/bin/podman: Trying to pull docker.io/ceph/ceph:v16.2.4... 2023-12-25T20:06:03.907 INFO:teuthology.orchestra.run.smithi050.stderr:/bin/podman: Getting image source signatures 2023-12-25T20:06:03.911 INFO:teuthology.orchestra.run.smithi050.stderr:/bin/podman: Copying blob sha256:551101690791055467b1840be133b2e6b029d263771bc6c46a96bb7acdacac39 2023-12-25T20:06:03.911 INFO:teuthology.orchestra.run.smithi050.stderr:/bin/podman: Copying blob sha256:7a0437f04f83f084b7ed68ad9c4a4947e12fc4e1b006b38129bac89114ec3621 2023-12-25T20:06:15.809 INFO:teuthology.orchestra.run.smithi050.stderr:/bin/podman: Copying config sha256:8d91d370c2b86c07de46146aba8d36718eaefa69b1880c77fa312fda6efd7d29 2023-12-25T20:06:15.884 INFO:teuthology.orchestra.run.smithi050.stderr:/bin/podman: Writing manifest to image destination 2023-12-25T20:06:16.255 INFO:teuthology.orchestra.run.smithi050.stderr:/bin/podman: 8d91d370c2b86c07de46146aba8d36718eaefa69b1880c77fa312fda6efd7d29 2023-12-25T20:06:16.267 INFO:teuthology.orchestra.run.smithi050.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi050 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --version 2023-12-25T20:06:16.853 INFO:teuthology.orchestra.run.smithi050.stderr:ceph: ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable) 2023-12-25T20:06:17.237 INFO:teuthology.orchestra.run.smithi050.stderr:Ceph version: ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable) 2023-12-25T20:06:17.238 INFO:teuthology.orchestra.run.smithi050.stderr:Extracting ceph user uid/gid from container image... 2023-12-25T20:06:17.238 INFO:teuthology.orchestra.run.smithi050.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi050 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 -c %u %g /var/lib/ceph 2023-12-25T20:06:17.536 INFO:teuthology.orchestra.run.smithi050.stderr:stat: 167 167 2023-12-25T20:06:17.900 INFO:teuthology.orchestra.run.smithi050.stderr:Creating initial keys... 2023-12-25T20:06:17.900 INFO:teuthology.orchestra.run.smithi050.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi050 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2023-12-25T20:06:18.458 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-authtool: AQC64IllklloGhAA0VMb8rDwWzSZbiT/P/ATTg== 2023-12-25T20:06:18.836 INFO:teuthology.orchestra.run.smithi050.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi050 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2023-12-25T20:06:19.169 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-authtool: AQC74IlliwbWBxAAlok5OC5bqaWj/jXUyOuz0g== 2023-12-25T20:06:19.487 INFO:teuthology.orchestra.run.smithi050.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi050 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2023-12-25T20:06:19.750 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-authtool: AQC74IllWTG+KxAAG0SurcoDw2fS9qaAhYdZeQ== 2023-12-25T20:06:20.087 INFO:teuthology.orchestra.run.smithi050.stderr:Creating initial monmap... 2023-12-25T20:06:20.087 INFO:teuthology.orchestra.run.smithi050.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/monmaptool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi050 -e CEPH_USE_RANDOM_NONCE=1 -v /tmp/ceph-tmpokqszfmp:/tmp/monmap:z docker.io/ceph/ceph:v16.2.4 --create --clobber --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 --addv smithi050 [v2:172.21.15.50:3300,v1:172.21.15.50:6789] /tmp/monmap 2023-12-25T20:06:20.383 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2023-12-25T20:06:20.384 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to 02c8f25c-a361-11ee-95a5-87774f69a715 2023-12-25T20:06:20.384 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2023-12-25T20:06:20.729 INFO:teuthology.orchestra.run.smithi050.stderr:monmaptool for smithi050 [v2:172.21.15.50:3300,v1:172.21.15.50:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2023-12-25T20:06:20.729 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/monmaptool: set fsid to 02c8f25c-a361-11ee-95a5-87774f69a715 2023-12-25T20:06:20.729 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2023-12-25T20:06:20.730 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-12-25T20:06:20.730 INFO:teuthology.orchestra.run.smithi050.stderr:Creating mon... 2023-12-25T20:06:20.730 INFO:teuthology.orchestra.run.smithi050.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-mon --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi050 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/02c8f25c-a361-11ee-95a5-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/02c8f25c-a361-11ee-95a5-87774f69a715/mon.smithi050:/var/lib/ceph/mon/ceph-smithi050:z -v /tmp/ceph-tmpjd6rzwuv:/tmp/keyring:z -v /tmp/ceph-tmpokqszfmp:/tmp/monmap:z docker.io/ceph/ceph:v16.2.4 --mkfs -i smithi050 --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -c /dev/null --monmap /tmp/monmap --keyring /tmp/keyring --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true 2023-12-25T20:06:21.059 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.026+0000 7f9198a17700 0 set uid:gid to 167:167 (ceph:ceph) 2023-12-25T20:06:21.059 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.026+0000 7f9198a17700 1 imported monmap: 2023-12-25T20:06:21.059 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: epoch 0 2023-12-25T20:06:21.060 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: fsid 02c8f25c-a361-11ee-95a5-87774f69a715 2023-12-25T20:06:21.060 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: last_changed 2023-12-25T20:06:20.367705+0000 2023-12-25T20:06:21.060 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: created 2023-12-25T20:06:20.367705+0000 2023-12-25T20:06:21.060 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: min_mon_release 0 (unknown) 2023-12-25T20:06:21.060 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: election_strategy: 1 2023-12-25T20:06:21.060 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: 0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.smithi050 2023-12-25T20:06:21.060 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: 2023-12-25T20:06:21.060 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.026+0000 7f9198a17700 0 /usr/bin/ceph-mon: set fsid to 02c8f25c-a361-11ee-95a5-87774f69a715 2023-12-25T20:06:21.060 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.027+0000 7f9198a17700 4 rocksdb: RocksDB version: 6.8.1 2023-12-25T20:06:21.060 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: 2023-12-25T20:06:21.060 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.027+0000 7f9198a17700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2023-12-25T20:06:21.060 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.027+0000 7f9198a17700 4 rocksdb: Compile date May 13 2021 2023-12-25T20:06:21.060 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.027+0000 7f9198a17700 4 rocksdb: DB SUMMARY 2023-12-25T20:06:21.060 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: 2023-12-25T20:06:21.061 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.027+0000 7f9198a17700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi050/store.db dir, Total Num: 0, files: 2023-12-25T20:06:21.061 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: 2023-12-25T20:06:21.061 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.027+0000 7f9198a17700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi050/store.db: 2023-12-25T20:06:21.061 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: 2023-12-25T20:06:21.061 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.027+0000 7f9198a17700 4 rocksdb: Options.error_if_exists: 0 2023-12-25T20:06:21.061 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.027+0000 7f9198a17700 4 rocksdb: Options.create_if_missing: 1 2023-12-25T20:06:21.061 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.027+0000 7f9198a17700 4 rocksdb: Options.paranoid_checks: 1 2023-12-25T20:06:21.061 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.027+0000 7f9198a17700 4 rocksdb: Options.env: 0x5607a8df51c0 2023-12-25T20:06:21.061 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.027+0000 7f9198a17700 4 rocksdb: Options.fs: Posix File System 2023-12-25T20:06:21.061 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.027+0000 7f9198a17700 4 rocksdb: Options.info_log: 0x5607aabb6ec0 2023-12-25T20:06:21.061 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.027+0000 7f9198a17700 4 rocksdb: Options.max_file_opening_threads: 16 2023-12-25T20:06:21.061 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.027+0000 7f9198a17700 4 rocksdb: Options.statistics: (nil) 2023-12-25T20:06:21.061 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.027+0000 7f9198a17700 4 rocksdb: Options.use_fsync: 0 2023-12-25T20:06:21.062 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.027+0000 7f9198a17700 4 rocksdb: Options.max_log_file_size: 0 2023-12-25T20:06:21.062 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.027+0000 7f9198a17700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2023-12-25T20:06:21.062 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.027+0000 7f9198a17700 4 rocksdb: Options.log_file_time_to_roll: 0 2023-12-25T20:06:21.062 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.027+0000 7f9198a17700 4 rocksdb: Options.keep_log_file_num: 1000 2023-12-25T20:06:21.062 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.027+0000 7f9198a17700 4 rocksdb: Options.recycle_log_file_num: 0 2023-12-25T20:06:21.062 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.027+0000 7f9198a17700 4 rocksdb: Options.allow_fallocate: 1 2023-12-25T20:06:21.062 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.027+0000 7f9198a17700 4 rocksdb: Options.allow_mmap_reads: 0 2023-12-25T20:06:21.062 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.027+0000 7f9198a17700 4 rocksdb: Options.allow_mmap_writes: 0 2023-12-25T20:06:21.062 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.027+0000 7f9198a17700 4 rocksdb: Options.use_direct_reads: 0 2023-12-25T20:06:21.062 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.027+0000 7f9198a17700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-12-25T20:06:21.062 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.027+0000 7f9198a17700 4 rocksdb: Options.create_missing_column_families: 0 2023-12-25T20:06:21.062 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.027+0000 7f9198a17700 4 rocksdb: Options.db_log_dir: 2023-12-25T20:06:21.062 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.027+0000 7f9198a17700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi050/store.db 2023-12-25T20:06:21.063 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.027+0000 7f9198a17700 4 rocksdb: Options.table_cache_numshardbits: 6 2023-12-25T20:06:21.063 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.027+0000 7f9198a17700 4 rocksdb: Options.max_subcompactions: 1 2023-12-25T20:06:21.063 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.027+0000 7f9198a17700 4 rocksdb: Options.max_background_flushes: -1 2023-12-25T20:06:21.063 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.027+0000 7f9198a17700 4 rocksdb: Options.WAL_ttl_seconds: 0 2023-12-25T20:06:21.063 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.027+0000 7f9198a17700 4 rocksdb: Options.WAL_size_limit_MB: 0 2023-12-25T20:06:21.063 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.027+0000 7f9198a17700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-12-25T20:06:21.063 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.027+0000 7f9198a17700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2023-12-25T20:06:21.063 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.027+0000 7f9198a17700 4 rocksdb: Options.is_fd_close_on_exec: 1 2023-12-25T20:06:21.063 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.027+0000 7f9198a17700 4 rocksdb: Options.advise_random_on_open: 1 2023-12-25T20:06:21.063 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.027+0000 7f9198a17700 4 rocksdb: Options.db_write_buffer_size: 0 2023-12-25T20:06:21.063 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.027+0000 7f9198a17700 4 rocksdb: Options.write_buffer_manager: 0x5607aabbd530 2023-12-25T20:06:21.063 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.027+0000 7f9198a17700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2023-12-25T20:06:21.064 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.027+0000 7f9198a17700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2023-12-25T20:06:21.064 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.027+0000 7f9198a17700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2023-12-25T20:06:21.064 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.027+0000 7f9198a17700 4 rocksdb: Options.use_adaptive_mutex: 0 2023-12-25T20:06:21.064 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.027+0000 7f9198a17700 4 rocksdb: Options.rate_limiter: (nil) 2023-12-25T20:06:21.064 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.027+0000 7f9198a17700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-12-25T20:06:21.064 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.027+0000 7f9198a17700 4 rocksdb: Options.wal_recovery_mode: 2 2023-12-25T20:06:21.064 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.027+0000 7f9198a17700 4 rocksdb: Options.enable_thread_tracking: 0 2023-12-25T20:06:21.064 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.027+0000 7f9198a17700 4 rocksdb: Options.enable_pipelined_write: 0 2023-12-25T20:06:21.064 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.027+0000 7f9198a17700 4 rocksdb: Options.unordered_write: 0 2023-12-25T20:06:21.064 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.027+0000 7f9198a17700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2023-12-25T20:06:21.064 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.027+0000 7f9198a17700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-12-25T20:06:21.064 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.027+0000 7f9198a17700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2023-12-25T20:06:21.064 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.027+0000 7f9198a17700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2023-12-25T20:06:21.064 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.027+0000 7f9198a17700 4 rocksdb: Options.row_cache: None 2023-12-25T20:06:21.065 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.027+0000 7f9198a17700 4 rocksdb: Options.wal_filter: None 2023-12-25T20:06:21.065 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.027+0000 7f9198a17700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2023-12-25T20:06:21.065 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.027+0000 7f9198a17700 4 rocksdb: Options.allow_ingest_behind: 0 2023-12-25T20:06:21.065 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.027+0000 7f9198a17700 4 rocksdb: Options.preserve_deletes: 0 2023-12-25T20:06:21.065 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.027+0000 7f9198a17700 4 rocksdb: Options.two_write_queues: 0 2023-12-25T20:06:21.065 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.027+0000 7f9198a17700 4 rocksdb: Options.manual_wal_flush: 0 2023-12-25T20:06:21.065 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.027+0000 7f9198a17700 4 rocksdb: Options.atomic_flush: 0 2023-12-25T20:06:21.065 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.027+0000 7f9198a17700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-12-25T20:06:21.065 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.027+0000 7f9198a17700 4 rocksdb: Options.persist_stats_to_disk: 0 2023-12-25T20:06:21.065 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.027+0000 7f9198a17700 4 rocksdb: Options.write_dbid_to_manifest: 0 2023-12-25T20:06:21.065 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.027+0000 7f9198a17700 4 rocksdb: Options.log_readahead_size: 0 2023-12-25T20:06:21.065 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.027+0000 7f9198a17700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2023-12-25T20:06:21.065 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.027+0000 7f9198a17700 4 rocksdb: Options.max_background_jobs: 2 2023-12-25T20:06:21.065 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.027+0000 7f9198a17700 4 rocksdb: Options.max_background_compactions: -1 2023-12-25T20:06:21.066 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.027+0000 7f9198a17700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2023-12-25T20:06:21.066 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.027+0000 7f9198a17700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-12-25T20:06:21.066 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.027+0000 7f9198a17700 4 rocksdb: Options.delayed_write_rate : 16777216 2023-12-25T20:06:21.066 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.027+0000 7f9198a17700 4 rocksdb: Options.max_total_wal_size: 0 2023-12-25T20:06:21.066 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.027+0000 7f9198a17700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-12-25T20:06:21.066 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.027+0000 7f9198a17700 4 rocksdb: Options.stats_dump_period_sec: 600 2023-12-25T20:06:21.066 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.027+0000 7f9198a17700 4 rocksdb: Options.stats_persist_period_sec: 600 2023-12-25T20:06:21.066 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.027+0000 7f9198a17700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2023-12-25T20:06:21.066 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.027+0000 7f9198a17700 4 rocksdb: Options.max_open_files: -1 2023-12-25T20:06:21.066 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.027+0000 7f9198a17700 4 rocksdb: Options.bytes_per_sync: 0 2023-12-25T20:06:21.066 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.027+0000 7f9198a17700 4 rocksdb: Options.wal_bytes_per_sync: 0 2023-12-25T20:06:21.066 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.027+0000 7f9198a17700 4 rocksdb: Options.strict_bytes_per_sync: 0 2023-12-25T20:06:21.066 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.027+0000 7f9198a17700 4 rocksdb: Options.compaction_readahead_size: 0 2023-12-25T20:06:21.067 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.027+0000 7f9198a17700 4 rocksdb: Compression algorithms supported: 2023-12-25T20:06:21.067 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.027+0000 7f9198a17700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2023-12-25T20:06:21.067 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.027+0000 7f9198a17700 4 rocksdb: kZSTD supported: 0 2023-12-25T20:06:21.067 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.027+0000 7f9198a17700 4 rocksdb: kXpressCompression supported: 0 2023-12-25T20:06:21.067 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.027+0000 7f9198a17700 4 rocksdb: kLZ4HCCompression supported: 1 2023-12-25T20:06:21.067 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.027+0000 7f9198a17700 4 rocksdb: kLZ4Compression supported: 1 2023-12-25T20:06:21.067 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.027+0000 7f9198a17700 4 rocksdb: kBZip2Compression supported: 0 2023-12-25T20:06:21.067 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.027+0000 7f9198a17700 4 rocksdb: kZlibCompression supported: 1 2023-12-25T20:06:21.067 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.027+0000 7f9198a17700 4 rocksdb: kSnappyCompression supported: 1 2023-12-25T20:06:21.068 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.027+0000 7f9198a17700 4 rocksdb: Fast CRC32 supported: Supported on x86 2023-12-25T20:06:21.068 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.028+0000 7f9198a17700 4 rocksdb: [db_impl/db_impl_open.cc:273] Creating manifest 1 2023-12-25T20:06:21.068 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: 2023-12-25T20:06:21.068 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.028+0000 7f9198a17700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi050/store.db/MANIFEST-000001 2023-12-25T20:06:21.068 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: 2023-12-25T20:06:21.068 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.028+0000 7f9198a17700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2023-12-25T20:06:21.068 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: 2023-12-25T20:06:21.068 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.028+0000 7f9198a17700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-12-25T20:06:21.068 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.028+0000 7f9198a17700 4 rocksdb: Options.merge_operator: 2023-12-25T20:06:21.068 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.028+0000 7f9198a17700 4 rocksdb: Options.compaction_filter: None 2023-12-25T20:06:21.068 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.028+0000 7f9198a17700 4 rocksdb: Options.compaction_filter_factory: None 2023-12-25T20:06:21.068 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.028+0000 7f9198a17700 4 rocksdb: Options.memtable_factory: SkipListFactory 2023-12-25T20:06:21.068 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.028+0000 7f9198a17700 4 rocksdb: Options.table_factory: BlockBasedTable 2023-12-25T20:06:21.068 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.028+0000 7f9198a17700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5607aaace120) 2023-12-25T20:06:21.068 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks: 1 2023-12-25T20:06:21.069 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks_with_high_priority: 0 2023-12-25T20:06:21.069 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: pin_l0_filter_and_index_blocks_in_cache: 0 2023-12-25T20:06:21.069 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: pin_top_level_index_and_filter: 1 2023-12-25T20:06:21.069 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: index_type: 0 2023-12-25T20:06:21.069 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: data_block_index_type: 0 2023-12-25T20:06:21.069 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: index_shortening: 1 2023-12-25T20:06:21.069 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: data_block_hash_table_util_ratio: 0.750000 2023-12-25T20:06:21.069 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: hash_index_allow_collision: 1 2023-12-25T20:06:21.069 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: checksum: 1 2023-12-25T20:06:21.069 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: no_block_cache: 0 2023-12-25T20:06:21.069 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: block_cache: 0x5607aab04d10 2023-12-25T20:06:21.069 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: block_cache_name: BinnedLRUCache 2023-12-25T20:06:21.069 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: block_cache_options: 2023-12-25T20:06:21.069 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: capacity : 536870912 2023-12-25T20:06:21.070 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: num_shard_bits : 4 2023-12-25T20:06:21.070 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: strict_capacity_limit : 0 2023-12-25T20:06:21.070 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: high_pri_pool_ratio: 0.000 2023-12-25T20:06:21.070 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: block_cache_compressed: (nil) 2023-12-25T20:06:21.070 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: persistent_cache: (nil) 2023-12-25T20:06:21.070 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: block_size: 4096 2023-12-25T20:06:21.070 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: block_size_deviation: 10 2023-12-25T20:06:21.070 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: block_restart_interval: 16 2023-12-25T20:06:21.070 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: index_block_restart_interval: 1 2023-12-25T20:06:21.070 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: metadata_block_size: 4096 2023-12-25T20:06:21.070 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: partition_filters: 0 2023-12-25T20:06:21.070 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: use_delta_encoding: 1 2023-12-25T20:06:21.070 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: filter_policy: rocksdb.BuiltinBloomFilter 2023-12-25T20:06:21.070 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: whole_key_filtering: 1 2023-12-25T20:06:21.070 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: verify_compression: 0 2023-12-25T20:06:21.070 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: read_amp_bytes_per_bit: 0 2023-12-25T20:06:21.071 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: format_version: 2 2023-12-25T20:06:21.071 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: enable_index_compression: 1 2023-12-25T20:06:21.071 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: block_align: 0 2023-12-25T20:06:21.071 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: 2023-12-25T20:06:21.071 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.028+0000 7f9198a17700 4 rocksdb: Options.write_buffer_size: 33554432 2023-12-25T20:06:21.071 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.028+0000 7f9198a17700 4 rocksdb: Options.max_write_buffer_number: 2 2023-12-25T20:06:21.071 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.028+0000 7f9198a17700 4 rocksdb: Options.compression: NoCompression 2023-12-25T20:06:21.071 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.028+0000 7f9198a17700 4 rocksdb: Options.bottommost_compression: Disabled 2023-12-25T20:06:21.071 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.028+0000 7f9198a17700 4 rocksdb: Options.prefix_extractor: nullptr 2023-12-25T20:06:21.071 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.028+0000 7f9198a17700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-12-25T20:06:21.071 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.028+0000 7f9198a17700 4 rocksdb: Options.num_levels: 7 2023-12-25T20:06:21.071 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.028+0000 7f9198a17700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-12-25T20:06:21.071 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.028+0000 7f9198a17700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-12-25T20:06:21.071 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.028+0000 7f9198a17700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-12-25T20:06:21.071 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.028+0000 7f9198a17700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-12-25T20:06:21.072 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.028+0000 7f9198a17700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2023-12-25T20:06:21.072 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.028+0000 7f9198a17700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-12-25T20:06:21.072 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.028+0000 7f9198a17700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-12-25T20:06:21.072 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.028+0000 7f9198a17700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-12-25T20:06:21.072 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.028+0000 7f9198a17700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2023-12-25T20:06:21.072 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.028+0000 7f9198a17700 4 rocksdb: Options.compression_opts.window_bits: -14 2023-12-25T20:06:21.072 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.028+0000 7f9198a17700 4 rocksdb: Options.compression_opts.level: 32767 2023-12-25T20:06:21.072 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.028+0000 7f9198a17700 4 rocksdb: Options.compression_opts.strategy: 0 2023-12-25T20:06:21.072 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.028+0000 7f9198a17700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-12-25T20:06:21.072 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.028+0000 7f9198a17700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-12-25T20:06:21.072 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.028+0000 7f9198a17700 4 rocksdb: Options.compression_opts.enabled: false 2023-12-25T20:06:21.072 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.028+0000 7f9198a17700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-12-25T20:06:21.072 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.028+0000 7f9198a17700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-12-25T20:06:21.073 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.028+0000 7f9198a17700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2023-12-25T20:06:21.073 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.028+0000 7f9198a17700 4 rocksdb: Options.target_file_size_base: 67108864 2023-12-25T20:06:21.073 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.028+0000 7f9198a17700 4 rocksdb: Options.target_file_size_multiplier: 1 2023-12-25T20:06:21.073 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.028+0000 7f9198a17700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2023-12-25T20:06:21.073 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.028+0000 7f9198a17700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-12-25T20:06:21.073 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.028+0000 7f9198a17700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-12-25T20:06:21.073 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.028+0000 7f9198a17700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-12-25T20:06:21.073 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.028+0000 7f9198a17700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-12-25T20:06:21.073 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.028+0000 7f9198a17700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-12-25T20:06:21.073 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.028+0000 7f9198a17700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-12-25T20:06:21.073 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.028+0000 7f9198a17700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-12-25T20:06:21.073 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.028+0000 7f9198a17700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-12-25T20:06:21.073 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.028+0000 7f9198a17700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-12-25T20:06:21.074 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.028+0000 7f9198a17700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-12-25T20:06:21.074 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.028+0000 7f9198a17700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2023-12-25T20:06:21.074 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.028+0000 7f9198a17700 4 rocksdb: Options.arena_block_size: 4194304 2023-12-25T20:06:21.074 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.028+0000 7f9198a17700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-12-25T20:06:21.074 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.028+0000 7f9198a17700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-12-25T20:06:21.074 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.028+0000 7f9198a17700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2023-12-25T20:06:21.074 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.028+0000 7f9198a17700 4 rocksdb: Options.disable_auto_compactions: 0 2023-12-25T20:06:21.074 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.028+0000 7f9198a17700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-12-25T20:06:21.074 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.028+0000 7f9198a17700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-12-25T20:06:21.074 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.028+0000 7f9198a17700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-12-25T20:06:21.074 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.028+0000 7f9198a17700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-12-25T20:06:21.074 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.028+0000 7f9198a17700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-12-25T20:06:21.074 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.028+0000 7f9198a17700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-12-25T20:06:21.075 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.028+0000 7f9198a17700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-12-25T20:06:21.075 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.028+0000 7f9198a17700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-12-25T20:06:21.075 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.028+0000 7f9198a17700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-12-25T20:06:21.075 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.028+0000 7f9198a17700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-12-25T20:06:21.075 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.028+0000 7f9198a17700 4 rocksdb: Options.table_properties_collectors: 2023-12-25T20:06:21.075 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.028+0000 7f9198a17700 4 rocksdb: Options.inplace_update_support: 0 2023-12-25T20:06:21.075 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.028+0000 7f9198a17700 4 rocksdb: Options.inplace_update_num_locks: 10000 2023-12-25T20:06:21.075 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.028+0000 7f9198a17700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-12-25T20:06:21.075 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.028+0000 7f9198a17700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2023-12-25T20:06:21.075 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.028+0000 7f9198a17700 4 rocksdb: Options.memtable_huge_page_size: 0 2023-12-25T20:06:21.075 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.028+0000 7f9198a17700 4 rocksdb: Options.bloom_locality: 0 2023-12-25T20:06:21.075 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.028+0000 7f9198a17700 4 rocksdb: Options.max_successive_merges: 0 2023-12-25T20:06:21.076 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.028+0000 7f9198a17700 4 rocksdb: Options.optimize_filters_for_hits: 0 2023-12-25T20:06:21.076 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.028+0000 7f9198a17700 4 rocksdb: Options.paranoid_file_checks: 0 2023-12-25T20:06:21.076 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.028+0000 7f9198a17700 4 rocksdb: Options.force_consistency_checks: 0 2023-12-25T20:06:21.076 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.028+0000 7f9198a17700 4 rocksdb: Options.report_bg_io_stats: 0 2023-12-25T20:06:21.076 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.028+0000 7f9198a17700 4 rocksdb: Options.ttl: 2592000 2023-12-25T20:06:21.076 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.028+0000 7f9198a17700 4 rocksdb: Options.periodic_compaction_seconds: 0 2023-12-25T20:06:21.076 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.028+0000 7f9198a17700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi050/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-12-25T20:06:21.076 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: 2023-12-25T20:06:21.076 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.028+0000 7f9198a17700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 0 2023-12-25T20:06:21.076 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: 2023-12-25T20:06:21.076 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.029+0000 7f9198a17700 4 rocksdb: DB pointer 0x5607aabc9800 2023-12-25T20:06:21.076 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.029+0000 7f91818b0700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2023-12-25T20:06:21.076 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.029+0000 7f91818b0700 4 rocksdb: [db_impl/db_impl.cc:851] 2023-12-25T20:06:21.076 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: ** DB Stats ** 2023-12-25T20:06:21.077 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2023-12-25T20:06:21.077 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-12-25T20:06:21.077 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-12-25T20:06:21.077 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-12-25T20:06:21.077 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-12-25T20:06:21.077 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2023-12-25T20:06:21.077 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-12-25T20:06:21.077 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: 2023-12-25T20:06:21.077 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2023-12-25T20:06:21.077 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-12-25T20:06:21.077 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-12-25T20:06:21.077 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-12-25T20:06:21.077 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-12-25T20:06:21.077 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: 2023-12-25T20:06:21.078 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2023-12-25T20:06:21.078 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-12-25T20:06:21.078 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-12-25T20:06:21.078 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2023-12-25T20:06:21.078 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2023-12-25T20:06:21.078 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2023-12-25T20:06:21.078 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2023-12-25T20:06:21.078 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2023-12-25T20:06:21.078 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2023-12-25T20:06:21.078 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-12-25T20:06:21.078 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-12-25T20:06:21.078 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-12-25T20:06:21.078 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: 2023-12-25T20:06:21.078 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2023-12-25T20:06:21.079 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: 2023-12-25T20:06:21.079 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2023-12-25T20:06:21.079 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-12-25T20:06:21.079 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-12-25T20:06:21.079 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-12-25T20:06:21.079 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-12-25T20:06:21.079 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: 2023-12-25T20:06:21.079 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2023-12-25T20:06:21.079 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-12-25T20:06:21.079 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-12-25T20:06:21.079 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2023-12-25T20:06:21.079 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2023-12-25T20:06:21.079 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2023-12-25T20:06:21.080 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2023-12-25T20:06:21.080 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2023-12-25T20:06:21.080 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2023-12-25T20:06:21.080 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-12-25T20:06:21.080 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-12-25T20:06:21.080 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-12-25T20:06:21.080 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: 2023-12-25T20:06:21.080 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2023-12-25T20:06:21.080 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: 2023-12-25T20:06:21.080 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.030+0000 7f9198a17700 4 rocksdb: [db_impl/db_impl.cc:397] Shutdown: canceling all background work 2023-12-25T20:06:21.080 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.031+0000 7f9198a17700 4 rocksdb: [db_impl/db_impl.cc:573] Shutdown complete 2023-12-25T20:06:21.080 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph-mon: debug 2023-12-25T20:06:21.031+0000 7f9198a17700 0 /usr/bin/ceph-mon: created monfs at /var/lib/ceph/mon/ceph-smithi050 for mon.smithi050 2023-12-25T20:06:21.416 INFO:teuthology.orchestra.run.smithi050.stderr:create mon.smithi050 on 2023-12-25T20:06:21.417 INFO:teuthology.orchestra.run.smithi050.stderr:Running command: install -d -m0770 -o 167 -g 167 /var/run/ceph/02c8f25c-a361-11ee-95a5-87774f69a715 2023-12-25T20:06:21.421 INFO:teuthology.orchestra.run.smithi050.stderr:Running command: systemctl disable ceph.target 2023-12-25T20:06:21.428 INFO:teuthology.orchestra.run.smithi050.stderr:systemctl: Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2023-12-25T20:06:21.521 INFO:teuthology.orchestra.run.smithi050.stderr:Running command: systemctl enable ceph.target 2023-12-25T20:06:21.527 INFO:teuthology.orchestra.run.smithi050.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2023-12-25T20:06:21.614 INFO:teuthology.orchestra.run.smithi050.stderr:Running command: systemctl start ceph.target 2023-12-25T20:06:21.622 INFO:teuthology.orchestra.run.smithi050.stderr:Running command: systemctl enable ceph-02c8f25c-a361-11ee-95a5-87774f69a715.target 2023-12-25T20:06:21.628 INFO:teuthology.orchestra.run.smithi050.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-02c8f25c-a361-11ee-95a5-87774f69a715.target → /etc/systemd/system/ceph-02c8f25c-a361-11ee-95a5-87774f69a715.target. 2023-12-25T20:06:21.628 INFO:teuthology.orchestra.run.smithi050.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-02c8f25c-a361-11ee-95a5-87774f69a715.target → /etc/systemd/system/ceph-02c8f25c-a361-11ee-95a5-87774f69a715.target. 2023-12-25T20:06:21.714 INFO:teuthology.orchestra.run.smithi050.stderr:Running command: systemctl start ceph-02c8f25c-a361-11ee-95a5-87774f69a715.target 2023-12-25T20:06:21.722 INFO:teuthology.orchestra.run.smithi050.stderr:Running command: systemctl daemon-reload 2023-12-25T20:06:21.812 INFO:teuthology.orchestra.run.smithi050.stderr:Running command: systemctl stop ceph-02c8f25c-a361-11ee-95a5-87774f69a715@mon.smithi050 2023-12-25T20:06:21.838 INFO:teuthology.orchestra.run.smithi050.stderr:Running command: systemctl reset-failed ceph-02c8f25c-a361-11ee-95a5-87774f69a715@mon.smithi050 2023-12-25T20:06:21.844 INFO:teuthology.orchestra.run.smithi050.stderr:systemctl: Failed to reset failed state of unit ceph-02c8f25c-a361-11ee-95a5-87774f69a715@mon.smithi050.service: Unit ceph-02c8f25c-a361-11ee-95a5-87774f69a715@mon.smithi050.service not loaded. 2023-12-25T20:06:21.844 INFO:teuthology.orchestra.run.smithi050.stderr:Running command: systemctl enable ceph-02c8f25c-a361-11ee-95a5-87774f69a715@mon.smithi050 2023-12-25T20:06:21.851 INFO:teuthology.orchestra.run.smithi050.stderr:systemctl: Created symlink /etc/systemd/system/ceph-02c8f25c-a361-11ee-95a5-87774f69a715.target.wants/ceph-02c8f25c-a361-11ee-95a5-87774f69a715@mon.smithi050.service → /etc/systemd/system/ceph-02c8f25c-a361-11ee-95a5-87774f69a715@.service. 2023-12-25T20:06:21.944 INFO:teuthology.orchestra.run.smithi050.stderr:Running command: systemctl start ceph-02c8f25c-a361-11ee-95a5-87774f69a715@mon.smithi050 2023-12-25T20:06:22.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:21 smithi050 systemd[1]: Starting Ceph mon.smithi050 for 02c8f25c-a361-11ee-95a5-87774f69a715... 2023-12-25T20:06:22.271 INFO:teuthology.orchestra.run.smithi050.stderr:Running command: systemctl is-enabled firewalld.service 2023-12-25T20:06:22.279 INFO:teuthology.orchestra.run.smithi050.stderr:systemctl: disabled 2023-12-25T20:06:22.280 INFO:teuthology.orchestra.run.smithi050.stderr:Running command: systemctl is-active firewalld.service 2023-12-25T20:06:22.287 INFO:teuthology.orchestra.run.smithi050.stderr:systemctl: inactive 2023-12-25T20:06:22.288 INFO:teuthology.orchestra.run.smithi050.stderr:firewalld.service is not enabled 2023-12-25T20:06:22.288 INFO:teuthology.orchestra.run.smithi050.stderr:Not possible to enable service . firewalld.service is not available 2023-12-25T20:06:22.288 INFO:teuthology.orchestra.run.smithi050.stderr:Not possible to open ports <[]>. firewalld.service is not available 2023-12-25T20:06:22.289 INFO:teuthology.orchestra.run.smithi050.stderr:Waiting for mon to start... 2023-12-25T20:06:22.289 INFO:teuthology.orchestra.run.smithi050.stderr:Waiting for mon... 2023-12-25T20:06:22.289 INFO:teuthology.orchestra.run.smithi050.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi050 -e CEPH_USE_RANDOM_NONCE=1 -v /var/lib/ceph/02c8f25c-a361-11ee-95a5-87774f69a715/mon.smithi050:/var/lib/ceph/mon/ceph-smithi050:z -v /tmp/ceph-tmps7kf4jlu:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpu2g3lwcx:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status 2023-12-25T20:06:22.525 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:22 smithi050 bash[111347]: 23e09f0e7d2eb2e874b9c5f1b3167c6f2abfee10f320714f974b767275026c6e 2023-12-25T20:06:22.525 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:22 smithi050 systemd[1]: Started Ceph mon.smithi050 for 02c8f25c-a361-11ee-95a5-87774f69a715. 2023-12-25T20:06:22.525 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:22 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111361]: cluster 2023-12-25T20:06:22.287293+0000 mon.smithi050 (mon.0) 0 : [INF] mkfs 02c8f25c-a361-11ee-95a5-87774f69a715 2023-12-25T20:06:22.525 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:22 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111361]: cluster 2023-12-25T20:06:22.271877+0000 mon.smithi050 (mon.0) 1 : cluster [INF] mon.smithi050 is new leader, mons smithi050 in quorum (ranks 0) 2023-12-25T20:06:22.719 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:22.716+0000 7fd67ddce700 1 Processor -- start 2023-12-25T20:06:22.719 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:22.716+0000 7fd67ddce700 1 -- start start 2023-12-25T20:06:22.719 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:22.716+0000 7fd67ddce700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd678070430 0x7fd6780fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-25T20:06:22.719 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:22.716+0000 7fd67ddce700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6780fedd0 con 0x7fd678070430 2023-12-25T20:06:22.719 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:22.717+0000 7fd6777fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd678070430 0x7fd6780fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:06:22.719 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:22.717+0000 7fd6777fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd678070430 0x7fd6780fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36094/0 (socket says 172.21.15.50:36094) 2023-12-25T20:06:22.719 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:22.717+0000 7fd6777fe700 1 -- 172.21.15.50:0/305250430 learned_addr learned my addr 172.21.15.50:0/305250430 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-25T20:06:22.719 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:22.717+0000 7fd6777fe700 1 -- 172.21.15.50:0/305250430 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6780fef10 con 0x7fd678070430 2023-12-25T20:06:22.720 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:22.717+0000 7fd6777fe700 1 --2- 172.21.15.50:0/305250430 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd678070430 0x7fd6780fe8e0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7fd668009960 tx=0x7fd6680092d0).ready entity=mon.0 client_cookie=1cde27be068edb3 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:06:22.720 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:22.717+0000 7fd6767fc700 1 -- 172.21.15.50:0/305250430 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd66800b9c0 con 0x7fd678070430 2023-12-25T20:06:22.720 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:22.717+0000 7fd6767fc700 1 -- 172.21.15.50:0/305250430 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd66800bb20 con 0x7fd678070430 2023-12-25T20:06:22.720 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:22.717+0000 7fd6767fc700 1 -- 172.21.15.50:0/305250430 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd66800bdf0 con 0x7fd678070430 2023-12-25T20:06:22.720 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:22.718+0000 7fd67ddce700 1 -- 172.21.15.50:0/305250430 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd678070430 msgr2=0x7fd6780fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:06:22.720 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:22.718+0000 7fd67ddce700 1 --2- 172.21.15.50:0/305250430 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd678070430 0x7fd6780fe8e0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7fd668009960 tx=0x7fd6680092d0).stop 2023-12-25T20:06:22.720 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:22.718+0000 7fd67ddce700 1 -- 172.21.15.50:0/305250430 shutdown_connections 2023-12-25T20:06:22.720 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:22.718+0000 7fd67ddce700 1 --2- 172.21.15.50:0/305250430 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd678070430 0x7fd6780fe8e0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:06:22.720 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:22.718+0000 7fd67ddce700 1 -- 172.21.15.50:0/305250430 >> 172.21.15.50:0/305250430 conn(0x7fd6780f63c0 msgr2=0x7fd6780f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:06:22.722 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:22.718+0000 7fd67ddce700 1 -- 172.21.15.50:0/305250430 shutdown_connections 2023-12-25T20:06:22.722 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:22.718+0000 7fd67ddce700 1 -- 172.21.15.50:0/305250430 wait complete. 2023-12-25T20:06:22.723 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:22.719+0000 7fd67ddce700 1 Processor -- start 2023-12-25T20:06:22.723 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:22.719+0000 7fd67ddce700 1 -- start start 2023-12-25T20:06:22.723 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:22.719+0000 7fd67ddce700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd678070430 0x7fd6780fc360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:06:22.723 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:22.719+0000 7fd67ddce700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6780fa7e0 con 0x7fd678070430 2023-12-25T20:06:22.723 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:22.719+0000 7fd6777fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd678070430 0x7fd6780fc360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:06:22.723 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:22.719+0000 7fd6777fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd678070430 0x7fd6780fc360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36110/0 (socket says 172.21.15.50:36110) 2023-12-25T20:06:22.723 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:22.719+0000 7fd6777fe700 1 -- 172.21.15.50:0/2097426890 learned_addr learned my addr 172.21.15.50:0/2097426890 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-25T20:06:22.723 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:22.719+0000 7fd6777fe700 1 -- 172.21.15.50:0/2097426890 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd668009570 con 0x7fd678070430 2023-12-25T20:06:22.723 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:22.719+0000 7fd6777fe700 1 --2- 172.21.15.50:0/2097426890 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd678070430 0x7fd6780fc360 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7fd668009f20 tx=0x7fd668017000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:06:22.723 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:22.720+0000 7fd66ffff700 1 -- 172.21.15.50:0/2097426890 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd66801c420 con 0x7fd678070430 2023-12-25T20:06:22.723 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:22.720+0000 7fd66ffff700 1 -- 172.21.15.50:0/2097426890 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd66801c580 con 0x7fd678070430 2023-12-25T20:06:22.724 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:22.720+0000 7fd67ddce700 1 -- 172.21.15.50:0/2097426890 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6780fa980 con 0x7fd678070430 2023-12-25T20:06:22.724 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:22.720+0000 7fd66ffff700 1 -- 172.21.15.50:0/2097426890 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd66801c850 con 0x7fd678070430 2023-12-25T20:06:22.724 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:22.720+0000 7fd67ddce700 1 -- 172.21.15.50:0/2097426890 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6780fad90 con 0x7fd678070430 2023-12-25T20:06:22.724 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:22.720+0000 7fd66ffff700 1 -- 172.21.15.50:0/2097426890 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7fd66802c070 con 0x7fd678070430 2023-12-25T20:06:22.724 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:22.720+0000 7fd66ffff700 1 -- 172.21.15.50:0/2097426890 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fd668025620 con 0x7fd678070430 2023-12-25T20:06:22.724 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:22.721+0000 7fd67ddce700 1 -- 172.21.15.50:0/2097426890 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd678045ad0 con 0x7fd678070430 2023-12-25T20:06:22.724 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:22.723+0000 7fd66ffff700 1 -- 172.21.15.50:0/2097426890 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7fd66802a060 con 0x7fd678070430 2023-12-25T20:06:22.796 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:22.793+0000 7fd67ddce700 1 -- 172.21.15.50:0/2097426890 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7fd678103e60 con 0x7fd678070430 2023-12-25T20:06:22.796 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:22.793+0000 7fd66ffff700 1 -- 172.21.15.50:0/2097426890 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+325 (secure 0 0 0) 0x7fd668020070 con 0x7fd678070430 2023-12-25T20:06:22.796 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: cluster: 2023-12-25T20:06:22.796 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: id: 02c8f25c-a361-11ee-95a5-87774f69a715 2023-12-25T20:06:22.796 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: health: HEALTH_OK 2023-12-25T20:06:22.796 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:22.796 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: services: 2023-12-25T20:06:22.797 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: mon: 1 daemons, quorum smithi050 (age 0.507946s) 2023-12-25T20:06:22.797 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: mgr: no daemons active 2023-12-25T20:06:22.797 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2023-12-25T20:06:22.797 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:22.797 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: data: 2023-12-25T20:06:22.797 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2023-12-25T20:06:22.797 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2023-12-25T20:06:22.797 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2023-12-25T20:06:22.797 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: pgs: 2023-12-25T20:06:22.797 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:22.797 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:22.794+0000 7fd67ddce700 1 -- 172.21.15.50:0/2097426890 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd678070430 msgr2=0x7fd6780fc360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:06:22.797 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:22.794+0000 7fd67ddce700 1 --2- 172.21.15.50:0/2097426890 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd678070430 0x7fd6780fc360 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7fd668009f20 tx=0x7fd668017000).stop 2023-12-25T20:06:22.797 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:22.794+0000 7fd67ddce700 1 -- 172.21.15.50:0/2097426890 shutdown_connections 2023-12-25T20:06:22.797 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:22.794+0000 7fd67ddce700 1 --2- 172.21.15.50:0/2097426890 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd678070430 0x7fd6780fc360 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:06:22.797 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:22.794+0000 7fd67ddce700 1 -- 172.21.15.50:0/2097426890 >> 172.21.15.50:0/2097426890 conn(0x7fd6780f63c0 msgr2=0x7fd6780f8070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:06:22.798 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:22.794+0000 7fd67ddce700 1 -- 172.21.15.50:0/2097426890 shutdown_connections 2023-12-25T20:06:22.798 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:22.794+0000 7fd67ddce700 1 -- 172.21.15.50:0/2097426890 wait complete. 2023-12-25T20:06:23.143 INFO:teuthology.orchestra.run.smithi050.stderr:mon is available 2023-12-25T20:06:23.144 INFO:teuthology.orchestra.run.smithi050.stderr:Assimilating anything we can from ceph.conf... 2023-12-25T20:06:23.144 INFO:teuthology.orchestra.run.smithi050.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi050 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/02c8f25c-a361-11ee-95a5-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmps7kf4jlu:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpu2g3lwcx:/etc/ceph/ceph.conf:z -v /var/lib/ceph/02c8f25c-a361-11ee-95a5-87774f69a715/mon.smithi050:/var/lib/ceph/mon/ceph-smithi050:z docker.io/ceph/ceph:v16.2.4 config assimilate-conf -i /var/lib/ceph/mon/ceph-smithi050/config 2023-12-25T20:06:23.468 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:23 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111361]: cluster 2023-12-25T20:06:22.286661+0000 mon.smithi050 (mon.0) 2 : cluster [INF] mon.smithi050 is new leader, mons smithi050 in quorum (ranks 0) 2023-12-25T20:06:23.469 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:23 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111361]: cluster 2023-12-25T20:06:22.286867+0000 mon.smithi050 (mon.0) 3 : cluster [DBG] monmap e1: 1 mons at {smithi050=[v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0]} 2023-12-25T20:06:23.469 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:23 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111361]: cluster 2023-12-25T20:06:22.288839+0000 mon.smithi050 (mon.0) 4 : cluster [DBG] fsmap 2023-12-25T20:06:23.469 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:23 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111361]: cluster 2023-12-25T20:06:22.290835+0000 mon.smithi050 (mon.0) 5 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2023-12-25T20:06:23.469 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:23 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111361]: cluster 2023-12-25T20:06:22.291320+0000 mon.smithi050 (mon.0) 6 : cluster [DBG] mgrmap e1: no daemons active 2023-12-25T20:06:23.469 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:23 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111361]: audit 2023-12-25T20:06:22.794571+0000 mon.smithi050 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.50:0/2097426890' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2023-12-25T20:06:23.523 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:23.522+0000 7f98a551c700 1 Processor -- start 2023-12-25T20:06:23.526 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:23.522+0000 7f98a551c700 1 -- start start 2023-12-25T20:06:23.526 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:23.523+0000 7f98a551c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f98a00fab80 0x7f98a00fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-25T20:06:23.526 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:23.523+0000 7f98a551c700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98a00fb490 con 0x7f98a00fab80 2023-12-25T20:06:23.526 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:23.523+0000 7f989effd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f98a00fab80 0x7f98a00fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:06:23.526 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:23.523+0000 7f989effd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f98a00fab80 0x7f98a00fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36126/0 (socket says 172.21.15.50:36126) 2023-12-25T20:06:23.527 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:23.523+0000 7f989effd700 1 -- 172.21.15.50:0/4130106999 learned_addr learned my addr 172.21.15.50:0/4130106999 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-25T20:06:23.527 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:23.523+0000 7f989effd700 1 -- 172.21.15.50:0/4130106999 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98a00fb5d0 con 0x7f98a00fab80 2023-12-25T20:06:23.527 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:23.523+0000 7f989effd700 1 --2- 172.21.15.50:0/4130106999 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f98a00fab80 0x7f98a00fafa0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7f989400b870 tx=0x7f9894006f70).ready entity=mon.0 client_cookie=6e39e27f7ba470aa server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:06:23.527 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:23.524+0000 7f989dffb700 1 -- 172.21.15.50:0/4130106999 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f989400da20 con 0x7f98a00fab80 2023-12-25T20:06:23.527 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:23.524+0000 7f989dffb700 1 -- 172.21.15.50:0/4130106999 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f989400db80 con 0x7f98a00fab80 2023-12-25T20:06:23.527 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:23.524+0000 7f989dffb700 1 -- 172.21.15.50:0/4130106999 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f989400de50 con 0x7f98a00fab80 2023-12-25T20:06:23.527 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:23.524+0000 7f98a551c700 1 -- 172.21.15.50:0/4130106999 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f98a00fab80 msgr2=0x7f98a00fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:06:23.527 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:23.524+0000 7f98a551c700 1 --2- 172.21.15.50:0/4130106999 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f98a00fab80 0x7f98a00fafa0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7f989400b870 tx=0x7f9894006f70).stop 2023-12-25T20:06:23.527 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:23.524+0000 7f98a551c700 1 -- 172.21.15.50:0/4130106999 shutdown_connections 2023-12-25T20:06:23.527 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:23.524+0000 7f98a551c700 1 --2- 172.21.15.50:0/4130106999 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f98a00fab80 0x7f98a00fafa0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:06:23.527 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:23.524+0000 7f98a551c700 1 -- 172.21.15.50:0/4130106999 >> 172.21.15.50:0/4130106999 conn(0x7f98a00f63c0 msgr2=0x7f98a00f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:06:23.527 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:23.524+0000 7f98a551c700 1 -- 172.21.15.50:0/4130106999 shutdown_connections 2023-12-25T20:06:23.528 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:23.524+0000 7f98a551c700 1 -- 172.21.15.50:0/4130106999 wait complete. 2023-12-25T20:06:23.528 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:23.525+0000 7f98a551c700 1 Processor -- start 2023-12-25T20:06:23.528 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:23.525+0000 7f98a551c700 1 -- start start 2023-12-25T20:06:23.528 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:23.525+0000 7f98a551c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f98a00fab80 0x7f98a010c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:06:23.528 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:23.525+0000 7f98a551c700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98a010c940 con 0x7f98a00fab80 2023-12-25T20:06:23.528 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:23.525+0000 7f989effd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f98a00fab80 0x7f98a010c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:06:23.528 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:23.525+0000 7f989effd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f98a00fab80 0x7f98a010c450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36136/0 (socket says 172.21.15.50:36136) 2023-12-25T20:06:23.528 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:23.525+0000 7f989effd700 1 -- 172.21.15.50:0/741476600 learned_addr learned my addr 172.21.15.50:0/741476600 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-25T20:06:23.529 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:23.526+0000 7f989effd700 1 -- 172.21.15.50:0/741476600 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f989400b480 con 0x7f98a00fab80 2023-12-25T20:06:23.529 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:23.526+0000 7f989effd700 1 --2- 172.21.15.50:0/741476600 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f98a00fab80 0x7f98a010c450 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7f9894006c70 tx=0x7f9894006f70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:06:23.529 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:23.526+0000 7f98877fe700 1 -- 172.21.15.50:0/741476600 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9894019450 con 0x7f98a00fab80 2023-12-25T20:06:23.530 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:23.526+0000 7f98877fe700 1 -- 172.21.15.50:0/741476600 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98940195b0 con 0x7f98a00fab80 2023-12-25T20:06:23.530 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:23.526+0000 7f98877fe700 1 -- 172.21.15.50:0/741476600 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9894019880 con 0x7f98a00fab80 2023-12-25T20:06:23.530 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:23.526+0000 7f98a551c700 1 -- 172.21.15.50:0/741476600 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98a010cb40 con 0x7f98a00fab80 2023-12-25T20:06:23.530 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:23.526+0000 7f98a551c700 1 -- 172.21.15.50:0/741476600 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98a010cf50 con 0x7f98a00fab80 2023-12-25T20:06:23.530 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:23.526+0000 7f98877fe700 1 -- 172.21.15.50:0/741476600 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f989402a070 con 0x7f98a00fab80 2023-12-25T20:06:23.530 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:23.526+0000 7f98877fe700 1 -- 172.21.15.50:0/741476600 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f98940239b0 con 0x7f98a00fab80 2023-12-25T20:06:23.530 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:23.527+0000 7f98a551c700 1 -- 172.21.15.50:0/741476600 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98a0045ad0 con 0x7f98a00fab80 2023-12-25T20:06:23.530 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:23.530+0000 7f98877fe700 1 -- 172.21.15.50:0/741476600 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f98940429c0 con 0x7f98a00fab80 2023-12-25T20:06:23.597 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:23.597+0000 7f98a551c700 1 -- 172.21.15.50:0/741476600 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f98a00fbec0 con 0x7f98a00fab80 2023-12-25T20:06:23.606 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:23.600+0000 7f98877fe700 1 -- 172.21.15.50:0/741476600 <== mon.0 v2:172.21.15.50:3300/0 7 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f989400f840 con 0x7f98a00fab80 2023-12-25T20:06:23.607 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:23.600+0000 7f98877fe700 1 -- 172.21.15.50:0/741476600 <== mon.0 v2:172.21.15.50:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+464 (secure 0 0 0) 0x7f9894031030 con 0x7f98a00fab80 2023-12-25T20:06:23.607 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:23.607 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: [global] 2023-12-25T20:06:23.607 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: fsid = 02c8f25c-a361-11ee-95a5-87774f69a715 2023-12-25T20:06:23.607 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.50:3300,v1:172.21.15.50:6789] 2023-12-25T20:06:23.607 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2023-12-25T20:06:23.607 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2023-12-25T20:06:23.607 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2023-12-25T20:06:23.607 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2023-12-25T20:06:23.607 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:23.607 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: [mgr] 2023-12-25T20:06:23.607 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2023-12-25T20:06:23.608 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:23.608 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: [osd] 2023-12-25T20:06:23.608 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: mon_osd_backfillfull_ratio = 0.85 2023-12-25T20:06:23.608 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: mon_osd_full_ratio = 0.9 2023-12-25T20:06:23.608 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: mon_osd_nearfull_ratio = 0.8 2023-12-25T20:06:23.608 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2023-12-25T20:06:23.608 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: osd_memory_target_autotune = true 2023-12-25T20:06:23.608 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2023-12-25T20:06:23.608 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:23.601+0000 7f98a551c700 1 -- 172.21.15.50:0/741476600 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f98a00fab80 msgr2=0x7f98a010c450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:06:23.608 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:23.601+0000 7f98a551c700 1 --2- 172.21.15.50:0/741476600 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f98a00fab80 0x7f98a010c450 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7f9894006c70 tx=0x7f9894006f70).stop 2023-12-25T20:06:23.608 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:23.601+0000 7f98a551c700 1 -- 172.21.15.50:0/741476600 shutdown_connections 2023-12-25T20:06:23.608 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:23.601+0000 7f98a551c700 1 --2- 172.21.15.50:0/741476600 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f98a00fab80 0x7f98a010c450 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:06:23.608 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:23.601+0000 7f98a551c700 1 -- 172.21.15.50:0/741476600 >> 172.21.15.50:0/741476600 conn(0x7f98a00f63c0 msgr2=0x7f98a00f85c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:06:23.608 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:23.601+0000 7f98a551c700 1 -- 172.21.15.50:0/741476600 shutdown_connections 2023-12-25T20:06:23.609 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:23.601+0000 7f98a551c700 1 -- 172.21.15.50:0/741476600 wait complete. 2023-12-25T20:06:23.952 INFO:teuthology.orchestra.run.smithi050.stderr:Generating new minimal ceph.conf... 2023-12-25T20:06:23.953 INFO:teuthology.orchestra.run.smithi050.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi050 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/02c8f25c-a361-11ee-95a5-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmps7kf4jlu:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpu2g3lwcx:/etc/ceph/ceph.conf:z -v /var/lib/ceph/02c8f25c-a361-11ee-95a5-87774f69a715/mon.smithi050:/var/lib/ceph/mon/ceph-smithi050:z docker.io/ceph/ceph:v16.2.4 config generate-minimal-conf -o /var/lib/ceph/mon/ceph-smithi050/config 2023-12-25T20:06:24.350 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:24.347+0000 7f6090d10700 1 Processor -- start 2023-12-25T20:06:24.350 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:24.347+0000 7f6090d10700 1 -- start start 2023-12-25T20:06:24.350 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:24.348+0000 7f6090d10700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f608c070430 0x7f608c0fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-25T20:06:24.350 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:24.348+0000 7f6090d10700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f608c0fedd0 con 0x7f608c070430 2023-12-25T20:06:24.350 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:24.348+0000 7f608a59c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f608c070430 0x7f608c0fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:06:24.350 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:24.348+0000 7f608a59c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f608c070430 0x7f608c0fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36138/0 (socket says 172.21.15.50:36138) 2023-12-25T20:06:24.350 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:24.348+0000 7f608a59c700 1 -- 172.21.15.50:0/1235060164 learned_addr learned my addr 172.21.15.50:0/1235060164 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-25T20:06:24.350 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:24.348+0000 7f608a59c700 1 -- 172.21.15.50:0/1235060164 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f608c0fef10 con 0x7f608c070430 2023-12-25T20:06:24.350 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:24.348+0000 7f608a59c700 1 --2- 172.21.15.50:0/1235060164 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f608c070430 0x7f608c0fe8e0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f607400b5f0 tx=0x7f6074006ee0).ready entity=mon.0 client_cookie=f24b62bc1d548903 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:06:24.350 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:24.348+0000 7f608959a700 1 -- 172.21.15.50:0/1235060164 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f607400d600 con 0x7f608c070430 2023-12-25T20:06:24.350 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:24.349+0000 7f608959a700 1 -- 172.21.15.50:0/1235060164 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f607400dbd0 con 0x7f608c070430 2023-12-25T20:06:24.350 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:24.349+0000 7f608959a700 1 -- 172.21.15.50:0/1235060164 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6074011c00 con 0x7f608c070430 2023-12-25T20:06:24.353 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:24.349+0000 7f6090d10700 1 -- 172.21.15.50:0/1235060164 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f608c070430 msgr2=0x7f608c0fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:06:24.353 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:24.349+0000 7f6090d10700 1 --2- 172.21.15.50:0/1235060164 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f608c070430 0x7f608c0fe8e0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f607400b5f0 tx=0x7f6074006ee0).stop 2023-12-25T20:06:24.353 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:24.349+0000 7f6090d10700 1 -- 172.21.15.50:0/1235060164 shutdown_connections 2023-12-25T20:06:24.353 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:24.349+0000 7f6090d10700 1 --2- 172.21.15.50:0/1235060164 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f608c070430 0x7f608c0fe8e0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:06:24.353 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:24.349+0000 7f6090d10700 1 -- 172.21.15.50:0/1235060164 >> 172.21.15.50:0/1235060164 conn(0x7f608c0f63c0 msgr2=0x7f608c0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:06:24.353 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:24.349+0000 7f6090d10700 1 -- 172.21.15.50:0/1235060164 shutdown_connections 2023-12-25T20:06:24.353 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:24.349+0000 7f6090d10700 1 -- 172.21.15.50:0/1235060164 wait complete. 2023-12-25T20:06:24.353 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:24.350+0000 7f6090d10700 1 Processor -- start 2023-12-25T20:06:24.353 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:24.350+0000 7f6090d10700 1 -- start start 2023-12-25T20:06:24.353 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:24.350+0000 7f6090d10700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f608c070430 0x7f608c0fc360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:06:24.354 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:24.350+0000 7f6090d10700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f608c0fa7e0 con 0x7f608c070430 2023-12-25T20:06:24.354 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:24.351+0000 7f608a59c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f608c070430 0x7f608c0fc360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:06:24.354 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:24.351+0000 7f608a59c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f608c070430 0x7f608c0fc360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36150/0 (socket says 172.21.15.50:36150) 2023-12-25T20:06:24.354 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:24.351+0000 7f608a59c700 1 -- 172.21.15.50:0/3898521940 learned_addr learned my addr 172.21.15.50:0/3898521940 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-25T20:06:24.354 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:24.351+0000 7f608a59c700 1 -- 172.21.15.50:0/3898521940 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f607400b200 con 0x7f608c070430 2023-12-25T20:06:24.354 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:24.351+0000 7f608a59c700 1 --2- 172.21.15.50:0/3898521940 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f608c070430 0x7f608c0fc360 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f607400e040 tx=0x7f6074006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:06:24.354 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:24.351+0000 7f6082ffd700 1 -- 172.21.15.50:0/3898521940 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f607400d9f0 con 0x7f608c070430 2023-12-25T20:06:24.355 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:24.351+0000 7f6082ffd700 1 -- 172.21.15.50:0/3898521940 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f607401a9f0 con 0x7f608c070430 2023-12-25T20:06:24.355 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:24.351+0000 7f6090d10700 1 -- 172.21.15.50:0/3898521940 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f608c0fa980 con 0x7f608c070430 2023-12-25T20:06:24.355 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:24.351+0000 7f6082ffd700 1 -- 172.21.15.50:0/3898521940 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f607401b8c0 con 0x7f608c070430 2023-12-25T20:06:24.355 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:24.351+0000 7f6090d10700 1 -- 172.21.15.50:0/3898521940 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f608c0fad90 con 0x7f608c070430 2023-12-25T20:06:24.355 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:24.351+0000 7f6082ffd700 1 -- 172.21.15.50:0/3898521940 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f607400f070 con 0x7f608c070430 2023-12-25T20:06:24.355 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:24.351+0000 7f6082ffd700 1 -- 172.21.15.50:0/3898521940 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f6074025ae0 con 0x7f608c070430 2023-12-25T20:06:24.356 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:24.352+0000 7f6090d10700 1 -- 172.21.15.50:0/3898521940 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f608c045ad0 con 0x7f608c070430 2023-12-25T20:06:24.358 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:24.355+0000 7f6082ffd700 1 -- 172.21.15.50:0/3898521940 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f6074016070 con 0x7f608c070430 2023-12-25T20:06:24.423 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:24.422+0000 7f6090d10700 1 -- 172.21.15.50:0/3898521940 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f608c104820 con 0x7f608c070430 2023-12-25T20:06:24.423 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:24.423+0000 7f6082ffd700 1 -- 172.21.15.50:0/3898521940 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7f607401bd70 con 0x7f608c070430 2023-12-25T20:06:24.425 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:24.424+0000 7f6090d10700 1 -- 172.21.15.50:0/3898521940 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f608c070430 msgr2=0x7f608c0fc360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:06:24.425 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:24.424+0000 7f6090d10700 1 --2- 172.21.15.50:0/3898521940 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f608c070430 0x7f608c0fc360 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f607400e040 tx=0x7f6074006ee0).stop 2023-12-25T20:06:24.425 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:24.424+0000 7f6090d10700 1 -- 172.21.15.50:0/3898521940 shutdown_connections 2023-12-25T20:06:24.425 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:24.424+0000 7f6090d10700 1 --2- 172.21.15.50:0/3898521940 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f608c070430 0x7f608c0fc360 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:06:24.425 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:24.424+0000 7f6090d10700 1 -- 172.21.15.50:0/3898521940 >> 172.21.15.50:0/3898521940 conn(0x7f608c0f63c0 msgr2=0x7f608c0f7f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:06:24.425 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:24.424+0000 7f6090d10700 1 -- 172.21.15.50:0/3898521940 shutdown_connections 2023-12-25T20:06:24.425 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:24.424+0000 7f6090d10700 1 -- 172.21.15.50:0/3898521940 wait complete. 2023-12-25T20:06:24.637 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:24 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111361]: audit 2023-12-25T20:06:23.598420+0000 mon.smithi050 (mon.0) 8 : audit [INF] from='client.? 172.21.15.50:0/741476600' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-12-25T20:06:24.637 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:24 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111361]: audit 2023-12-25T20:06:23.601352+0000 mon.smithi050 (mon.0) 9 : audit [INF] from='client.? 172.21.15.50:0/741476600' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2023-12-25T20:06:24.819 INFO:teuthology.orchestra.run.smithi050.stderr:Restarting the monitor... 2023-12-25T20:06:24.819 INFO:teuthology.orchestra.run.smithi050.stderr:Running command: systemctl restart ceph-02c8f25c-a361-11ee-95a5-87774f69a715@mon.smithi050 2023-12-25T20:06:24.897 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:24 smithi050 systemd[1]: Stopping Ceph mon.smithi050 for 02c8f25c-a361-11ee-95a5-87774f69a715... 2023-12-25T20:06:25.184 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:24 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111361]: debug 2023-12-25T20:06:24.896+0000 7ff6fe5ba700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi050 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2023-12-25T20:06:25.184 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:24 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111361]: debug 2023-12-25T20:06:24.896+0000 7ff6fe5ba700 -1 mon.smithi050@0(leader) e1 *** Got Signal Terminated *** 2023-12-25T20:06:25.468 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 podman[111682]: ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050 2023-12-25T20:06:25.469 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 systemd[1]: ceph-02c8f25c-a361-11ee-95a5-87774f69a715@mon.smithi050.service: Succeeded. 2023-12-25T20:06:25.469 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 systemd[1]: Stopped Ceph mon.smithi050 for 02c8f25c-a361-11ee-95a5-87774f69a715. 2023-12-25T20:06:25.469 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 systemd[1]: Starting Ceph mon.smithi050 for 02c8f25c-a361-11ee-95a5-87774f69a715... 2023-12-25T20:06:25.728 INFO:teuthology.orchestra.run.smithi050.stderr:Setting mon public_network to 172.21.0.0/20 2023-12-25T20:06:25.728 INFO:teuthology.orchestra.run.smithi050.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi050 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/02c8f25c-a361-11ee-95a5-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmps7kf4jlu:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpu2g3lwcx:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mon public_network 172.21.0.0/20 2023-12-25T20:06:26.099 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:26.097+0000 7fd280abb700 1 Processor -- start 2023-12-25T20:06:26.100 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:26.097+0000 7fd280abb700 1 -- start start 2023-12-25T20:06:26.100 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:26.098+0000 7fd280abb700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd27c0703f0 0x7fd27c100d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-25T20:06:26.100 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:26.098+0000 7fd280abb700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd27c0fc710 con 0x7fd27c0703f0 2023-12-25T20:06:26.100 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:26.098+0000 7fd27a59c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd27c0703f0 0x7fd27c100d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:06:26.100 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:26.098+0000 7fd27a59c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd27c0703f0 0x7fd27c100d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36158/0 (socket says 172.21.15.50:36158) 2023-12-25T20:06:26.100 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:26.098+0000 7fd27a59c700 1 -- 172.21.15.50:0/1328823128 learned_addr learned my addr 172.21.15.50:0/1328823128 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-25T20:06:26.101 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:26.098+0000 7fd27a59c700 1 -- 172.21.15.50:0/1328823128 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd27c101210 con 0x7fd27c0703f0 2023-12-25T20:06:26.101 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:26.099+0000 7fd27a59c700 1 --2- 172.21.15.50:0/1328823128 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd27c0703f0 0x7fd27c100d20 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7fd26c00b5f0 tx=0x7fd26c006ee0).ready entity=mon.0 client_cookie=8774f30d803080e4 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:06:26.101 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:26.099+0000 7fd27959a700 1 -- 172.21.15.50:0/1328823128 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd26c00d600 con 0x7fd27c0703f0 2023-12-25T20:06:26.102 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:26.099+0000 7fd27959a700 1 -- 172.21.15.50:0/1328823128 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fd26c00dbd0 con 0x7fd27c0703f0 2023-12-25T20:06:26.102 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:26.099+0000 7fd27959a700 1 -- 172.21.15.50:0/1328823128 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd26c011c00 con 0x7fd27c0703f0 2023-12-25T20:06:26.102 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:26.099+0000 7fd280abb700 1 -- 172.21.15.50:0/1328823128 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd27c0703f0 msgr2=0x7fd27c100d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:06:26.102 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:26.099+0000 7fd280abb700 1 --2- 172.21.15.50:0/1328823128 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd27c0703f0 0x7fd27c100d20 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7fd26c00b5f0 tx=0x7fd26c006ee0).stop 2023-12-25T20:06:26.102 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:26.099+0000 7fd280abb700 1 -- 172.21.15.50:0/1328823128 shutdown_connections 2023-12-25T20:06:26.102 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:26.099+0000 7fd280abb700 1 --2- 172.21.15.50:0/1328823128 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd27c0703f0 0x7fd27c100d20 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:06:26.102 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:26.099+0000 7fd280abb700 1 -- 172.21.15.50:0/1328823128 >> 172.21.15.50:0/1328823128 conn(0x7fd27c0f63c0 msgr2=0x7fd27c0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:06:26.102 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:26.099+0000 7fd280abb700 1 -- 172.21.15.50:0/1328823128 shutdown_connections 2023-12-25T20:06:26.103 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:26.099+0000 7fd280abb700 1 -- 172.21.15.50:0/1328823128 wait complete. 2023-12-25T20:06:26.103 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:26.100+0000 7fd280abb700 1 Processor -- start 2023-12-25T20:06:26.103 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:26.100+0000 7fd280abb700 1 -- start start 2023-12-25T20:06:26.103 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:26.101+0000 7fd280abb700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd27c0703f0 0x7fd27c10ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:06:26.103 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:26.101+0000 7fd280abb700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd27c10d1c0 con 0x7fd27c0703f0 2023-12-25T20:06:26.103 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:26.101+0000 7fd27a59c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd27c0703f0 0x7fd27c10ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:06:26.103 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:26.101+0000 7fd27a59c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd27c0703f0 0x7fd27c10ccd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36166/0 (socket says 172.21.15.50:36166) 2023-12-25T20:06:26.103 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:26.101+0000 7fd27a59c700 1 -- 172.21.15.50:0/4251166667 learned_addr learned my addr 172.21.15.50:0/4251166667 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-25T20:06:26.103 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:26.101+0000 7fd27a59c700 1 -- 172.21.15.50:0/4251166667 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd26c00b200 con 0x7fd27c0703f0 2023-12-25T20:06:26.104 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:26.101+0000 7fd27a59c700 1 --2- 172.21.15.50:0/4251166667 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd27c0703f0 0x7fd27c10ccd0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7fd26c00e040 tx=0x7fd26c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:06:26.104 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:26.101+0000 7fd272ffd700 1 -- 172.21.15.50:0/4251166667 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd26c00d9f0 con 0x7fd27c0703f0 2023-12-25T20:06:26.104 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:26.101+0000 7fd272ffd700 1 -- 172.21.15.50:0/4251166667 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fd26c01a9f0 con 0x7fd27c0703f0 2023-12-25T20:06:26.104 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:26.101+0000 7fd280abb700 1 -- 172.21.15.50:0/4251166667 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd27c10d3c0 con 0x7fd27c0703f0 2023-12-25T20:06:26.104 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:26.101+0000 7fd272ffd700 1 -- 172.21.15.50:0/4251166667 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd26c01b8c0 con 0x7fd27c0703f0 2023-12-25T20:06:26.104 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:26.101+0000 7fd280abb700 1 -- 172.21.15.50:0/4251166667 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd27c10d7d0 con 0x7fd27c0703f0 2023-12-25T20:06:26.104 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:26.101+0000 7fd272ffd700 1 -- 172.21.15.50:0/4251166667 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7fd26c00f070 con 0x7fd27c0703f0 2023-12-25T20:06:26.104 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:26.102+0000 7fd272ffd700 1 -- 172.21.15.50:0/4251166667 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fd26c025ae0 con 0x7fd27c0703f0 2023-12-25T20:06:26.104 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:26.102+0000 7fd280abb700 1 -- 172.21.15.50:0/4251166667 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd27c045ad0 con 0x7fd27c0703f0 2023-12-25T20:06:26.105 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:26.105+0000 7fd272ffd700 1 -- 172.21.15.50:0/4251166667 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7fd26c016070 con 0x7fd27c0703f0 2023-12-25T20:06:26.172 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:26.171+0000 7fd280abb700 1 -- 172.21.15.50:0/4251166667 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7fd27c10e6d0 con 0x7fd27c0703f0 2023-12-25T20:06:26.174 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:26.173+0000 7fd272ffd700 1 -- 172.21.15.50:0/4251166667 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7fd26c01bd70 con 0x7fd27c0703f0 2023-12-25T20:06:26.175 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:26.174+0000 7fd280abb700 1 -- 172.21.15.50:0/4251166667 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd27c0703f0 msgr2=0x7fd27c10ccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:06:26.175 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:26.174+0000 7fd280abb700 1 --2- 172.21.15.50:0/4251166667 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd27c0703f0 0x7fd27c10ccd0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7fd26c00e040 tx=0x7fd26c006ee0).stop 2023-12-25T20:06:26.175 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:26.174+0000 7fd280abb700 1 -- 172.21.15.50:0/4251166667 shutdown_connections 2023-12-25T20:06:26.175 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:26.174+0000 7fd280abb700 1 --2- 172.21.15.50:0/4251166667 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd27c0703f0 0x7fd27c10ccd0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:06:26.175 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:26.174+0000 7fd280abb700 1 -- 172.21.15.50:0/4251166667 >> 172.21.15.50:0/4251166667 conn(0x7fd27c0f63c0 msgr2=0x7fd27c0f7f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:06:26.176 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:26.174+0000 7fd280abb700 1 -- 172.21.15.50:0/4251166667 shutdown_connections 2023-12-25T20:06:26.176 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:26.174+0000 7fd280abb700 1 -- 172.21.15.50:0/4251166667 wait complete. 2023-12-25T20:06:26.203 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.719+0000 7f06f3069700 0 set uid:gid to 167:167 (ceph:ceph) 2023-12-25T20:06:26.203 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.719+0000 7f06f3069700 0 ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable), process ceph-mon, pid 7 2023-12-25T20:06:26.203 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.719+0000 7f06f3069700 0 pidfile_write: ignore empty --pid-file 2023-12-25T20:06:26.204 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 bash[111755]: a2b104cee965ff058bf945169fa59ff210eeb5640a3a00cb8cae1f1a11cd4af0 2023-12-25T20:06:26.204 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.723+0000 7f06f3069700 0 load: jerasure load: lrc load: isa 2023-12-25T20:06:26.204 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: RocksDB version: 6.8.1 2023-12-25T20:06:26.204 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: 2023-12-25T20:06:26.204 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2023-12-25T20:06:26.204 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Compile date May 13 2021 2023-12-25T20:06:26.204 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: DB SUMMARY 2023-12-25T20:06:26.204 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: 2023-12-25T20:06:26.204 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: CURRENT file: CURRENT 2023-12-25T20:06:26.204 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: 2023-12-25T20:06:26.204 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: IDENTITY file: IDENTITY 2023-12-25T20:06:26.204 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: 2023-12-25T20:06:26.204 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: MANIFEST file: MANIFEST-000005 size: 131 Bytes 2023-12-25T20:06:26.204 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: 2023-12-25T20:06:26.205 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi050/store.db dir, Total Num: 1, files: 000004.sst 2023-12-25T20:06:26.205 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: 2023-12-25T20:06:26.205 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi050/store.db: 000006.log size: 92912 ; 2023-12-25T20:06:26.205 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: 2023-12-25T20:06:26.205 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.error_if_exists: 0 2023-12-25T20:06:26.205 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.create_if_missing: 0 2023-12-25T20:06:26.205 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.paranoid_checks: 1 2023-12-25T20:06:26.205 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.env: 0x5637f9d161c0 2023-12-25T20:06:26.205 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.fs: Posix File System 2023-12-25T20:06:26.205 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.info_log: 0x5637fa805960 2023-12-25T20:06:26.205 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.max_file_opening_threads: 16 2023-12-25T20:06:26.205 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.statistics: (nil) 2023-12-25T20:06:26.205 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.use_fsync: 0 2023-12-25T20:06:26.205 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.max_log_file_size: 0 2023-12-25T20:06:26.206 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2023-12-25T20:06:26.206 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.log_file_time_to_roll: 0 2023-12-25T20:06:26.206 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.keep_log_file_num: 1000 2023-12-25T20:06:26.206 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.recycle_log_file_num: 0 2023-12-25T20:06:26.206 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.allow_fallocate: 1 2023-12-25T20:06:26.206 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.allow_mmap_reads: 0 2023-12-25T20:06:26.206 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.allow_mmap_writes: 0 2023-12-25T20:06:26.206 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.use_direct_reads: 0 2023-12-25T20:06:26.206 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-12-25T20:06:26.206 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.create_missing_column_families: 0 2023-12-25T20:06:26.206 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.db_log_dir: 2023-12-25T20:06:26.206 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi050/store.db 2023-12-25T20:06:26.206 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.table_cache_numshardbits: 6 2023-12-25T20:06:26.206 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.max_subcompactions: 1 2023-12-25T20:06:26.207 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.max_background_flushes: -1 2023-12-25T20:06:26.207 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.WAL_ttl_seconds: 0 2023-12-25T20:06:26.207 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.WAL_size_limit_MB: 0 2023-12-25T20:06:26.207 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-12-25T20:06:26.207 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2023-12-25T20:06:26.207 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.is_fd_close_on_exec: 1 2023-12-25T20:06:26.207 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.advise_random_on_open: 1 2023-12-25T20:06:26.207 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.db_write_buffer_size: 0 2023-12-25T20:06:26.207 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.write_buffer_manager: 0x5637fb47a030 2023-12-25T20:06:26.207 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2023-12-25T20:06:26.207 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2023-12-25T20:06:26.207 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2023-12-25T20:06:26.207 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.use_adaptive_mutex: 0 2023-12-25T20:06:26.207 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.rate_limiter: (nil) 2023-12-25T20:06:26.207 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-12-25T20:06:26.208 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.wal_recovery_mode: 2 2023-12-25T20:06:26.208 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.enable_thread_tracking: 0 2023-12-25T20:06:26.208 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.enable_pipelined_write: 0 2023-12-25T20:06:26.208 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.unordered_write: 0 2023-12-25T20:06:26.208 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2023-12-25T20:06:26.208 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-12-25T20:06:26.208 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2023-12-25T20:06:26.208 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2023-12-25T20:06:26.208 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.row_cache: None 2023-12-25T20:06:26.208 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.wal_filter: None 2023-12-25T20:06:26.208 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2023-12-25T20:06:26.208 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.allow_ingest_behind: 0 2023-12-25T20:06:26.208 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.preserve_deletes: 0 2023-12-25T20:06:26.208 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.two_write_queues: 0 2023-12-25T20:06:26.209 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.manual_wal_flush: 0 2023-12-25T20:06:26.209 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.atomic_flush: 0 2023-12-25T20:06:26.209 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-12-25T20:06:26.209 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.persist_stats_to_disk: 0 2023-12-25T20:06:26.209 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.write_dbid_to_manifest: 0 2023-12-25T20:06:26.209 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.log_readahead_size: 0 2023-12-25T20:06:26.209 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2023-12-25T20:06:26.209 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.max_background_jobs: 2 2023-12-25T20:06:26.209 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.max_background_compactions: -1 2023-12-25T20:06:26.209 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2023-12-25T20:06:26.209 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-12-25T20:06:26.209 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.delayed_write_rate : 16777216 2023-12-25T20:06:26.209 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.max_total_wal_size: 0 2023-12-25T20:06:26.209 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-12-25T20:06:26.209 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.stats_dump_period_sec: 600 2023-12-25T20:06:26.210 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.stats_persist_period_sec: 600 2023-12-25T20:06:26.210 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2023-12-25T20:06:26.210 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.max_open_files: -1 2023-12-25T20:06:26.210 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.bytes_per_sync: 0 2023-12-25T20:06:26.210 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.wal_bytes_per_sync: 0 2023-12-25T20:06:26.210 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.strict_bytes_per_sync: 0 2023-12-25T20:06:26.211 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.compaction_readahead_size: 0 2023-12-25T20:06:26.211 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Compression algorithms supported: 2023-12-25T20:06:26.211 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2023-12-25T20:06:26.211 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: kZSTD supported: 0 2023-12-25T20:06:26.211 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: kXpressCompression supported: 0 2023-12-25T20:06:26.211 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: kLZ4HCCompression supported: 1 2023-12-25T20:06:26.212 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: kLZ4Compression supported: 1 2023-12-25T20:06:26.212 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: kBZip2Compression supported: 0 2023-12-25T20:06:26.212 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: kZlibCompression supported: 1 2023-12-25T20:06:26.212 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: kSnappyCompression supported: 1 2023-12-25T20:06:26.212 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Fast CRC32 supported: Supported on x86 2023-12-25T20:06:26.212 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi050/store.db/MANIFEST-000005 2023-12-25T20:06:26.212 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: 2023-12-25T20:06:26.212 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2023-12-25T20:06:26.212 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: 2023-12-25T20:06:26.212 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-12-25T20:06:26.212 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.merge_operator: 2023-12-25T20:06:26.212 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.compaction_filter: None 2023-12-25T20:06:26.212 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.compaction_filter_factory: None 2023-12-25T20:06:26.212 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.memtable_factory: SkipListFactory 2023-12-25T20:06:26.213 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.table_factory: BlockBasedTable 2023-12-25T20:06:26.213 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5637fa71c128) 2023-12-25T20:06:26.213 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cache_index_and_filter_blocks: 1 2023-12-25T20:06:26.213 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cache_index_and_filter_blocks_with_high_priority: 0 2023-12-25T20:06:26.213 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: pin_l0_filter_and_index_blocks_in_cache: 0 2023-12-25T20:06:26.213 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: pin_top_level_index_and_filter: 1 2023-12-25T20:06:26.213 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: index_type: 0 2023-12-25T20:06:26.213 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: data_block_index_type: 0 2023-12-25T20:06:26.213 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: index_shortening: 1 2023-12-25T20:06:26.213 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: data_block_hash_table_util_ratio: 0.750000 2023-12-25T20:06:26.213 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: hash_index_allow_collision: 1 2023-12-25T20:06:26.213 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: checksum: 1 2023-12-25T20:06:26.213 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: no_block_cache: 0 2023-12-25T20:06:26.213 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: block_cache: 0x5637fa752f10 2023-12-25T20:06:26.214 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: block_cache_name: BinnedLRUCache 2023-12-25T20:06:26.214 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: block_cache_options: 2023-12-25T20:06:26.214 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: capacity : 536870912 2023-12-25T20:06:26.214 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: num_shard_bits : 4 2023-12-25T20:06:26.215 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: strict_capacity_limit : 0 2023-12-25T20:06:26.215 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: high_pri_pool_ratio: 0.000 2023-12-25T20:06:26.215 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: block_cache_compressed: (nil) 2023-12-25T20:06:26.215 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: persistent_cache: (nil) 2023-12-25T20:06:26.216 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: block_size: 4096 2023-12-25T20:06:26.216 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: block_size_deviation: 10 2023-12-25T20:06:26.216 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: block_restart_interval: 16 2023-12-25T20:06:26.216 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: index_block_restart_interval: 1 2023-12-25T20:06:26.216 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: metadata_block_size: 4096 2023-12-25T20:06:26.216 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: partition_filters: 0 2023-12-25T20:06:26.216 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: use_delta_encoding: 1 2023-12-25T20:06:26.216 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: filter_policy: rocksdb.BuiltinBloomFilter 2023-12-25T20:06:26.216 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: whole_key_filtering: 1 2023-12-25T20:06:26.216 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: verify_compression: 0 2023-12-25T20:06:26.216 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: read_amp_bytes_per_bit: 0 2023-12-25T20:06:26.216 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: format_version: 2 2023-12-25T20:06:26.216 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: enable_index_compression: 1 2023-12-25T20:06:26.216 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: block_align: 0 2023-12-25T20:06:26.217 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: 2023-12-25T20:06:26.217 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.write_buffer_size: 33554432 2023-12-25T20:06:26.217 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.max_write_buffer_number: 2 2023-12-25T20:06:26.217 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.compression: NoCompression 2023-12-25T20:06:26.217 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.bottommost_compression: Disabled 2023-12-25T20:06:26.217 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.prefix_extractor: nullptr 2023-12-25T20:06:26.217 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-12-25T20:06:26.217 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.num_levels: 7 2023-12-25T20:06:26.217 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-12-25T20:06:26.217 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-12-25T20:06:26.217 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-12-25T20:06:26.217 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-12-25T20:06:26.217 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2023-12-25T20:06:26.217 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-12-25T20:06:26.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-12-25T20:06:26.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-12-25T20:06:26.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2023-12-25T20:06:26.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.compression_opts.window_bits: -14 2023-12-25T20:06:26.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.compression_opts.level: 32767 2023-12-25T20:06:26.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.compression_opts.strategy: 0 2023-12-25T20:06:26.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-12-25T20:06:26.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-12-25T20:06:26.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.compression_opts.enabled: false 2023-12-25T20:06:26.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-12-25T20:06:26.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-12-25T20:06:26.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2023-12-25T20:06:26.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.target_file_size_base: 67108864 2023-12-25T20:06:26.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.target_file_size_multiplier: 1 2023-12-25T20:06:26.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2023-12-25T20:06:26.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-12-25T20:06:26.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-12-25T20:06:26.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-12-25T20:06:26.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-12-25T20:06:26.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-12-25T20:06:26.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-12-25T20:06:26.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-12-25T20:06:26.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-12-25T20:06:26.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-12-25T20:06:26.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-12-25T20:06:26.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2023-12-25T20:06:26.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.arena_block_size: 4194304 2023-12-25T20:06:26.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-12-25T20:06:26.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-12-25T20:06:26.220 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2023-12-25T20:06:26.220 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.disable_auto_compactions: 0 2023-12-25T20:06:26.220 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-12-25T20:06:26.220 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-12-25T20:06:26.220 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-12-25T20:06:26.220 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-12-25T20:06:26.220 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-12-25T20:06:26.220 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-12-25T20:06:26.220 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-12-25T20:06:26.220 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-12-25T20:06:26.220 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-12-25T20:06:26.220 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-12-25T20:06:26.220 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.table_properties_collectors: 2023-12-25T20:06:26.220 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.inplace_update_support: 0 2023-12-25T20:06:26.221 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.inplace_update_num_locks: 10000 2023-12-25T20:06:26.221 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-12-25T20:06:26.221 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2023-12-25T20:06:26.221 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.memtable_huge_page_size: 0 2023-12-25T20:06:26.221 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.bloom_locality: 0 2023-12-25T20:06:26.221 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.max_successive_merges: 0 2023-12-25T20:06:26.221 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.optimize_filters_for_hits: 0 2023-12-25T20:06:26.221 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.paranoid_file_checks: 0 2023-12-25T20:06:26.221 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.force_consistency_checks: 0 2023-12-25T20:06:26.221 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.report_bg_io_stats: 0 2023-12-25T20:06:26.221 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.ttl: 2592000 2023-12-25T20:06:26.221 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.724+0000 7f06f3069700 4 rocksdb: Options.periodic_compaction_seconds: 0 2023-12-25T20:06:26.221 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.725+0000 7f06f3069700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi050/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 5, log_number is 4,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-12-25T20:06:26.221 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: 2023-12-25T20:06:26.222 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.725+0000 7f06f3069700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 4 2023-12-25T20:06:26.222 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: 2023-12-25T20:06:26.222 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.725+0000 7f06f3069700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1703534785726955, "job": 1, "event": "recovery_started", "log_files": [6]} 2023-12-25T20:06:26.222 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.725+0000 7f06f3069700 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #6 mode 2 2023-12-25T20:06:26.222 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.726+0000 7f06f3069700 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2023-12-25T20:06:26.222 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 systemd[1]: Started Ceph mon.smithi050 for 02c8f25c-a361-11ee-95a5-87774f69a715. 2023-12-25T20:06:26.222 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.726+0000 7f06f3069700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1703534785727884, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 7, "file_size": 88692, "table_properties": {"data_size": 86807, "index_size": 362, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 709, "raw_key_size": 12384, "raw_average_key_size": 49, "raw_value_size": 79948, "raw_average_value_size": 321, "num_data_blocks": 11, "num_entries": 249, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1703534785, "oldest_key_time": 3, "file_creation_time": 0}} 2023-12-25T20:06:26.222 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.726+0000 7f06f3069700 4 rocksdb: [version_set.cc:3826] Creating manifest 8 2023-12-25T20:06:26.222 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: 2023-12-25T20:06:26.222 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.727+0000 7f06f3069700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1703534785728369, "job": 1, "event": "recovery_finished"} 2023-12-25T20:06:26.222 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.727+0000 7f06f3069700 4 rocksdb: DB pointer 0x5637fa817800 2023-12-25T20:06:26.222 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.728+0000 7f06d9e81700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2023-12-25T20:06:26.222 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.728+0000 7f06d9e81700 4 rocksdb: [db_impl/db_impl.cc:851] 2023-12-25T20:06:26.222 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: ** DB Stats ** 2023-12-25T20:06:26.223 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: Uptime(secs): 0.0 total, 0.0 interval 2023-12-25T20:06:26.223 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-12-25T20:06:26.223 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-12-25T20:06:26.223 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-12-25T20:06:26.223 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-12-25T20:06:26.223 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2023-12-25T20:06:26.223 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-12-25T20:06:26.223 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: 2023-12-25T20:06:26.223 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: ** Compaction Stats [default] ** 2023-12-25T20:06:26.223 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-12-25T20:06:26.223 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-12-25T20:06:26.223 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: L0 2/0 88.31 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 135.6 0.00 0.00 1 0.001 0 0 2023-12-25T20:06:26.223 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: Sum 2/0 88.31 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 135.6 0.00 0.00 1 0.001 0 0 2023-12-25T20:06:26.223 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 135.6 0.00 0.00 1 0.001 0 0 2023-12-25T20:06:26.223 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: 2023-12-25T20:06:26.224 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: ** Compaction Stats [default] ** 2023-12-25T20:06:26.224 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-12-25T20:06:26.224 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-12-25T20:06:26.224 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 135.6 0.00 0.00 1 0.001 0 0 2023-12-25T20:06:26.224 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: Uptime(secs): 0.0 total, 0.0 interval 2023-12-25T20:06:26.224 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: Flush(GB): cumulative 0.000, interval 0.000 2023-12-25T20:06:26.224 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: AddFile(GB): cumulative 0.000, interval 0.000 2023-12-25T20:06:26.224 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: AddFile(Total Files): cumulative 0, interval 0 2023-12-25T20:06:26.224 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: AddFile(L0 Files): cumulative 0, interval 0 2023-12-25T20:06:26.224 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: AddFile(Keys): cumulative 0, interval 0 2023-12-25T20:06:26.224 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: Cumulative compaction: 0.00 GB write, 27.75 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-12-25T20:06:26.224 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: Interval compaction: 0.00 GB write, 27.75 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-12-25T20:06:26.224 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-12-25T20:06:26.224 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: 2023-12-25T20:06:26.224 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: ** File Read Latency Histogram By Level [default] ** 2023-12-25T20:06:26.225 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: 2023-12-25T20:06:26.225 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: ** Compaction Stats [default] ** 2023-12-25T20:06:26.225 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-12-25T20:06:26.225 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-12-25T20:06:26.225 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: L0 2/0 88.31 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 135.6 0.00 0.00 1 0.001 0 0 2023-12-25T20:06:26.225 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: Sum 2/0 88.31 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 135.6 0.00 0.00 1 0.001 0 0 2023-12-25T20:06:26.225 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-12-25T20:06:26.225 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: 2023-12-25T20:06:26.225 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: ** Compaction Stats [default] ** 2023-12-25T20:06:26.225 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-12-25T20:06:26.225 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-12-25T20:06:26.225 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 135.6 0.00 0.00 1 0.001 0 0 2023-12-25T20:06:26.225 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: Uptime(secs): 0.0 total, 0.0 interval 2023-12-25T20:06:26.225 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: Flush(GB): cumulative 0.000, interval 0.000 2023-12-25T20:06:26.226 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: AddFile(GB): cumulative 0.000, interval 0.000 2023-12-25T20:06:26.226 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: AddFile(Total Files): cumulative 0, interval 0 2023-12-25T20:06:26.226 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: AddFile(L0 Files): cumulative 0, interval 0 2023-12-25T20:06:26.226 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: AddFile(Keys): cumulative 0, interval 0 2023-12-25T20:06:26.226 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: Cumulative compaction: 0.00 GB write, 27.47 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-12-25T20:06:26.226 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-12-25T20:06:26.226 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-12-25T20:06:26.226 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: 2023-12-25T20:06:26.226 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: ** File Read Latency Histogram By Level [default] ** 2023-12-25T20:06:26.226 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: 2023-12-25T20:06:26.226 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.728+0000 7f06f3069700 0 starting mon.smithi050 rank 0 at public addrs [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] at bind addrs [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi050 fsid 02c8f25c-a361-11ee-95a5-87774f69a715 2023-12-25T20:06:26.226 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.729+0000 7f06f3069700 1 mon.smithi050@-1(???) e1 preinit fsid 02c8f25c-a361-11ee-95a5-87774f69a715 2023-12-25T20:06:26.226 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.730+0000 7f06f3069700 0 mon.smithi050@-1(???).mds e1 new map 2023-12-25T20:06:26.226 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.730+0000 7f06f3069700 0 mon.smithi050@-1(???).mds e1 print_map 2023-12-25T20:06:26.227 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: e1 2023-12-25T20:06:26.227 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: enable_multiple, ever_enabled_multiple: 1,1 2023-12-25T20:06:26.227 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-12-25T20:06:26.227 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: legacy client fscid: -1 2023-12-25T20:06:26.227 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: 2023-12-25T20:06:26.227 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: No filesystems configured 2023-12-25T20:06:26.227 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: 2023-12-25T20:06:26.227 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.730+0000 7f06f3069700 0 mon.smithi050@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2023-12-25T20:06:26.227 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.730+0000 7f06f3069700 0 mon.smithi050@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-12-25T20:06:26.227 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.730+0000 7f06f3069700 0 mon.smithi050@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-12-25T20:06:26.227 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.730+0000 7f06f3069700 0 mon.smithi050@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-12-25T20:06:26.227 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:06:23.598420+0000 mon.smithi050 (mon.0) 8 : audit [INF] from='client.? 172.21.15.50:0/741476600' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-12-25T20:06:26.227 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:06:23.601352+0000 mon.smithi050 (mon.0) 9 : audit [INF] from='client.? 172.21.15.50:0/741476600' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2023-12-25T20:06:26.227 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.730+0000 7f06f3069700 1 mon.smithi050@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2023-12-25T20:06:26.227 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.731+0000 7f06f3069700 20 expand_channel_meta expand map: {default=false} 2023-12-25T20:06:26.228 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.731+0000 7f06f3069700 20 expand_channel_meta from 'false' to 'false' 2023-12-25T20:06:26.228 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.731+0000 7f06f3069700 20 expand_channel_meta expanded map: {default=false} 2023-12-25T20:06:26.228 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.731+0000 7f06f3069700 20 expand_channel_meta expand map: {default=info} 2023-12-25T20:06:26.228 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.731+0000 7f06f3069700 20 expand_channel_meta from 'info' to 'info' 2023-12-25T20:06:26.228 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.731+0000 7f06f3069700 20 expand_channel_meta expanded map: {default=info} 2023-12-25T20:06:26.228 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.731+0000 7f06f3069700 20 expand_channel_meta expand map: {default=daemon} 2023-12-25T20:06:26.228 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.731+0000 7f06f3069700 20 expand_channel_meta from 'daemon' to 'daemon' 2023-12-25T20:06:26.228 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.731+0000 7f06f3069700 20 expand_channel_meta expanded map: {default=daemon} 2023-12-25T20:06:26.228 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.731+0000 7f06f3069700 20 expand_channel_meta expand map: {default=debug} 2023-12-25T20:06:26.228 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.731+0000 7f06f3069700 20 expand_channel_meta from 'debug' to 'debug' 2023-12-25T20:06:26.228 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:06:25.731+0000 7f06f3069700 20 expand_channel_meta expanded map: {default=debug} 2023-12-25T20:06:26.588 INFO:teuthology.orchestra.run.smithi050.stderr:Wrote config to /etc/ceph/ceph.conf 2023-12-25T20:06:26.588 INFO:teuthology.orchestra.run.smithi050.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2023-12-25T20:06:26.588 INFO:teuthology.orchestra.run.smithi050.stderr:Creating mgr... 2023-12-25T20:06:26.589 INFO:teuthology.orchestra.run.smithi050.stderr:Verifying port 9283 ... 2023-12-25T20:06:26.590 INFO:teuthology.orchestra.run.smithi050.stderr:Running command: systemctl daemon-reload 2023-12-25T20:06:26.690 INFO:teuthology.orchestra.run.smithi050.stderr:Running command: systemctl stop ceph-02c8f25c-a361-11ee-95a5-87774f69a715@mgr.smithi050.nbzhun 2023-12-25T20:06:26.715 INFO:teuthology.orchestra.run.smithi050.stderr:Running command: systemctl reset-failed ceph-02c8f25c-a361-11ee-95a5-87774f69a715@mgr.smithi050.nbzhun 2023-12-25T20:06:26.721 INFO:teuthology.orchestra.run.smithi050.stderr:systemctl: Failed to reset failed state of unit ceph-02c8f25c-a361-11ee-95a5-87774f69a715@mgr.smithi050.nbzhun.service: Unit ceph-02c8f25c-a361-11ee-95a5-87774f69a715@mgr.smithi050.nbzhun.service not loaded. 2023-12-25T20:06:26.721 INFO:teuthology.orchestra.run.smithi050.stderr:Running command: systemctl enable ceph-02c8f25c-a361-11ee-95a5-87774f69a715@mgr.smithi050.nbzhun 2023-12-25T20:06:26.727 INFO:teuthology.orchestra.run.smithi050.stderr:systemctl: Created symlink /etc/systemd/system/ceph-02c8f25c-a361-11ee-95a5-87774f69a715.target.wants/ceph-02c8f25c-a361-11ee-95a5-87774f69a715@mgr.smithi050.nbzhun.service → /etc/systemd/system/ceph-02c8f25c-a361-11ee-95a5-87774f69a715@.service. 2023-12-25T20:06:26.816 INFO:teuthology.orchestra.run.smithi050.stderr:Running command: systemctl start ceph-02c8f25c-a361-11ee-95a5-87774f69a715@mgr.smithi050.nbzhun 2023-12-25T20:06:26.969 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:26 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:06:25.746893+0000 mon.smithi050 (mon.0) 1 : cluster [INF] mon.smithi050 is new leader, mons smithi050 in quorum (ranks 0) 2023-12-25T20:06:26.969 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:26 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:06:25.746979+0000 mon.smithi050 (mon.0) 2 : cluster [DBG] monmap e1: 1 mons at {smithi050=[v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0]} 2023-12-25T20:06:26.969 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:26 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:06:25.747019+0000 mon.smithi050 (mon.0) 3 : cluster [DBG] fsmap 2023-12-25T20:06:26.969 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:26 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:06:25.747051+0000 mon.smithi050 (mon.0) 4 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2023-12-25T20:06:26.969 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:26 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:06:25.747445+0000 mon.smithi050 (mon.0) 5 : cluster [DBG] mgrmap e1: no daemons active 2023-12-25T20:06:26.969 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:26 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:06:26.174795+0000 mon.smithi050 (mon.0) 6 : audit [INF] from='client.? 172.21.15.50:0/4251166667' entity='client.admin' 2023-12-25T20:06:27.145 INFO:teuthology.orchestra.run.smithi050.stderr:Running command: systemctl is-enabled firewalld.service 2023-12-25T20:06:27.155 INFO:teuthology.orchestra.run.smithi050.stderr:systemctl: disabled 2023-12-25T20:06:27.155 INFO:teuthology.orchestra.run.smithi050.stderr:Running command: systemctl is-active firewalld.service 2023-12-25T20:06:27.164 INFO:teuthology.orchestra.run.smithi050.stderr:systemctl: inactive 2023-12-25T20:06:27.165 INFO:teuthology.orchestra.run.smithi050.stderr:firewalld.service is not enabled 2023-12-25T20:06:27.165 INFO:teuthology.orchestra.run.smithi050.stderr:Not possible to enable service . firewalld.service is not available 2023-12-25T20:06:27.165 INFO:teuthology.orchestra.run.smithi050.stderr:Not possible to open ports <[]>. firewalld.service is not available 2023-12-25T20:06:27.165 INFO:teuthology.orchestra.run.smithi050.stderr:Running command: systemctl is-enabled firewalld.service 2023-12-25T20:06:27.173 INFO:teuthology.orchestra.run.smithi050.stderr:systemctl: disabled 2023-12-25T20:06:27.174 INFO:teuthology.orchestra.run.smithi050.stderr:Running command: systemctl is-active firewalld.service 2023-12-25T20:06:27.182 INFO:teuthology.orchestra.run.smithi050.stderr:systemctl: inactive 2023-12-25T20:06:27.183 INFO:teuthology.orchestra.run.smithi050.stderr:firewalld.service is not enabled 2023-12-25T20:06:27.183 INFO:teuthology.orchestra.run.smithi050.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2023-12-25T20:06:27.183 INFO:teuthology.orchestra.run.smithi050.stderr:Waiting for mgr to start... 2023-12-25T20:06:27.183 INFO:teuthology.orchestra.run.smithi050.stderr:Waiting for mgr... 2023-12-25T20:06:27.183 INFO:teuthology.orchestra.run.smithi050.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi050 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/02c8f25c-a361-11ee-95a5-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmps7kf4jlu:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpu2g3lwcx:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2023-12-25T20:06:27.567 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:27.562+0000 7f0ccaabd700 1 Processor -- start 2023-12-25T20:06:27.567 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:27.562+0000 7f0ccaabd700 1 -- start start 2023-12-25T20:06:27.567 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:27.562+0000 7f0ccaabd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0cc4070430 0x7f0cc40fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-25T20:06:27.567 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:27.562+0000 7f0ccaabd700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cc40fedd0 con 0x7f0cc4070430 2023-12-25T20:06:27.567 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:27.563+0000 7f0cc8859700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0cc4070430 0x7f0cc40fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:06:27.567 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:27.563+0000 7f0cc8859700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0cc4070430 0x7f0cc40fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:43964/0 (socket says 172.21.15.50:43964) 2023-12-25T20:06:27.567 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:27.563+0000 7f0cc8859700 1 -- 172.21.15.50:0/2656276140 learned_addr learned my addr 172.21.15.50:0/2656276140 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-25T20:06:27.567 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:27.563+0000 7f0cc8859700 1 -- 172.21.15.50:0/2656276140 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cc40fef10 con 0x7f0cc4070430 2023-12-25T20:06:27.568 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:27.563+0000 7f0cc8859700 1 --2- 172.21.15.50:0/2656276140 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0cc4070430 0x7f0cc40fe8e0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f0cb800b5f0 tx=0x7f0cb8006ee0).ready entity=mon.0 client_cookie=13f82d6944fa7322 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:06:27.568 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:27.563+0000 7f0cc37fe700 1 -- 172.21.15.50:0/2656276140 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0cb800d600 con 0x7f0cc4070430 2023-12-25T20:06:27.568 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:27.563+0000 7f0cc37fe700 1 -- 172.21.15.50:0/2656276140 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f0cb800dbd0 con 0x7f0cc4070430 2023-12-25T20:06:27.568 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:27.563+0000 7f0cc37fe700 1 -- 172.21.15.50:0/2656276140 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0cb8011c00 con 0x7f0cc4070430 2023-12-25T20:06:27.568 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:27.564+0000 7f0ccaabd700 1 -- 172.21.15.50:0/2656276140 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0cc4070430 msgr2=0x7f0cc40fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:06:27.568 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:27.564+0000 7f0ccaabd700 1 --2- 172.21.15.50:0/2656276140 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0cc4070430 0x7f0cc40fe8e0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f0cb800b5f0 tx=0x7f0cb8006ee0).stop 2023-12-25T20:06:27.568 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:27.564+0000 7f0ccaabd700 1 -- 172.21.15.50:0/2656276140 shutdown_connections 2023-12-25T20:06:27.568 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:27.564+0000 7f0ccaabd700 1 --2- 172.21.15.50:0/2656276140 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0cc4070430 0x7f0cc40fe8e0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:06:27.568 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:27.564+0000 7f0ccaabd700 1 -- 172.21.15.50:0/2656276140 >> 172.21.15.50:0/2656276140 conn(0x7f0cc40f63c0 msgr2=0x7f0cc40f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:06:27.568 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:27.564+0000 7f0ccaabd700 1 -- 172.21.15.50:0/2656276140 shutdown_connections 2023-12-25T20:06:27.568 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:27.564+0000 7f0ccaabd700 1 -- 172.21.15.50:0/2656276140 wait complete. 2023-12-25T20:06:27.568 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:27.565+0000 7f0ccaabd700 1 Processor -- start 2023-12-25T20:06:27.568 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:27.565+0000 7f0ccaabd700 1 -- start start 2023-12-25T20:06:27.569 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:27.565+0000 7f0ccaabd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0cc4070430 0x7f0cc410aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:06:27.569 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:27.565+0000 7f0ccaabd700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cc410aef0 con 0x7f0cc4070430 2023-12-25T20:06:27.569 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:27.565+0000 7f0cc8859700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0cc4070430 0x7f0cc410aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:06:27.569 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:27.565+0000 7f0cc8859700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0cc4070430 0x7f0cc410aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:43974/0 (socket says 172.21.15.50:43974) 2023-12-25T20:06:27.569 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:27.565+0000 7f0cc8859700 1 -- 172.21.15.50:0/150425348 learned_addr learned my addr 172.21.15.50:0/150425348 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-25T20:06:27.569 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:27.566+0000 7f0cc8859700 1 -- 172.21.15.50:0/150425348 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cb800b200 con 0x7f0cc4070430 2023-12-25T20:06:27.569 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:27.566+0000 7f0cc8859700 1 --2- 172.21.15.50:0/150425348 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0cc4070430 0x7f0cc410aa00 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f0cb800b190 tx=0x7f0cb8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:06:27.569 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:27.566+0000 7f0cc17fa700 1 -- 172.21.15.50:0/150425348 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0cb801b410 con 0x7f0cc4070430 2023-12-25T20:06:27.569 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:27.566+0000 7f0ccaabd700 1 -- 172.21.15.50:0/150425348 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cc410b0f0 con 0x7f0cc4070430 2023-12-25T20:06:27.569 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:27.566+0000 7f0cc17fa700 1 -- 172.21.15.50:0/150425348 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f0cb801b9e0 con 0x7f0cc4070430 2023-12-25T20:06:27.569 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:27.566+0000 7f0cc17fa700 1 -- 172.21.15.50:0/150425348 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0cb801a880 con 0x7f0cc4070430 2023-12-25T20:06:27.570 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:27.566+0000 7f0ccaabd700 1 -- 172.21.15.50:0/150425348 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cc410b500 con 0x7f0cc4070430 2023-12-25T20:06:27.570 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:27.567+0000 7f0cc17fa700 1 -- 172.21.15.50:0/150425348 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f0cb800f070 con 0x7f0cc4070430 2023-12-25T20:06:27.570 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:27.567+0000 7f0cc17fa700 1 -- 172.21.15.50:0/150425348 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f0cb8025820 con 0x7f0cc4070430 2023-12-25T20:06:27.570 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:27.567+0000 7f0ccaabd700 1 -- 172.21.15.50:0/150425348 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cc4045ad0 con 0x7f0cc4070430 2023-12-25T20:06:27.570 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:27.570+0000 7f0cc17fa700 1 -- 172.21.15.50:0/150425348 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f0cb8016070 con 0x7f0cc4070430 2023-12-25T20:06:27.647 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:27.640+0000 7f0ccaabd700 1 -- 172.21.15.50:0/150425348 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f0cc4104890 con 0x7f0cc4070430 2023-12-25T20:06:27.647 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:27.640+0000 7f0cc17fa700 1 -- 172.21.15.50:0/150425348 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7f0cb801aaf0 con 0x7f0cc4070430 2023-12-25T20:06:27.648 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:27.648 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: { 2023-12-25T20:06:27.648 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "fsid": "02c8f25c-a361-11ee-95a5-87774f69a715", 2023-12-25T20:06:27.648 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "health": { 2023-12-25T20:06:27.648 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2023-12-25T20:06:27.648 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "checks": {}, 2023-12-25T20:06:27.648 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "mutes": [] 2023-12-25T20:06:27.648 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: }, 2023-12-25T20:06:27.648 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "election_epoch": 5, 2023-12-25T20:06:27.648 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "quorum": [ 2023-12-25T20:06:27.648 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 0 2023-12-25T20:06:27.648 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: ], 2023-12-25T20:06:27.648 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "quorum_names": [ 2023-12-25T20:06:27.649 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "smithi050" 2023-12-25T20:06:27.649 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: ], 2023-12-25T20:06:27.649 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "quorum_age": 1, 2023-12-25T20:06:27.649 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "monmap": { 2023-12-25T20:06:27.649 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "epoch": 1, 2023-12-25T20:06:27.649 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2023-12-25T20:06:27.649 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "num_mons": 1 2023-12-25T20:06:27.649 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: }, 2023-12-25T20:06:27.649 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "osdmap": { 2023-12-25T20:06:27.649 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "epoch": 1, 2023-12-25T20:06:27.649 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "num_osds": 0, 2023-12-25T20:06:27.649 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "num_up_osds": 0, 2023-12-25T20:06:27.649 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "osd_up_since": 0, 2023-12-25T20:06:27.650 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "num_in_osds": 0, 2023-12-25T20:06:27.650 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "osd_in_since": 0, 2023-12-25T20:06:27.650 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2023-12-25T20:06:27.650 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: }, 2023-12-25T20:06:27.650 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "pgmap": { 2023-12-25T20:06:27.650 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "pgs_by_state": [], 2023-12-25T20:06:27.650 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "num_pgs": 0, 2023-12-25T20:06:27.650 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "num_pools": 0, 2023-12-25T20:06:27.650 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "num_objects": 0, 2023-12-25T20:06:27.650 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "data_bytes": 0, 2023-12-25T20:06:27.650 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "bytes_used": 0, 2023-12-25T20:06:27.651 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "bytes_avail": 0, 2023-12-25T20:06:27.651 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "bytes_total": 0 2023-12-25T20:06:27.651 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: }, 2023-12-25T20:06:27.651 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "fsmap": { 2023-12-25T20:06:27.651 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "epoch": 1, 2023-12-25T20:06:27.651 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "by_rank": [], 2023-12-25T20:06:27.651 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "up:standby": 0 2023-12-25T20:06:27.651 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: }, 2023-12-25T20:06:27.651 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "mgrmap": { 2023-12-25T20:06:27.651 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "available": false, 2023-12-25T20:06:27.651 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "num_standbys": 0, 2023-12-25T20:06:27.651 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "modules": [ 2023-12-25T20:06:27.651 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "iostat", 2023-12-25T20:06:27.651 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "restful" 2023-12-25T20:06:27.652 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: ], 2023-12-25T20:06:27.652 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "services": {} 2023-12-25T20:06:27.652 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: }, 2023-12-25T20:06:27.652 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "servicemap": { 2023-12-25T20:06:27.652 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "epoch": 1, 2023-12-25T20:06:27.652 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "modified": "2023-12-25T20:06:22.287710+0000", 2023-12-25T20:06:27.652 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "services": {} 2023-12-25T20:06:27.652 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: }, 2023-12-25T20:06:27.652 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "progress_events": {} 2023-12-25T20:06:27.652 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: } 2023-12-25T20:06:27.652 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:27.641+0000 7f0ccaabd700 1 -- 172.21.15.50:0/150425348 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0cc4070430 msgr2=0x7f0cc410aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:06:27.652 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:27.641+0000 7f0ccaabd700 1 --2- 172.21.15.50:0/150425348 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0cc4070430 0x7f0cc410aa00 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f0cb800b190 tx=0x7f0cb8006ee0).stop 2023-12-25T20:06:27.652 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:27.641+0000 7f0ccaabd700 1 -- 172.21.15.50:0/150425348 shutdown_connections 2023-12-25T20:06:27.652 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:27.641+0000 7f0ccaabd700 1 --2- 172.21.15.50:0/150425348 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0cc4070430 0x7f0cc410aa00 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:06:27.652 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:27.641+0000 7f0ccaabd700 1 -- 172.21.15.50:0/150425348 >> 172.21.15.50:0/150425348 conn(0x7f0cc40f63c0 msgr2=0x7f0cc40f7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:06:27.653 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:27.641+0000 7f0ccaabd700 1 -- 172.21.15.50:0/150425348 shutdown_connections 2023-12-25T20:06:27.653 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:27.641+0000 7f0ccaabd700 1 -- 172.21.15.50:0/150425348 wait complete. 2023-12-25T20:06:27.968 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:27 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:06:27.641596+0000 mon.smithi050 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.50:0/150425348' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-12-25T20:06:28.018 INFO:teuthology.orchestra.run.smithi050.stderr:mgr not available, waiting (1/15)... 2023-12-25T20:06:30.020 INFO:teuthology.orchestra.run.smithi050.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi050 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/02c8f25c-a361-11ee-95a5-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmps7kf4jlu:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpu2g3lwcx:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2023-12-25T20:06:30.422 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:30.421+0000 7fdf67ac8700 1 Processor -- start 2023-12-25T20:06:30.423 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:30.421+0000 7fdf67ac8700 1 -- start start 2023-12-25T20:06:30.423 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:30.422+0000 7fdf67ac8700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdf600fab80 0x7fdf600fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-25T20:06:30.423 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:30.422+0000 7fdf67ac8700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf600fb490 con 0x7fdf600fab80 2023-12-25T20:06:30.423 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:30.422+0000 7fdf65864700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdf600fab80 0x7fdf600fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:06:30.423 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:30.422+0000 7fdf65864700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdf600fab80 0x7fdf600fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:43978/0 (socket says 172.21.15.50:43978) 2023-12-25T20:06:30.423 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:30.422+0000 7fdf65864700 1 -- 172.21.15.50:0/2370479322 learned_addr learned my addr 172.21.15.50:0/2370479322 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-25T20:06:30.423 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:30.422+0000 7fdf65864700 1 -- 172.21.15.50:0/2370479322 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf600fb5d0 con 0x7fdf600fab80 2023-12-25T20:06:30.423 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:30.422+0000 7fdf65864700 1 --2- 172.21.15.50:0/2370479322 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdf600fab80 0x7fdf600fafa0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fdf5c00b5f0 tx=0x7fdf5c006ee0).ready entity=mon.0 client_cookie=724dae8b4ac79451 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:06:30.425 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:30.423+0000 7fdf64862700 1 -- 172.21.15.50:0/2370479322 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdf5c00d600 con 0x7fdf600fab80 2023-12-25T20:06:30.425 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:30.423+0000 7fdf64862700 1 -- 172.21.15.50:0/2370479322 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fdf5c00dbd0 con 0x7fdf600fab80 2023-12-25T20:06:30.425 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:30.423+0000 7fdf64862700 1 -- 172.21.15.50:0/2370479322 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdf5c011c00 con 0x7fdf600fab80 2023-12-25T20:06:30.425 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:30.424+0000 7fdf67ac8700 1 -- 172.21.15.50:0/2370479322 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdf600fab80 msgr2=0x7fdf600fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:06:30.425 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:30.424+0000 7fdf67ac8700 1 --2- 172.21.15.50:0/2370479322 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdf600fab80 0x7fdf600fafa0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fdf5c00b5f0 tx=0x7fdf5c006ee0).stop 2023-12-25T20:06:30.426 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:30.424+0000 7fdf67ac8700 1 -- 172.21.15.50:0/2370479322 shutdown_connections 2023-12-25T20:06:30.426 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:30.424+0000 7fdf67ac8700 1 --2- 172.21.15.50:0/2370479322 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdf600fab80 0x7fdf600fafa0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:06:30.426 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:30.424+0000 7fdf67ac8700 1 -- 172.21.15.50:0/2370479322 >> 172.21.15.50:0/2370479322 conn(0x7fdf600f63c0 msgr2=0x7fdf600f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:06:30.426 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:30.424+0000 7fdf67ac8700 1 -- 172.21.15.50:0/2370479322 shutdown_connections 2023-12-25T20:06:30.426 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:30.424+0000 7fdf67ac8700 1 -- 172.21.15.50:0/2370479322 wait complete. 2023-12-25T20:06:30.426 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:30.425+0000 7fdf67ac8700 1 Processor -- start 2023-12-25T20:06:30.426 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:30.425+0000 7fdf67ac8700 1 -- start start 2023-12-25T20:06:30.426 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:30.425+0000 7fdf67ac8700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdf600fab80 0x7fdf6010d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:06:30.426 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:30.425+0000 7fdf67ac8700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf60070450 con 0x7fdf600fab80 2023-12-25T20:06:30.426 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:30.425+0000 7fdf65864700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdf600fab80 0x7fdf6010d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:06:30.427 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:30.425+0000 7fdf65864700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdf600fab80 0x7fdf6010d740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:43980/0 (socket says 172.21.15.50:43980) 2023-12-25T20:06:30.427 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:30.425+0000 7fdf65864700 1 -- 172.21.15.50:0/693297885 learned_addr learned my addr 172.21.15.50:0/693297885 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-25T20:06:30.427 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:30.425+0000 7fdf65864700 1 -- 172.21.15.50:0/693297885 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf5c00b200 con 0x7fdf600fab80 2023-12-25T20:06:30.427 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:30.425+0000 7fdf65864700 1 --2- 172.21.15.50:0/693297885 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdf600fab80 0x7fdf6010d740 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fdf5c00e040 tx=0x7fdf5c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:06:30.427 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:30.425+0000 7fdf4e7fc700 1 -- 172.21.15.50:0/693297885 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdf5c01b410 con 0x7fdf600fab80 2023-12-25T20:06:30.427 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:30.425+0000 7fdf4e7fc700 1 -- 172.21.15.50:0/693297885 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fdf5c01b9e0 con 0x7fdf600fab80 2023-12-25T20:06:30.428 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:30.426+0000 7fdf67ac8700 1 -- 172.21.15.50:0/693297885 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf60070650 con 0x7fdf600fab80 2023-12-25T20:06:30.428 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:30.426+0000 7fdf4e7fc700 1 -- 172.21.15.50:0/693297885 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdf5c01a820 con 0x7fdf600fab80 2023-12-25T20:06:30.428 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:30.426+0000 7fdf67ac8700 1 -- 172.21.15.50:0/693297885 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf6010ddf0 con 0x7fdf600fab80 2023-12-25T20:06:30.428 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:30.426+0000 7fdf4e7fc700 1 -- 172.21.15.50:0/693297885 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7fdf5c00f070 con 0x7fdf600fab80 2023-12-25T20:06:30.428 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:30.426+0000 7fdf4e7fc700 1 -- 172.21.15.50:0/693297885 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fdf5c025b30 con 0x7fdf600fab80 2023-12-25T20:06:30.428 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:30.426+0000 7fdf67ac8700 1 -- 172.21.15.50:0/693297885 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf60045ad0 con 0x7fdf600fab80 2023-12-25T20:06:30.428 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:30.429+0000 7fdf4e7fc700 1 -- 172.21.15.50:0/693297885 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7fdf5c016070 con 0x7fdf600fab80 2023-12-25T20:06:30.504 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:30.498+0000 7fdf67ac8700 1 -- 172.21.15.50:0/693297885 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fdf600fc930 con 0x7fdf600fab80 2023-12-25T20:06:30.504 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:30.499+0000 7fdf4e7fc700 1 -- 172.21.15.50:0/693297885 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7fdf5c006a70 con 0x7fdf600fab80 2023-12-25T20:06:30.505 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:30.505 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: { 2023-12-25T20:06:30.505 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "fsid": "02c8f25c-a361-11ee-95a5-87774f69a715", 2023-12-25T20:06:30.505 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "health": { 2023-12-25T20:06:30.505 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2023-12-25T20:06:30.505 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "checks": {}, 2023-12-25T20:06:30.505 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "mutes": [] 2023-12-25T20:06:30.505 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: }, 2023-12-25T20:06:30.505 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "election_epoch": 5, 2023-12-25T20:06:30.505 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "quorum": [ 2023-12-25T20:06:30.505 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 0 2023-12-25T20:06:30.505 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: ], 2023-12-25T20:06:30.505 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "quorum_names": [ 2023-12-25T20:06:30.505 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "smithi050" 2023-12-25T20:06:30.505 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: ], 2023-12-25T20:06:30.506 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "quorum_age": 4, 2023-12-25T20:06:30.506 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "monmap": { 2023-12-25T20:06:30.506 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "epoch": 1, 2023-12-25T20:06:30.506 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2023-12-25T20:06:30.506 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "num_mons": 1 2023-12-25T20:06:30.506 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: }, 2023-12-25T20:06:30.506 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "osdmap": { 2023-12-25T20:06:30.506 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "epoch": 1, 2023-12-25T20:06:30.506 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "num_osds": 0, 2023-12-25T20:06:30.506 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "num_up_osds": 0, 2023-12-25T20:06:30.506 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "osd_up_since": 0, 2023-12-25T20:06:30.506 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "num_in_osds": 0, 2023-12-25T20:06:30.506 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "osd_in_since": 0, 2023-12-25T20:06:30.506 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2023-12-25T20:06:30.507 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: }, 2023-12-25T20:06:30.507 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "pgmap": { 2023-12-25T20:06:30.507 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "pgs_by_state": [], 2023-12-25T20:06:30.507 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "num_pgs": 0, 2023-12-25T20:06:30.507 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "num_pools": 0, 2023-12-25T20:06:30.507 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "num_objects": 0, 2023-12-25T20:06:30.507 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "data_bytes": 0, 2023-12-25T20:06:30.507 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "bytes_used": 0, 2023-12-25T20:06:30.507 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "bytes_avail": 0, 2023-12-25T20:06:30.507 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "bytes_total": 0 2023-12-25T20:06:30.507 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: }, 2023-12-25T20:06:30.507 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "fsmap": { 2023-12-25T20:06:30.507 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "epoch": 1, 2023-12-25T20:06:30.507 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "by_rank": [], 2023-12-25T20:06:30.507 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "up:standby": 0 2023-12-25T20:06:30.508 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: }, 2023-12-25T20:06:30.508 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "mgrmap": { 2023-12-25T20:06:30.508 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "available": false, 2023-12-25T20:06:30.508 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "num_standbys": 0, 2023-12-25T20:06:30.508 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "modules": [ 2023-12-25T20:06:30.508 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "iostat", 2023-12-25T20:06:30.508 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "restful" 2023-12-25T20:06:30.508 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: ], 2023-12-25T20:06:30.508 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "services": {} 2023-12-25T20:06:30.508 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: }, 2023-12-25T20:06:30.508 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "servicemap": { 2023-12-25T20:06:30.508 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "epoch": 1, 2023-12-25T20:06:30.508 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "modified": "2023-12-25T20:06:22.287710+0000", 2023-12-25T20:06:30.508 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "services": {} 2023-12-25T20:06:30.508 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: }, 2023-12-25T20:06:30.508 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "progress_events": {} 2023-12-25T20:06:30.509 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: } 2023-12-25T20:06:30.509 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:30.500+0000 7fdf67ac8700 1 -- 172.21.15.50:0/693297885 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdf600fab80 msgr2=0x7fdf6010d740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:06:30.509 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:30.500+0000 7fdf67ac8700 1 --2- 172.21.15.50:0/693297885 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdf600fab80 0x7fdf6010d740 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fdf5c00e040 tx=0x7fdf5c006ee0).stop 2023-12-25T20:06:30.509 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:30.500+0000 7fdf67ac8700 1 -- 172.21.15.50:0/693297885 shutdown_connections 2023-12-25T20:06:30.509 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:30.500+0000 7fdf67ac8700 1 --2- 172.21.15.50:0/693297885 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdf600fab80 0x7fdf6010d740 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:06:30.509 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:30.500+0000 7fdf67ac8700 1 -- 172.21.15.50:0/693297885 >> 172.21.15.50:0/693297885 conn(0x7fdf600f63c0 msgr2=0x7fdf600f7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:06:30.509 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:30.500+0000 7fdf67ac8700 1 -- 172.21.15.50:0/693297885 shutdown_connections 2023-12-25T20:06:30.509 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:30.500+0000 7fdf67ac8700 1 -- 172.21.15.50:0/693297885 wait complete. 2023-12-25T20:06:30.853 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:30 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:06:30.499315+0000 mon.smithi050 (mon.0) 8 : audit [DBG] from='client.? 172.21.15.50:0/693297885' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-12-25T20:06:30.867 INFO:teuthology.orchestra.run.smithi050.stderr:mgr not available, waiting (2/15)... 2023-12-25T20:06:32.719 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:32 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:06:32.319267+0000 mon.smithi050 (mon.0) 9 : cluster [INF] Activating manager daemon smithi050.nbzhun 2023-12-25T20:06:32.719 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:32 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:06:32.321668+0000 mon.smithi050 (mon.0) 10 : cluster [DBG] mgrmap e2: smithi050.nbzhun(active, starting, since 0.00252194s) 2023-12-25T20:06:32.719 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:32 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:06:32.322623+0000 mon.smithi050 (mon.0) 11 : audit [DBG] from='mgr.14100 172.21.15.50:0/269848152' entity='mgr.smithi050.nbzhun' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-12-25T20:06:32.719 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:32 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:06:32.322834+0000 mon.smithi050 (mon.0) 12 : audit [DBG] from='mgr.14100 172.21.15.50:0/269848152' entity='mgr.smithi050.nbzhun' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-12-25T20:06:32.719 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:32 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:06:32.322961+0000 mon.smithi050 (mon.0) 13 : audit [DBG] from='mgr.14100 172.21.15.50:0/269848152' entity='mgr.smithi050.nbzhun' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-12-25T20:06:32.719 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:32 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:06:32.323121+0000 mon.smithi050 (mon.0) 14 : audit [DBG] from='mgr.14100 172.21.15.50:0/269848152' entity='mgr.smithi050.nbzhun' cmd=[{"prefix": "mon metadata", "id": "smithi050"}]: dispatch 2023-12-25T20:06:32.719 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:32 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:06:32.323309+0000 mon.smithi050 (mon.0) 15 : audit [DBG] from='mgr.14100 172.21.15.50:0/269848152' entity='mgr.smithi050.nbzhun' cmd=[{"prefix": "mgr metadata", "who": "smithi050.nbzhun", "id": "smithi050.nbzhun"}]: dispatch 2023-12-25T20:06:32.719 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:32 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:06:32.336509+0000 mon.smithi050 (mon.0) 16 : cluster [INF] Manager daemon smithi050.nbzhun is now available 2023-12-25T20:06:32.720 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:32 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:06:32.343846+0000 mon.smithi050 (mon.0) 17 : audit [INF] from='mgr.14100 172.21.15.50:0/269848152' entity='mgr.smithi050.nbzhun' 2023-12-25T20:06:32.720 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:32 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:06:32.348655+0000 mon.smithi050 (mon.0) 18 : audit [INF] from='mgr.14100 172.21.15.50:0/269848152' entity='mgr.smithi050.nbzhun' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi050.nbzhun/mirror_snapshot_schedule"}]: dispatch 2023-12-25T20:06:32.720 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:32 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:06:32.349818+0000 mon.smithi050 (mon.0) 19 : audit [INF] from='mgr.14100 172.21.15.50:0/269848152' entity='mgr.smithi050.nbzhun' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi050.nbzhun/trash_purge_schedule"}]: dispatch 2023-12-25T20:06:32.720 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:32 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:06:32.354898+0000 mon.smithi050 (mon.0) 20 : audit [INF] from='mgr.14100 172.21.15.50:0/269848152' entity='mgr.smithi050.nbzhun' 2023-12-25T20:06:32.720 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:32 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:06:32.356586+0000 mon.smithi050 (mon.0) 21 : audit [INF] from='mgr.14100 172.21.15.50:0/269848152' entity='mgr.smithi050.nbzhun' 2023-12-25T20:06:32.870 INFO:teuthology.orchestra.run.smithi050.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi050 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/02c8f25c-a361-11ee-95a5-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmps7kf4jlu:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpu2g3lwcx:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2023-12-25T20:06:33.272 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:33.272+0000 7f3796048700 1 Processor -- start 2023-12-25T20:06:33.274 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:33.272+0000 7f3796048700 1 -- start start 2023-12-25T20:06:33.274 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:33.272+0000 7f3796048700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f37900fd1f0 0x7f37900fd610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-25T20:06:33.274 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:33.272+0000 7f3796048700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37900fdb00 con 0x7f37900fd1f0 2023-12-25T20:06:33.274 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:33.272+0000 7f3795046700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f37900fd1f0 0x7f37900fd610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:06:33.274 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:33.272+0000 7f3795046700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f37900fd1f0 0x7f37900fd610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:44018/0 (socket says 172.21.15.50:44018) 2023-12-25T20:06:33.274 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:33.272+0000 7f3795046700 1 -- 172.21.15.50:0/3711190618 learned_addr learned my addr 172.21.15.50:0/3711190618 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-25T20:06:33.274 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:33.273+0000 7f3795046700 1 -- 172.21.15.50:0/3711190618 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37900fdc40 con 0x7f37900fd1f0 2023-12-25T20:06:33.274 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:33.273+0000 7f3795046700 1 --2- 172.21.15.50:0/3711190618 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f37900fd1f0 0x7f37900fd610 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f378c00b5f0 tx=0x7f378c006ee0).ready entity=mon.0 client_cookie=1a925e189479b0a server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:06:33.274 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:33.273+0000 7f377ffff700 1 -- 172.21.15.50:0/3711190618 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f378c00d600 con 0x7f37900fd1f0 2023-12-25T20:06:33.274 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:33.273+0000 7f377ffff700 1 -- 172.21.15.50:0/3711190618 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f378c00dbd0 con 0x7f37900fd1f0 2023-12-25T20:06:33.274 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:33.273+0000 7f377ffff700 1 -- 172.21.15.50:0/3711190618 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f378c011c00 con 0x7f37900fd1f0 2023-12-25T20:06:33.274 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:33.273+0000 7f3796048700 1 -- 172.21.15.50:0/3711190618 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f37900fd1f0 msgr2=0x7f37900fd610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:06:33.277 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:33.273+0000 7f3796048700 1 --2- 172.21.15.50:0/3711190618 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f37900fd1f0 0x7f37900fd610 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f378c00b5f0 tx=0x7f378c006ee0).stop 2023-12-25T20:06:33.277 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:33.273+0000 7f3796048700 1 -- 172.21.15.50:0/3711190618 shutdown_connections 2023-12-25T20:06:33.277 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:33.274+0000 7f3796048700 1 --2- 172.21.15.50:0/3711190618 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f37900fd1f0 0x7f37900fd610 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:06:33.277 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:33.274+0000 7f3796048700 1 -- 172.21.15.50:0/3711190618 >> 172.21.15.50:0/3711190618 conn(0x7f37900f59d0 msgr2=0x7f37900f7df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:06:33.277 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:33.274+0000 7f3796048700 1 -- 172.21.15.50:0/3711190618 shutdown_connections 2023-12-25T20:06:33.277 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:33.274+0000 7f3796048700 1 -- 172.21.15.50:0/3711190618 wait complete. 2023-12-25T20:06:33.277 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:33.274+0000 7f3796048700 1 Processor -- start 2023-12-25T20:06:33.277 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:33.274+0000 7f3796048700 1 -- start start 2023-12-25T20:06:33.278 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:33.274+0000 7f3796048700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f37900fd1f0 0x7f379010a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:06:33.278 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:33.274+0000 7f3796048700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f379010adc0 con 0x7f37900fd1f0 2023-12-25T20:06:33.278 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:33.275+0000 7f3795046700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f37900fd1f0 0x7f379010a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:06:33.278 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:33.275+0000 7f3795046700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f37900fd1f0 0x7f379010a8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:44024/0 (socket says 172.21.15.50:44024) 2023-12-25T20:06:33.278 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:33.275+0000 7f3795046700 1 -- 172.21.15.50:0/1232248913 learned_addr learned my addr 172.21.15.50:0/1232248913 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-25T20:06:33.278 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:33.275+0000 7f3795046700 1 -- 172.21.15.50:0/1232248913 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f378c00b200 con 0x7f37900fd1f0 2023-12-25T20:06:33.278 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:33.275+0000 7f3795046700 1 --2- 172.21.15.50:0/1232248913 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f37900fd1f0 0x7f379010a8d0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f378c00b1d0 tx=0x7f378c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:06:33.278 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:33.275+0000 7f377dffb700 1 -- 172.21.15.50:0/1232248913 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f378c011690 con 0x7f37900fd1f0 2023-12-25T20:06:33.278 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:33.275+0000 7f3796048700 1 -- 172.21.15.50:0/1232248913 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f379010afc0 con 0x7f37900fd1f0 2023-12-25T20:06:33.278 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:33.275+0000 7f377dffb700 1 -- 172.21.15.50:0/1232248913 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f378c01b880 con 0x7f37900fd1f0 2023-12-25T20:06:33.278 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:33.275+0000 7f377dffb700 1 -- 172.21.15.50:0/1232248913 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f378c01a650 con 0x7f37900fd1f0 2023-12-25T20:06:33.278 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:33.275+0000 7f3796048700 1 -- 172.21.15.50:0/1232248913 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f379010b3d0 con 0x7f37900fd1f0 2023-12-25T20:06:33.278 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:33.276+0000 7f377dffb700 1 -- 172.21.15.50:0/1232248913 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 2) v1 ==== 35959+0+0 (secure 0 0 0) 0x7f378c00f070 con 0x7f37900fd1f0 2023-12-25T20:06:33.278 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:33.276+0000 7f377dffb700 1 -- 172.21.15.50:0/1232248913 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f378c040e30 con 0x7f37900fd1f0 2023-12-25T20:06:33.278 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:33.277+0000 7f3796048700 1 -- 172.21.15.50:0/1232248913 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3790059100 con 0x7f37900fd1f0 2023-12-25T20:06:33.279 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:33.280+0000 7f377dffb700 1 -- 172.21.15.50:0/1232248913 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f378c016070 con 0x7f37900fd1f0 2023-12-25T20:06:33.322 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:33.322+0000 7f377dffb700 1 -- 172.21.15.50:0/1232248913 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mgrmap(e 3) v1 ==== 36029+0+0 (secure 0 0 0) 0x7f378c01a870 con 0x7f37900fd1f0 2023-12-25T20:06:33.322 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:33.322+0000 7f377dffb700 1 --2- 172.21.15.50:0/1232248913 >> [v2:172.21.15.50:6800/1929648594,v1:172.21.15.50:6801/1929648594] conn(0x7f3778030d90 0x7f3778033180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:06:33.323 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:33.322+0000 7f3794845700 1 --2- 172.21.15.50:0/1232248913 >> [v2:172.21.15.50:6800/1929648594,v1:172.21.15.50:6801/1929648594] conn(0x7f3778030d90 0x7f3778033180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:06:33.323 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:33.323+0000 7f3794845700 1 --2- 172.21.15.50:0/1232248913 >> [v2:172.21.15.50:6800/1929648594,v1:172.21.15.50:6801/1929648594] conn(0x7f3778030d90 0x7f3778033180 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7f37800096a0 tx=0x7f3780006b40).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:06:33.358 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:33.351+0000 7f3796048700 1 -- 172.21.15.50:0/1232248913 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f3790045ad0 con 0x7f37900fd1f0 2023-12-25T20:06:33.358 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:33.352+0000 7f377dffb700 1 -- 172.21.15.50:0/1232248913 <== mon.0 v2:172.21.15.50:3300/0 8 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1229 (secure 0 0 0) 0x7f378c019c00 con 0x7f37900fd1f0 2023-12-25T20:06:33.358 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:33.358 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: { 2023-12-25T20:06:33.358 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "fsid": "02c8f25c-a361-11ee-95a5-87774f69a715", 2023-12-25T20:06:33.358 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "health": { 2023-12-25T20:06:33.358 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2023-12-25T20:06:33.359 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "checks": {}, 2023-12-25T20:06:33.359 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "mutes": [] 2023-12-25T20:06:33.359 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: }, 2023-12-25T20:06:33.359 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "election_epoch": 5, 2023-12-25T20:06:33.359 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "quorum": [ 2023-12-25T20:06:33.359 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 0 2023-12-25T20:06:33.359 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: ], 2023-12-25T20:06:33.359 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "quorum_names": [ 2023-12-25T20:06:33.360 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "smithi050" 2023-12-25T20:06:33.360 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: ], 2023-12-25T20:06:33.360 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "quorum_age": 7, 2023-12-25T20:06:33.360 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "monmap": { 2023-12-25T20:06:33.360 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "epoch": 1, 2023-12-25T20:06:33.360 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2023-12-25T20:06:33.360 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "num_mons": 1 2023-12-25T20:06:33.361 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: }, 2023-12-25T20:06:33.361 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "osdmap": { 2023-12-25T20:06:33.361 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "epoch": 1, 2023-12-25T20:06:33.361 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "num_osds": 0, 2023-12-25T20:06:33.361 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "num_up_osds": 0, 2023-12-25T20:06:33.361 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "osd_up_since": 0, 2023-12-25T20:06:33.361 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "num_in_osds": 0, 2023-12-25T20:06:33.361 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "osd_in_since": 0, 2023-12-25T20:06:33.361 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2023-12-25T20:06:33.362 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: }, 2023-12-25T20:06:33.362 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "pgmap": { 2023-12-25T20:06:33.362 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "pgs_by_state": [], 2023-12-25T20:06:33.362 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "num_pgs": 0, 2023-12-25T20:06:33.362 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "num_pools": 0, 2023-12-25T20:06:33.362 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "num_objects": 0, 2023-12-25T20:06:33.362 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "data_bytes": 0, 2023-12-25T20:06:33.362 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "bytes_used": 0, 2023-12-25T20:06:33.362 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "bytes_avail": 0, 2023-12-25T20:06:33.362 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "bytes_total": 0 2023-12-25T20:06:33.362 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: }, 2023-12-25T20:06:33.363 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "fsmap": { 2023-12-25T20:06:33.363 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "epoch": 1, 2023-12-25T20:06:33.363 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "by_rank": [], 2023-12-25T20:06:33.363 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "up:standby": 0 2023-12-25T20:06:33.363 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: }, 2023-12-25T20:06:33.363 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "mgrmap": { 2023-12-25T20:06:33.363 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "available": true, 2023-12-25T20:06:33.364 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "num_standbys": 0, 2023-12-25T20:06:33.364 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "modules": [ 2023-12-25T20:06:33.364 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "iostat", 2023-12-25T20:06:33.364 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "restful" 2023-12-25T20:06:33.364 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: ], 2023-12-25T20:06:33.364 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "services": {} 2023-12-25T20:06:33.364 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: }, 2023-12-25T20:06:33.365 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "servicemap": { 2023-12-25T20:06:33.365 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "epoch": 1, 2023-12-25T20:06:33.365 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "modified": "2023-12-25T20:06:22.287710+0000", 2023-12-25T20:06:33.365 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "services": {} 2023-12-25T20:06:33.365 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: }, 2023-12-25T20:06:33.365 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "progress_events": {} 2023-12-25T20:06:33.365 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: } 2023-12-25T20:06:33.366 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:33.353+0000 7f3796048700 1 -- 172.21.15.50:0/1232248913 >> [v2:172.21.15.50:6800/1929648594,v1:172.21.15.50:6801/1929648594] conn(0x7f3778030d90 msgr2=0x7f3778033180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:06:33.366 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:33.353+0000 7f3796048700 1 --2- 172.21.15.50:0/1232248913 >> [v2:172.21.15.50:6800/1929648594,v1:172.21.15.50:6801/1929648594] conn(0x7f3778030d90 0x7f3778033180 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7f37800096a0 tx=0x7f3780006b40).stop 2023-12-25T20:06:33.366 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:33.353+0000 7f3796048700 1 -- 172.21.15.50:0/1232248913 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f37900fd1f0 msgr2=0x7f379010a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:06:33.366 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:33.353+0000 7f3796048700 1 --2- 172.21.15.50:0/1232248913 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f37900fd1f0 0x7f379010a8d0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f378c00b1d0 tx=0x7f378c006ee0).stop 2023-12-25T20:06:33.366 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:33.353+0000 7f3796048700 1 -- 172.21.15.50:0/1232248913 shutdown_connections 2023-12-25T20:06:33.366 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:33.353+0000 7f3796048700 1 --2- 172.21.15.50:0/1232248913 >> [v2:172.21.15.50:6800/1929648594,v1:172.21.15.50:6801/1929648594] conn(0x7f3778030d90 0x7f3778033180 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:06:33.366 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:33.353+0000 7f3796048700 1 --2- 172.21.15.50:0/1232248913 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f37900fd1f0 0x7f379010a8d0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:06:33.366 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:33.353+0000 7f3796048700 1 -- 172.21.15.50:0/1232248913 >> 172.21.15.50:0/1232248913 conn(0x7f37900f59d0 msgr2=0x7f37900ff140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:06:33.367 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:33.353+0000 7f3796048700 1 -- 172.21.15.50:0/1232248913 shutdown_connections 2023-12-25T20:06:33.367 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:33.353+0000 7f3796048700 1 -- 172.21.15.50:0/1232248913 wait complete. 2023-12-25T20:06:33.692 INFO:teuthology.orchestra.run.smithi050.stderr:mgr is available 2023-12-25T20:06:33.692 INFO:teuthology.orchestra.run.smithi050.stderr:Enabling cephadm module... 2023-12-25T20:06:33.692 INFO:teuthology.orchestra.run.smithi050.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi050 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/02c8f25c-a361-11ee-95a5-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmps7kf4jlu:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpu2g3lwcx:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable cephadm 2023-12-25T20:06:34.041 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:34.040+0000 7f267c83a700 1 Processor -- start 2023-12-25T20:06:34.041 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:34.040+0000 7f267c83a700 1 -- start start 2023-12-25T20:06:34.041 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:34.040+0000 7f267c83a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2678070390 0x7f2678100d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-25T20:06:34.041 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:34.040+0000 7f267c83a700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26780fc770 con 0x7f2678070390 2023-12-25T20:06:34.042 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:34.040+0000 7f267659c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2678070390 0x7f2678100d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:06:34.042 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:34.041+0000 7f267659c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2678070390 0x7f2678100d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:44028/0 (socket says 172.21.15.50:44028) 2023-12-25T20:06:34.042 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:34.041+0000 7f267659c700 1 -- 172.21.15.50:0/2436075257 learned_addr learned my addr 172.21.15.50:0/2436075257 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-25T20:06:34.042 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:34.041+0000 7f267659c700 1 -- 172.21.15.50:0/2436075257 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2678101270 con 0x7f2678070390 2023-12-25T20:06:34.042 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:34.041+0000 7f267659c700 1 --2- 172.21.15.50:0/2436075257 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2678070390 0x7f2678100d80 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f266c00b5f0 tx=0x7f266c006ee0).ready entity=mon.0 client_cookie=d762ba74c7f7323e server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:06:34.042 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:34.041+0000 7f267559a700 1 -- 172.21.15.50:0/2436075257 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f266c00d600 con 0x7f2678070390 2023-12-25T20:06:34.042 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:34.041+0000 7f267559a700 1 -- 172.21.15.50:0/2436075257 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f266c00dbd0 con 0x7f2678070390 2023-12-25T20:06:34.042 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:34.041+0000 7f267559a700 1 -- 172.21.15.50:0/2436075257 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f266c011c00 con 0x7f2678070390 2023-12-25T20:06:34.042 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:34.042+0000 7f267c83a700 1 -- 172.21.15.50:0/2436075257 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2678070390 msgr2=0x7f2678100d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:06:34.042 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:34.042+0000 7f267c83a700 1 --2- 172.21.15.50:0/2436075257 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2678070390 0x7f2678100d80 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f266c00b5f0 tx=0x7f266c006ee0).stop 2023-12-25T20:06:34.045 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:34.042+0000 7f267c83a700 1 -- 172.21.15.50:0/2436075257 shutdown_connections 2023-12-25T20:06:34.045 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:34.042+0000 7f267c83a700 1 --2- 172.21.15.50:0/2436075257 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2678070390 0x7f2678100d80 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:06:34.045 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:34.042+0000 7f267c83a700 1 -- 172.21.15.50:0/2436075257 >> 172.21.15.50:0/2436075257 conn(0x7f26780f63c0 msgr2=0x7f26780f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:06:34.045 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:34.042+0000 7f267c83a700 1 -- 172.21.15.50:0/2436075257 shutdown_connections 2023-12-25T20:06:34.045 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:34.042+0000 7f267c83a700 1 -- 172.21.15.50:0/2436075257 wait complete. 2023-12-25T20:06:34.045 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:34.043+0000 7f267c83a700 1 Processor -- start 2023-12-25T20:06:34.045 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:34.043+0000 7f267c83a700 1 -- start start 2023-12-25T20:06:34.045 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:34.043+0000 7f267c83a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2678070390 0x7f267810cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:06:34.045 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:34.043+0000 7f267c83a700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f267810d0f0 con 0x7f2678070390 2023-12-25T20:06:34.045 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:34.043+0000 7f267659c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2678070390 0x7f267810cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:06:34.045 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:34.043+0000 7f267659c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2678070390 0x7f267810cc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:44038/0 (socket says 172.21.15.50:44038) 2023-12-25T20:06:34.046 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:34.043+0000 7f267659c700 1 -- 172.21.15.50:0/2555542931 learned_addr learned my addr 172.21.15.50:0/2555542931 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-25T20:06:34.046 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:34.043+0000 7f267659c700 1 -- 172.21.15.50:0/2555542931 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f266c00b200 con 0x7f2678070390 2023-12-25T20:06:34.046 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:34.043+0000 7f267659c700 1 --2- 172.21.15.50:0/2555542931 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2678070390 0x7f267810cc00 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f266c00b190 tx=0x7f266c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:06:34.046 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:34.044+0000 7f2666ffd700 1 -- 172.21.15.50:0/2555542931 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f266c01b410 con 0x7f2678070390 2023-12-25T20:06:34.046 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:34.044+0000 7f2666ffd700 1 -- 172.21.15.50:0/2555542931 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f266c01b9e0 con 0x7f2678070390 2023-12-25T20:06:34.046 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:34.044+0000 7f267c83a700 1 -- 172.21.15.50:0/2555542931 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f267810d2f0 con 0x7f2678070390 2023-12-25T20:06:34.046 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:34.044+0000 7f2666ffd700 1 -- 172.21.15.50:0/2555542931 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f266c01a880 con 0x7f2678070390 2023-12-25T20:06:34.047 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:34.044+0000 7f267c83a700 1 -- 172.21.15.50:0/2555542931 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f267810d700 con 0x7f2678070390 2023-12-25T20:06:34.047 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:34.044+0000 7f2666ffd700 1 -- 172.21.15.50:0/2555542931 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 3) v1 ==== 36029+0+0 (secure 0 0 0) 0x7f266c00f070 con 0x7f2678070390 2023-12-25T20:06:34.047 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:34.045+0000 7f2666ffd700 1 --2- 172.21.15.50:0/2555542931 >> [v2:172.21.15.50:6800/1929648594,v1:172.21.15.50:6801/1929648594] conn(0x7f265c02fc50 0x7f265c032110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:06:34.047 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:34.045+0000 7f2666ffd700 1 -- 172.21.15.50:0/2555542931 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f266c041070 con 0x7f2678070390 2023-12-25T20:06:34.047 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:34.045+0000 7f2675d9b700 1 --2- 172.21.15.50:0/2555542931 >> [v2:172.21.15.50:6800/1929648594,v1:172.21.15.50:6801/1929648594] conn(0x7f265c02fc50 0x7f265c032110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:06:34.047 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:34.045+0000 7f2675d9b700 1 --2- 172.21.15.50:0/2555542931 >> [v2:172.21.15.50:6800/1929648594,v1:172.21.15.50:6801/1929648594] conn(0x7f265c02fc50 0x7f265c032110 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f26680096a0 tx=0x7f2668006b40).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:06:34.047 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:34.045+0000 7f267c83a700 1 -- 172.21.15.50:0/2555542931 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2678059100 con 0x7f2678070390 2023-12-25T20:06:34.049 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:34.050+0000 7f2666ffd700 1 -- 172.21.15.50:0/2555542931 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f266c016070 con 0x7f2678070390 2023-12-25T20:06:34.245 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:34.245+0000 7f267c83a700 1 -- 172.21.15.50:0/2555542931 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f2678045ad0 con 0x7f2678070390 2023-12-25T20:06:34.323 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:34.324+0000 7f2666ffd700 1 -- 172.21.15.50:0/2555542931 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mgrmap(e 4) v1 ==== 36040+0+0 (secure 0 0 0) 0x7f266c01b570 con 0x7f2678070390 2023-12-25T20:06:34.324 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:34.325+0000 7f2666ffd700 1 -- 172.21.15.50:0/2555542931 <== mon.0 v2:172.21.15.50:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v4) v1 ==== 86+0+0 (secure 0 0 0) 0x7f266c02fe10 con 0x7f2678070390 2023-12-25T20:06:34.327 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:34.326+0000 7f267c83a700 1 -- 172.21.15.50:0/2555542931 >> [v2:172.21.15.50:6800/1929648594,v1:172.21.15.50:6801/1929648594] conn(0x7f265c02fc50 msgr2=0x7f265c032110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:06:34.327 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:34.326+0000 7f267c83a700 1 --2- 172.21.15.50:0/2555542931 >> [v2:172.21.15.50:6800/1929648594,v1:172.21.15.50:6801/1929648594] conn(0x7f265c02fc50 0x7f265c032110 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f26680096a0 tx=0x7f2668006b40).stop 2023-12-25T20:06:34.327 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:34.326+0000 7f267c83a700 1 -- 172.21.15.50:0/2555542931 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2678070390 msgr2=0x7f267810cc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:06:34.327 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:34.326+0000 7f267c83a700 1 --2- 172.21.15.50:0/2555542931 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2678070390 0x7f267810cc00 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f266c00b190 tx=0x7f266c006ee0).stop 2023-12-25T20:06:34.327 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:34.326+0000 7f267c83a700 1 -- 172.21.15.50:0/2555542931 shutdown_connections 2023-12-25T20:06:34.327 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:34.326+0000 7f267c83a700 1 --2- 172.21.15.50:0/2555542931 >> [v2:172.21.15.50:6800/1929648594,v1:172.21.15.50:6801/1929648594] conn(0x7f265c02fc50 0x7f265c032110 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:06:34.327 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:34.326+0000 7f267c83a700 1 --2- 172.21.15.50:0/2555542931 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2678070390 0x7f267810cc00 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:06:34.327 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:34.326+0000 7f267c83a700 1 -- 172.21.15.50:0/2555542931 >> 172.21.15.50:0/2555542931 conn(0x7f26780f63c0 msgr2=0x7f26780f7ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:06:34.327 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:34.327+0000 7f267c83a700 1 -- 172.21.15.50:0/2555542931 shutdown_connections 2023-12-25T20:06:34.327 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:34.327+0000 7f267c83a700 1 -- 172.21.15.50:0/2555542931 wait complete. 2023-12-25T20:06:34.574 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:34 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:06:33.324951+0000 mon.smithi050 (mon.0) 22 : cluster [DBG] mgrmap e3: smithi050.nbzhun(active, since 1.00581s) 2023-12-25T20:06:34.575 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:34 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:06:33.352360+0000 mon.smithi050 (mon.0) 23 : audit [DBG] from='client.? 172.21.15.50:0/1232248913' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-12-25T20:06:34.575 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:34 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:06:34.246150+0000 mon.smithi050 (mon.0) 24 : audit [INF] from='client.? 172.21.15.50:0/2555542931' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2023-12-25T20:06:34.742 INFO:teuthology.orchestra.run.smithi050.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi050 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/02c8f25c-a361-11ee-95a5-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmps7kf4jlu:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpu2g3lwcx:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr stat 2023-12-25T20:06:35.135 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:35.134+0000 7fe5216e0700 1 Processor -- start 2023-12-25T20:06:35.136 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:35.134+0000 7fe5216e0700 1 -- start start 2023-12-25T20:06:35.136 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:35.134+0000 7fe5216e0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe51c0fd2b0 0x7fe51c0fd6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-25T20:06:35.136 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:35.134+0000 7fe5216e0700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe51c0fdbc0 con 0x7fe51c0fd2b0 2023-12-25T20:06:35.136 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:35.135+0000 7fe51affd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe51c0fd2b0 0x7fe51c0fd6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:06:35.136 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:35.135+0000 7fe51affd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe51c0fd2b0 0x7fe51c0fd6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:44078/0 (socket says 172.21.15.50:44078) 2023-12-25T20:06:35.136 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:35.135+0000 7fe51affd700 1 -- 172.21.15.50:0/3377718698 learned_addr learned my addr 172.21.15.50:0/3377718698 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-25T20:06:35.136 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:35.135+0000 7fe51affd700 1 -- 172.21.15.50:0/3377718698 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe51c0fdd00 con 0x7fe51c0fd2b0 2023-12-25T20:06:35.136 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:35.135+0000 7fe51affd700 1 --2- 172.21.15.50:0/3377718698 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe51c0fd2b0 0x7fe51c0fd6d0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7fe50c00b5f0 tx=0x7fe50c006ee0).ready entity=mon.0 client_cookie=c40e6ac50cd1058a server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:06:35.136 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:35.135+0000 7fe519ffb700 1 -- 172.21.15.50:0/3377718698 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe50c00d600 con 0x7fe51c0fd2b0 2023-12-25T20:06:35.136 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:35.135+0000 7fe519ffb700 1 -- 172.21.15.50:0/3377718698 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fe50c00dbd0 con 0x7fe51c0fd2b0 2023-12-25T20:06:35.136 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:35.135+0000 7fe519ffb700 1 -- 172.21.15.50:0/3377718698 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe50c011c00 con 0x7fe51c0fd2b0 2023-12-25T20:06:35.136 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:35.136+0000 7fe5216e0700 1 -- 172.21.15.50:0/3377718698 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe51c0fd2b0 msgr2=0x7fe51c0fd6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:06:35.136 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:35.136+0000 7fe5216e0700 1 --2- 172.21.15.50:0/3377718698 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe51c0fd2b0 0x7fe51c0fd6d0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7fe50c00b5f0 tx=0x7fe50c006ee0).stop 2023-12-25T20:06:35.139 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:35.136+0000 7fe5216e0700 1 -- 172.21.15.50:0/3377718698 shutdown_connections 2023-12-25T20:06:35.139 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:35.136+0000 7fe5216e0700 1 --2- 172.21.15.50:0/3377718698 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe51c0fd2b0 0x7fe51c0fd6d0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:06:35.139 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:35.136+0000 7fe5216e0700 1 -- 172.21.15.50:0/3377718698 >> 172.21.15.50:0/3377718698 conn(0x7fe51c0f5b10 msgr2=0x7fe51c0f7f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:06:35.139 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:35.136+0000 7fe5216e0700 1 -- 172.21.15.50:0/3377718698 shutdown_connections 2023-12-25T20:06:35.139 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:35.136+0000 7fe5216e0700 1 -- 172.21.15.50:0/3377718698 wait complete. 2023-12-25T20:06:35.139 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:35.137+0000 7fe5216e0700 1 Processor -- start 2023-12-25T20:06:35.139 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:35.137+0000 7fe5216e0700 1 -- start start 2023-12-25T20:06:35.140 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:35.137+0000 7fe5216e0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe51c0fd2b0 0x7fe51c10cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:06:35.140 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:35.137+0000 7fe5216e0700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe51c10d100 con 0x7fe51c0fd2b0 2023-12-25T20:06:35.140 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:35.137+0000 7fe51affd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe51c0fd2b0 0x7fe51c10cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:06:35.140 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:35.137+0000 7fe51affd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe51c0fd2b0 0x7fe51c10cc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:44080/0 (socket says 172.21.15.50:44080) 2023-12-25T20:06:35.140 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:35.137+0000 7fe51affd700 1 -- 172.21.15.50:0/41262331 learned_addr learned my addr 172.21.15.50:0/41262331 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-25T20:06:35.140 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:35.137+0000 7fe51affd700 1 -- 172.21.15.50:0/41262331 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe50c00b200 con 0x7fe51c0fd2b0 2023-12-25T20:06:35.140 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:35.137+0000 7fe51affd700 1 --2- 172.21.15.50:0/41262331 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe51c0fd2b0 0x7fe51c10cc10 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7fe50c002f40 tx=0x7fe50c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:06:35.140 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:35.138+0000 7fe5037fe700 1 -- 172.21.15.50:0/41262331 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe50c01b410 con 0x7fe51c0fd2b0 2023-12-25T20:06:35.140 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:35.138+0000 7fe5216e0700 1 -- 172.21.15.50:0/41262331 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe51c10d300 con 0x7fe51c0fd2b0 2023-12-25T20:06:35.140 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:35.138+0000 7fe5037fe700 1 -- 172.21.15.50:0/41262331 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fe50c01b9e0 con 0x7fe51c0fd2b0 2023-12-25T20:06:35.140 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:35.138+0000 7fe5037fe700 1 -- 172.21.15.50:0/41262331 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe50c01a8b0 con 0x7fe51c0fd2b0 2023-12-25T20:06:35.140 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:35.138+0000 7fe5216e0700 1 -- 172.21.15.50:0/41262331 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe51c10d710 con 0x7fe51c0fd2b0 2023-12-25T20:06:35.140 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:35.138+0000 7fe5037fe700 1 -- 172.21.15.50:0/41262331 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 4) v1 ==== 36040+0+0 (secure 0 0 0) 0x7fe50c00f070 con 0x7fe51c0fd2b0 2023-12-25T20:06:35.141 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:35.138+0000 7fe5037fe700 1 --2- 172.21.15.50:0/41262331 >> [v2:172.21.15.50:6800/1929648594,v1:172.21.15.50:6801/1929648594] conn(0x7fe50802fcc0 0x7fe508032180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:06:35.141 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:35.139+0000 7fe5037fe700 1 -- 172.21.15.50:0/41262331 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fe50c041fe0 con 0x7fe51c0fd2b0 2023-12-25T20:06:35.141 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:35.139+0000 7fe51a7fc700 1 -- 172.21.15.50:0/41262331 >> [v2:172.21.15.50:6800/1929648594,v1:172.21.15.50:6801/1929648594] conn(0x7fe50802fcc0 msgr2=0x7fe508032180 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:6800/1929648594 2023-12-25T20:06:35.141 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:35.139+0000 7fe51a7fc700 1 --2- 172.21.15.50:0/41262331 >> [v2:172.21.15.50:6800/1929648594,v1:172.21.15.50:6801/1929648594] conn(0x7fe50802fcc0 0x7fe508032180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-12-25T20:06:35.141 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:35.139+0000 7fe5216e0700 1 -- 172.21.15.50:0/41262331 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe51c059100 con 0x7fe51c0fd2b0 2023-12-25T20:06:35.143 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:35.143+0000 7fe5037fe700 1 -- 172.21.15.50:0/41262331 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe50c010360 con 0x7fe51c0fd2b0 2023-12-25T20:06:35.327 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:35.327+0000 7fe5037fe700 1 -- 172.21.15.50:0/41262331 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mgrmap(e 5) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fe50c025980 con 0x7fe51c0fd2b0 2023-12-25T20:06:35.340 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:35.340+0000 7fe51a7fc700 1 -- 172.21.15.50:0/41262331 >> [v2:172.21.15.50:6800/1929648594,v1:172.21.15.50:6801/1929648594] conn(0x7fe50802fcc0 msgr2=0x7fe508032180 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:6800/1929648594 2023-12-25T20:06:35.340 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:35.340+0000 7fe51a7fc700 1 --2- 172.21.15.50:0/41262331 >> [v2:172.21.15.50:6800/1929648594,v1:172.21.15.50:6801/1929648594] conn(0x7fe50802fcc0 0x7fe508032180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-12-25T20:06:35.343 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:35.343+0000 7fe5216e0700 1 -- 172.21.15.50:0/41262331 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fe51c045ad0 con 0x7fe51c0fd2b0 2023-12-25T20:06:35.344 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:35.343+0000 7fe5037fe700 1 -- 172.21.15.50:0/41262331 <== mon.0 v2:172.21.15.50:3300/0 8 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+103 (secure 0 0 0) 0x7fe50c022ca0 con 0x7fe51c0fd2b0 2023-12-25T20:06:35.344 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: { 2023-12-25T20:06:35.345 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "epoch": 5, 2023-12-25T20:06:35.345 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "available": true, 2023-12-25T20:06:35.345 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "active_name": "smithi050.nbzhun", 2023-12-25T20:06:35.345 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "num_standby": 0 2023-12-25T20:06:35.345 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: } 2023-12-25T20:06:35.347 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:35.345+0000 7fe5216e0700 1 -- 172.21.15.50:0/41262331 >> [v2:172.21.15.50:6800/1929648594,v1:172.21.15.50:6801/1929648594] conn(0x7fe50802fcc0 msgr2=0x7fe508032180 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-25T20:06:35.347 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:35.345+0000 7fe5216e0700 1 --2- 172.21.15.50:0/41262331 >> [v2:172.21.15.50:6800/1929648594,v1:172.21.15.50:6801/1929648594] conn(0x7fe50802fcc0 0x7fe508032180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-12-25T20:06:35.347 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:35.345+0000 7fe5216e0700 1 -- 172.21.15.50:0/41262331 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe51c0fd2b0 msgr2=0x7fe51c10cc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:06:35.347 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:35.345+0000 7fe5216e0700 1 --2- 172.21.15.50:0/41262331 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe51c0fd2b0 0x7fe51c10cc10 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7fe50c002f40 tx=0x7fe50c006ee0).stop 2023-12-25T20:06:35.347 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:35.345+0000 7fe5216e0700 1 -- 172.21.15.50:0/41262331 shutdown_connections 2023-12-25T20:06:35.347 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:35.345+0000 7fe5216e0700 1 --2- 172.21.15.50:0/41262331 >> [v2:172.21.15.50:6800/1929648594,v1:172.21.15.50:6801/1929648594] conn(0x7fe50802fcc0 0x7fe508032180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-12-25T20:06:35.348 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:35.345+0000 7fe5216e0700 1 --2- 172.21.15.50:0/41262331 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe51c0fd2b0 0x7fe51c10cc10 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:06:35.348 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:35.345+0000 7fe5216e0700 1 -- 172.21.15.50:0/41262331 >> 172.21.15.50:0/41262331 conn(0x7fe51c0f5b10 msgr2=0x7fe51c0ff2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:06:35.348 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:35.345+0000 7fe5216e0700 1 -- 172.21.15.50:0/41262331 shutdown_connections 2023-12-25T20:06:35.348 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:35.345+0000 7fe5216e0700 1 -- 172.21.15.50:0/41262331 wait complete. 2023-12-25T20:06:35.692 INFO:teuthology.orchestra.run.smithi050.stderr:Waiting for the mgr to restart... 2023-12-25T20:06:35.692 INFO:teuthology.orchestra.run.smithi050.stderr:Waiting for mgr epoch 5... 2023-12-25T20:06:35.692 INFO:teuthology.orchestra.run.smithi050.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi050 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/02c8f25c-a361-11ee-95a5-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmps7kf4jlu:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpu2g3lwcx:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 tell mgr mgr_status 2023-12-25T20:06:35.694 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:35 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:06:34.325265+0000 mon.smithi050 (mon.0) 25 : audit [INF] from='client.? 172.21.15.50:0/2555542931' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2023-12-25T20:06:35.694 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:35 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:06:34.325315+0000 mon.smithi050 (mon.0) 26 : cluster [DBG] mgrmap e4: smithi050.nbzhun(active, since 2s) 2023-12-25T20:06:36.069 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:36.068+0000 7fbb27778700 1 Processor -- start 2023-12-25T20:06:36.069 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:36.068+0000 7fbb27778700 1 -- start start 2023-12-25T20:06:36.069 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:36.069+0000 7fbb27778700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbb20070430 0x7fbb200fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-25T20:06:36.069 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:36.069+0000 7fbb27778700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb200fedd0 con 0x7fbb20070430 2023-12-25T20:06:36.069 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:36.069+0000 7fbb25514700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbb20070430 0x7fbb200fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:06:36.070 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:36.069+0000 7fbb25514700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbb20070430 0x7fbb200fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:44092/0 (socket says 172.21.15.50:44092) 2023-12-25T20:06:36.070 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:36.069+0000 7fbb25514700 1 -- 172.21.15.50:0/1498431613 learned_addr learned my addr 172.21.15.50:0/1498431613 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-25T20:06:36.070 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:36.069+0000 7fbb25514700 1 -- 172.21.15.50:0/1498431613 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb200fef10 con 0x7fbb20070430 2023-12-25T20:06:36.071 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:36.069+0000 7fbb25514700 1 --2- 172.21.15.50:0/1498431613 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbb20070430 0x7fbb200fe8e0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7fbb1c00b5f0 tx=0x7fbb1c006ee0).ready entity=mon.0 client_cookie=ec8c1ec114416b90 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:06:36.071 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:36.069+0000 7fbb0ffff700 1 -- 172.21.15.50:0/1498431613 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbb1c00d600 con 0x7fbb20070430 2023-12-25T20:06:36.071 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:36.069+0000 7fbb0ffff700 1 -- 172.21.15.50:0/1498431613 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fbb1c00dbd0 con 0x7fbb20070430 2023-12-25T20:06:36.071 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:36.069+0000 7fbb0ffff700 1 -- 172.21.15.50:0/1498431613 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbb1c011c00 con 0x7fbb20070430 2023-12-25T20:06:36.071 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:36.070+0000 7fbb27778700 1 -- 172.21.15.50:0/1498431613 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbb20070430 msgr2=0x7fbb200fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:06:36.071 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:36.070+0000 7fbb27778700 1 --2- 172.21.15.50:0/1498431613 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbb20070430 0x7fbb200fe8e0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7fbb1c00b5f0 tx=0x7fbb1c006ee0).stop 2023-12-25T20:06:36.071 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:36.071+0000 7fbb27778700 1 -- 172.21.15.50:0/1498431613 shutdown_connections 2023-12-25T20:06:36.071 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:36.071+0000 7fbb27778700 1 --2- 172.21.15.50:0/1498431613 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbb20070430 0x7fbb200fe8e0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:06:36.073 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:36.071+0000 7fbb27778700 1 -- 172.21.15.50:0/1498431613 >> 172.21.15.50:0/1498431613 conn(0x7fbb200f63c0 msgr2=0x7fbb200f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:06:36.073 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:36.071+0000 7fbb27778700 1 -- 172.21.15.50:0/1498431613 shutdown_connections 2023-12-25T20:06:36.073 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:36.071+0000 7fbb27778700 1 -- 172.21.15.50:0/1498431613 wait complete. 2023-12-25T20:06:36.073 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:36.071+0000 7fbb27778700 1 Processor -- start 2023-12-25T20:06:36.073 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:36.072+0000 7fbb27778700 1 -- start start 2023-12-25T20:06:36.073 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:36.072+0000 7fbb27778700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbb20070430 0x7fbb200fc360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:06:36.073 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:36.072+0000 7fbb27778700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb200fa7e0 con 0x7fbb20070430 2023-12-25T20:06:36.073 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:36.072+0000 7fbb25514700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbb20070430 0x7fbb200fc360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:06:36.073 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:36.072+0000 7fbb25514700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbb20070430 0x7fbb200fc360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:44096/0 (socket says 172.21.15.50:44096) 2023-12-25T20:06:36.074 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:36.072+0000 7fbb25514700 1 -- 172.21.15.50:0/417591961 learned_addr learned my addr 172.21.15.50:0/417591961 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-25T20:06:36.074 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:36.072+0000 7fbb25514700 1 -- 172.21.15.50:0/417591961 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb1c00b200 con 0x7fbb20070430 2023-12-25T20:06:36.074 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:36.072+0000 7fbb25514700 1 --2- 172.21.15.50:0/417591961 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbb20070430 0x7fbb200fc360 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7fbb1c00b190 tx=0x7fbb1c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:06:36.074 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:36.072+0000 7fbb0dffb700 1 -- 172.21.15.50:0/417591961 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbb1c01b410 con 0x7fbb20070430 2023-12-25T20:06:36.074 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:36.072+0000 7fbb0dffb700 1 -- 172.21.15.50:0/417591961 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fbb1c01b9e0 con 0x7fbb20070430 2023-12-25T20:06:36.074 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:36.072+0000 7fbb27778700 1 -- 172.21.15.50:0/417591961 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb200fa980 con 0x7fbb20070430 2023-12-25T20:06:36.074 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:36.072+0000 7fbb0dffb700 1 -- 172.21.15.50:0/417591961 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbb1c01a880 con 0x7fbb20070430 2023-12-25T20:06:36.074 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:36.073+0000 7fbb27778700 1 -- 172.21.15.50:0/417591961 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb200fad90 con 0x7fbb20070430 2023-12-25T20:06:36.075 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:36.074+0000 7fbb0dffb700 1 -- 172.21.15.50:0/417591961 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fbb1c00f070 con 0x7fbb20070430 2023-12-25T20:06:36.075 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:36.074+0000 7fbb0dffb700 1 --2- 172.21.15.50:0/417591961 >> [v2:172.21.15.50:6800/1929648594,v1:172.21.15.50:6801/1929648594] conn(0x7fbb0802fdb0 0x7fbb08032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:06:36.075 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:36.074+0000 7fbb0dffb700 1 -- 172.21.15.50:0/417591961 --> [v2:172.21.15.50:6800/1929648594,v1:172.21.15.50:6801/1929648594] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbb08032930 con 0x7fbb0802fdb0 2023-12-25T20:06:36.075 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:36.074+0000 7fbb0dffb700 1 -- 172.21.15.50:0/417591961 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fbb1c0411e0 con 0x7fbb20070430 2023-12-25T20:06:36.075 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:36.074+0000 7fbb24d13700 1 -- 172.21.15.50:0/417591961 >> [v2:172.21.15.50:6800/1929648594,v1:172.21.15.50:6801/1929648594] conn(0x7fbb0802fdb0 msgr2=0x7fbb08032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:6800/1929648594 2023-12-25T20:06:36.075 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:36.074+0000 7fbb24d13700 1 --2- 172.21.15.50:0/417591961 >> [v2:172.21.15.50:6800/1929648594,v1:172.21.15.50:6801/1929648594] conn(0x7fbb0802fdb0 0x7fbb08032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-12-25T20:06:36.275 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:36.275+0000 7fbb24d13700 1 -- 172.21.15.50:0/417591961 >> [v2:172.21.15.50:6800/1929648594,v1:172.21.15.50:6801/1929648594] conn(0x7fbb0802fdb0 msgr2=0x7fbb08032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:6800/1929648594 2023-12-25T20:06:36.275 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:36.275+0000 7fbb24d13700 1 --2- 172.21.15.50:0/417591961 >> [v2:172.21.15.50:6800/1929648594,v1:172.21.15.50:6801/1929648594] conn(0x7fbb0802fdb0 0x7fbb08032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-12-25T20:06:36.676 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:36.676+0000 7fbb24d13700 1 -- 172.21.15.50:0/417591961 >> [v2:172.21.15.50:6800/1929648594,v1:172.21.15.50:6801/1929648594] conn(0x7fbb0802fdb0 msgr2=0x7fbb08032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:6800/1929648594 2023-12-25T20:06:36.676 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:36.676+0000 7fbb24d13700 1 --2- 172.21.15.50:0/417591961 >> [v2:172.21.15.50:6800/1929648594,v1:172.21.15.50:6801/1929648594] conn(0x7fbb0802fdb0 0x7fbb08032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-12-25T20:06:36.718 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:36 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:06:35.329407+0000 mon.smithi050 (mon.0) 27 : cluster [DBG] mgrmap e5: smithi050.nbzhun(active, since 3s) 2023-12-25T20:06:36.719 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:36 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:06:35.343963+0000 mon.smithi050 (mon.0) 28 : audit [DBG] from='client.? 172.21.15.50:0/41262331' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-12-25T20:06:37.477 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:37.477+0000 7fbb24d13700 1 -- 172.21.15.50:0/417591961 >> [v2:172.21.15.50:6800/1929648594,v1:172.21.15.50:6801/1929648594] conn(0x7fbb0802fdb0 msgr2=0x7fbb08032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:6800/1929648594 2023-12-25T20:06:37.477 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:37.477+0000 7fbb24d13700 1 --2- 172.21.15.50:0/417591961 >> [v2:172.21.15.50:6800/1929648594,v1:172.21.15.50:6801/1929648594] conn(0x7fbb0802fdb0 0x7fbb08032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-12-25T20:06:39.078 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:39.077+0000 7fbb24d13700 1 -- 172.21.15.50:0/417591961 >> [v2:172.21.15.50:6800/1929648594,v1:172.21.15.50:6801/1929648594] conn(0x7fbb0802fdb0 msgr2=0x7fbb08032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:6800/1929648594 2023-12-25T20:06:39.078 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:39.077+0000 7fbb24d13700 1 --2- 172.21.15.50:0/417591961 >> [v2:172.21.15.50:6800/1929648594,v1:172.21.15.50:6801/1929648594] conn(0x7fbb0802fdb0 0x7fbb08032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 3.200000 2023-12-25T20:06:39.570 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:39.569+0000 7fbb0dffb700 1 -- 172.21.15.50:0/417591961 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mgrmap(e 6) v1 ==== 35970+0+0 (secure 0 0 0) 0x7fbb1c01b570 con 0x7fbb20070430 2023-12-25T20:06:39.570 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:39.569+0000 7fbb0dffb700 1 -- 172.21.15.50:0/417591961 >> [v2:172.21.15.50:6800/1929648594,v1:172.21.15.50:6801/1929648594] conn(0x7fbb0802fdb0 msgr2=0x7fbb08032270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-25T20:06:39.570 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:39.569+0000 7fbb0dffb700 1 --2- 172.21.15.50:0/417591961 >> [v2:172.21.15.50:6800/1929648594,v1:172.21.15.50:6801/1929648594] conn(0x7fbb0802fdb0 0x7fbb08032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-12-25T20:06:40.572 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:40.570+0000 7fbb0dffb700 1 -- 172.21.15.50:0/417591961 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mgrmap(e 7) v1 ==== 36040+0+0 (secure 0 0 0) 0x7fbb1c042a90 con 0x7fbb20070430 2023-12-25T20:06:40.572 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:40.570+0000 7fbb0dffb700 1 --2- 172.21.15.50:0/417591961 >> [v2:172.21.15.50:6800/4046643232,v1:172.21.15.50:6801/4046643232] conn(0x7fbb08033880 0x7fbb08035c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:06:40.572 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:40.570+0000 7fbb0dffb700 1 -- 172.21.15.50:0/417591961 --> [v2:172.21.15.50:6800/4046643232,v1:172.21.15.50:6801/4046643232] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbb08032930 con 0x7fbb08033880 2023-12-25T20:06:40.573 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:40.571+0000 7fbb24d13700 1 --2- 172.21.15.50:0/417591961 >> [v2:172.21.15.50:6800/4046643232,v1:172.21.15.50:6801/4046643232] conn(0x7fbb08033880 0x7fbb08035c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:06:40.573 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:40.572+0000 7fbb24d13700 1 --2- 172.21.15.50:0/417591961 >> [v2:172.21.15.50:6800/4046643232,v1:172.21.15.50:6801/4046643232] conn(0x7fbb08033880 0x7fbb08035c70 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7fbb10006fd0 tx=0x7fbb10006d70).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:06:40.573 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:40.572+0000 7fbb0dffb700 1 -- 172.21.15.50:0/417591961 <== mgr.14114 v2:172.21.15.50:6800/4046643232 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (secure 0 0 0) 0x7fbb08032930 con 0x7fbb08033880 2023-12-25T20:06:40.581 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:40.578+0000 7fbb27778700 1 -- 172.21.15.50:0/417591961 --> [v2:172.21.15.50:6800/4046643232,v1:172.21.15.50:6801/4046643232] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fbb20059100 con 0x7fbb08033880 2023-12-25T20:06:40.582 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:40.579+0000 7fbb0dffb700 1 -- 172.21.15.50:0/417591961 <== mgr.14114 v2:172.21.15.50:6800/4046643232 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fbb20059100 con 0x7fbb08033880 2023-12-25T20:06:40.582 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: { 2023-12-25T20:06:40.582 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "mgrmap_epoch": 7, 2023-12-25T20:06:40.582 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "initialized": true 2023-12-25T20:06:40.582 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: } 2023-12-25T20:06:40.582 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:40.579+0000 7fbb27778700 1 -- 172.21.15.50:0/417591961 >> [v2:172.21.15.50:6800/4046643232,v1:172.21.15.50:6801/4046643232] conn(0x7fbb08033880 msgr2=0x7fbb08035c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:06:40.582 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:40.579+0000 7fbb27778700 1 --2- 172.21.15.50:0/417591961 >> [v2:172.21.15.50:6800/4046643232,v1:172.21.15.50:6801/4046643232] conn(0x7fbb08033880 0x7fbb08035c70 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7fbb10006fd0 tx=0x7fbb10006d70).stop 2023-12-25T20:06:40.582 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:40.579+0000 7fbb27778700 1 -- 172.21.15.50:0/417591961 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbb20070430 msgr2=0x7fbb200fc360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:06:40.582 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:40.579+0000 7fbb27778700 1 --2- 172.21.15.50:0/417591961 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbb20070430 0x7fbb200fc360 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7fbb1c00b190 tx=0x7fbb1c006ee0).stop 2023-12-25T20:06:40.582 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:40.579+0000 7fbb27778700 1 -- 172.21.15.50:0/417591961 shutdown_connections 2023-12-25T20:06:40.582 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:40.579+0000 7fbb27778700 1 --2- 172.21.15.50:0/417591961 >> [v2:172.21.15.50:6800/4046643232,v1:172.21.15.50:6801/4046643232] conn(0x7fbb08033880 0x7fbb08035c70 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:06:40.582 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:40.579+0000 7fbb27778700 1 --2- 172.21.15.50:0/417591961 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbb20070430 0x7fbb200fc360 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:06:40.582 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:40.579+0000 7fbb27778700 1 --2- 172.21.15.50:0/417591961 >> [v2:172.21.15.50:6800/1929648594,v1:172.21.15.50:6801/1929648594] conn(0x7fbb0802fdb0 0x7fbb08032270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-12-25T20:06:40.583 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:40.579+0000 7fbb27778700 1 -- 172.21.15.50:0/417591961 >> 172.21.15.50:0/417591961 conn(0x7fbb200f63c0 msgr2=0x7fbb200f7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:06:40.583 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:40.579+0000 7fbb27778700 1 -- 172.21.15.50:0/417591961 shutdown_connections 2023-12-25T20:06:40.583 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:40.580+0000 7fbb27778700 1 -- 172.21.15.50:0/417591961 wait complete. 2023-12-25T20:06:40.969 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:40 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:06:39.518253+0000 mon.smithi050 (mon.0) 29 : cluster [INF] Active manager daemon smithi050.nbzhun restarted 2023-12-25T20:06:40.969 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:40 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:06:39.518693+0000 mon.smithi050 (mon.0) 30 : cluster [INF] Activating manager daemon smithi050.nbzhun 2023-12-25T20:06:40.969 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:40 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:06:39.520723+0000 mon.smithi050 (mon.0) 31 : cluster [DBG] osdmap e2: 0 total, 0 up, 0 in 2023-12-25T20:06:40.969 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:40 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:06:39.571249+0000 mon.smithi050 (mon.0) 32 : cluster [DBG] mgrmap e6: smithi050.nbzhun(active, starting, since 0.0526344s) 2023-12-25T20:06:40.969 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:40 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:06:39.572139+0000 mon.smithi050 (mon.0) 33 : audit [DBG] from='mgr.14114 172.21.15.50:0/3381094233' entity='mgr.smithi050.nbzhun' cmd=[{"prefix": "mon metadata", "id": "smithi050"}]: dispatch 2023-12-25T20:06:40.969 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:40 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:06:39.572753+0000 mon.smithi050 (mon.0) 34 : audit [DBG] from='mgr.14114 172.21.15.50:0/3381094233' entity='mgr.smithi050.nbzhun' cmd=[{"prefix": "mgr metadata", "who": "smithi050.nbzhun", "id": "smithi050.nbzhun"}]: dispatch 2023-12-25T20:06:40.969 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:40 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:06:39.573490+0000 mon.smithi050 (mon.0) 35 : audit [DBG] from='mgr.14114 172.21.15.50:0/3381094233' entity='mgr.smithi050.nbzhun' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-12-25T20:06:40.969 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:40 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:06:39.573633+0000 mon.smithi050 (mon.0) 36 : audit [DBG] from='mgr.14114 172.21.15.50:0/3381094233' entity='mgr.smithi050.nbzhun' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-12-25T20:06:40.970 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:40 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:06:39.573770+0000 mon.smithi050 (mon.0) 37 : audit [DBG] from='mgr.14114 172.21.15.50:0/3381094233' entity='mgr.smithi050.nbzhun' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-12-25T20:06:40.970 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:40 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:06:39.588068+0000 mon.smithi050 (mon.0) 38 : cluster [INF] Manager daemon smithi050.nbzhun is now available 2023-12-25T20:06:40.970 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:40 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:06:39.602614+0000 mon.smithi050 (mon.0) 39 : audit [INF] from='mgr.14114 172.21.15.50:0/3381094233' entity='mgr.smithi050.nbzhun' 2023-12-25T20:06:40.970 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:40 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:06:39.605146+0000 mon.smithi050 (mon.0) 40 : audit [INF] from='mgr.14114 172.21.15.50:0/3381094233' entity='mgr.smithi050.nbzhun' 2023-12-25T20:06:40.970 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:40 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:06:39.607548+0000 mon.smithi050 (mon.0) 41 : audit [INF] from='mgr.14114 172.21.15.50:0/3381094233' entity='mgr.smithi050.nbzhun' 2023-12-25T20:06:40.970 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:40 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:06:39.705710+0000 mon.smithi050 (mon.0) 42 : audit [INF] from='mgr.14114 172.21.15.50:0/3381094233' entity='mgr.smithi050.nbzhun' 2023-12-25T20:06:40.970 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:40 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:06:39.706399+0000 mon.smithi050 (mon.0) 43 : audit [DBG] from='mgr.14114 172.21.15.50:0/3381094233' entity='mgr.smithi050.nbzhun' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-25T20:06:40.970 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:40 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:06:39.707896+0000 mon.smithi050 (mon.0) 44 : audit [DBG] from='mgr.14114 172.21.15.50:0/3381094233' entity='mgr.smithi050.nbzhun' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-25T20:06:40.970 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:40 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:06:39.709739+0000 mon.smithi050 (mon.0) 45 : audit [DBG] from='mgr.14114 172.21.15.50:0/3381094233' entity='mgr.smithi050.nbzhun' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-25T20:06:40.970 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:40 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:06:39.720727+0000 mon.smithi050 (mon.0) 46 : audit [INF] from='mgr.14114 172.21.15.50:0/3381094233' entity='mgr.smithi050.nbzhun' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi050.nbzhun/mirror_snapshot_schedule"}]: dispatch 2023-12-25T20:06:40.970 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:40 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:06:39.722004+0000 mon.smithi050 (mon.0) 47 : audit [INF] from='mgr.14114 172.21.15.50:0/3381094233' entity='mgr.smithi050.nbzhun' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi050.nbzhun/trash_purge_schedule"}]: dispatch 2023-12-25T20:06:41.166 INFO:teuthology.orchestra.run.smithi050.stderr:mgr epoch 5 is available 2023-12-25T20:06:41.167 INFO:teuthology.orchestra.run.smithi050.stderr:Setting orchestrator backend to cephadm... 2023-12-25T20:06:41.167 INFO:teuthology.orchestra.run.smithi050.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi050 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/02c8f25c-a361-11ee-95a5-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmps7kf4jlu:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpu2g3lwcx:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch set backend cephadm 2023-12-25T20:06:41.518 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:41.515+0000 7ff9ca99f700 1 Processor -- start 2023-12-25T20:06:41.518 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:41.515+0000 7ff9ca99f700 1 -- start start 2023-12-25T20:06:41.518 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:41.515+0000 7ff9ca99f700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff9c40703b0 0x7ff9c4100bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-25T20:06:41.518 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:41.515+0000 7ff9ca99f700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9c40fc5a0 con 0x7ff9c40703b0 2023-12-25T20:06:41.518 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:41.516+0000 7ff9c999d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff9c40703b0 0x7ff9c4100bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:06:41.518 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:41.516+0000 7ff9c999d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff9c40703b0 0x7ff9c4100bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:37592/0 (socket says 172.21.15.50:37592) 2023-12-25T20:06:41.519 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:41.516+0000 7ff9c999d700 1 -- 172.21.15.50:0/1664982258 learned_addr learned my addr 172.21.15.50:0/1664982258 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-25T20:06:41.519 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:41.516+0000 7ff9c999d700 1 -- 172.21.15.50:0/1664982258 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9c41010a0 con 0x7ff9c40703b0 2023-12-25T20:06:41.519 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:41.516+0000 7ff9c999d700 1 --2- 172.21.15.50:0/1664982258 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff9c40703b0 0x7ff9c4100bb0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7ff9b800b5f0 tx=0x7ff9b8006ee0).ready entity=mon.0 client_cookie=69f74e732ea33b2e server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:06:41.519 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:41.516+0000 7ff9c899b700 1 -- 172.21.15.50:0/1664982258 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff9b800d600 con 0x7ff9c40703b0 2023-12-25T20:06:41.519 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:41.516+0000 7ff9c899b700 1 -- 172.21.15.50:0/1664982258 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7ff9b800dbd0 con 0x7ff9c40703b0 2023-12-25T20:06:41.519 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:41.516+0000 7ff9c899b700 1 -- 172.21.15.50:0/1664982258 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff9b8011c00 con 0x7ff9c40703b0 2023-12-25T20:06:41.519 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:41.517+0000 7ff9ca99f700 1 -- 172.21.15.50:0/1664982258 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff9c40703b0 msgr2=0x7ff9c4100bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:06:41.519 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:41.517+0000 7ff9ca99f700 1 --2- 172.21.15.50:0/1664982258 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff9c40703b0 0x7ff9c4100bb0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7ff9b800b5f0 tx=0x7ff9b8006ee0).stop 2023-12-25T20:06:41.519 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:41.517+0000 7ff9ca99f700 1 -- 172.21.15.50:0/1664982258 shutdown_connections 2023-12-25T20:06:41.519 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:41.517+0000 7ff9ca99f700 1 --2- 172.21.15.50:0/1664982258 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff9c40703b0 0x7ff9c4100bb0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:06:41.519 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:41.517+0000 7ff9ca99f700 1 -- 172.21.15.50:0/1664982258 >> 172.21.15.50:0/1664982258 conn(0x7ff9c40f6210 msgr2=0x7ff9c40f8630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:06:41.519 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:41.517+0000 7ff9ca99f700 1 -- 172.21.15.50:0/1664982258 shutdown_connections 2023-12-25T20:06:41.519 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:41.517+0000 7ff9ca99f700 1 -- 172.21.15.50:0/1664982258 wait complete. 2023-12-25T20:06:41.519 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:41.518+0000 7ff9ca99f700 1 Processor -- start 2023-12-25T20:06:41.520 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:41.518+0000 7ff9ca99f700 1 -- start start 2023-12-25T20:06:41.521 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:41.518+0000 7ff9ca99f700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff9c40703b0 0x7ff9c410ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:06:41.522 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:41.518+0000 7ff9ca99f700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9c410cf60 con 0x7ff9c40703b0 2023-12-25T20:06:41.522 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:41.518+0000 7ff9c999d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff9c40703b0 0x7ff9c410ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:06:41.522 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:41.518+0000 7ff9c999d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff9c40703b0 0x7ff9c410ca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:37596/0 (socket says 172.21.15.50:37596) 2023-12-25T20:06:41.522 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:41.518+0000 7ff9c999d700 1 -- 172.21.15.50:0/511719526 learned_addr learned my addr 172.21.15.50:0/511719526 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-25T20:06:41.522 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:41.518+0000 7ff9c999d700 1 -- 172.21.15.50:0/511719526 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9b800b200 con 0x7ff9c40703b0 2023-12-25T20:06:41.522 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:41.518+0000 7ff9c999d700 1 --2- 172.21.15.50:0/511719526 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff9c40703b0 0x7ff9c410ca70 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7ff9b800e040 tx=0x7ff9b8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:06:41.522 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:41.518+0000 7ff9b67fc700 1 -- 172.21.15.50:0/511719526 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff9b800d9f0 con 0x7ff9c40703b0 2023-12-25T20:06:41.522 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:41.519+0000 7ff9b67fc700 1 -- 172.21.15.50:0/511719526 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7ff9b801a9f0 con 0x7ff9c40703b0 2023-12-25T20:06:41.522 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:41.519+0000 7ff9ca99f700 1 -- 172.21.15.50:0/511719526 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9c410d160 con 0x7ff9c40703b0 2023-12-25T20:06:41.522 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:41.519+0000 7ff9b67fc700 1 -- 172.21.15.50:0/511719526 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff9b801b8c0 con 0x7ff9c40703b0 2023-12-25T20:06:41.522 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:41.519+0000 7ff9ca99f700 1 -- 172.21.15.50:0/511719526 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9c410d570 con 0x7ff9c40703b0 2023-12-25T20:06:41.522 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:41.520+0000 7ff9b67fc700 1 -- 172.21.15.50:0/511719526 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 7) v1 ==== 36040+0+0 (secure 0 0 0) 0x7ff9b800f070 con 0x7ff9c40703b0 2023-12-25T20:06:41.522 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:41.520+0000 7ff9b67fc700 1 --2- 172.21.15.50:0/511719526 >> [v2:172.21.15.50:6800/4046643232,v1:172.21.15.50:6801/4046643232] conn(0x7ff9ac02fd10 0x7ff9ac0321d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:06:41.522 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:41.520+0000 7ff9ca99f700 1 -- 172.21.15.50:0/511719526 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9c4045ad0 con 0x7ff9c40703b0 2023-12-25T20:06:41.523 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:41.520+0000 7ff9b67fc700 1 -- 172.21.15.50:0/511719526 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7ff9b80411a0 con 0x7ff9c40703b0 2023-12-25T20:06:41.523 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:41.520+0000 7ff9c919c700 1 --2- 172.21.15.50:0/511719526 >> [v2:172.21.15.50:6800/4046643232,v1:172.21.15.50:6801/4046643232] conn(0x7ff9ac02fd10 0x7ff9ac0321d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:06:41.523 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:41.520+0000 7ff9c919c700 1 --2- 172.21.15.50:0/511719526 >> [v2:172.21.15.50:6800/4046643232,v1:172.21.15.50:6801/4046643232] conn(0x7ff9ac02fd10 0x7ff9ac0321d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7ff9c00096a0 tx=0x7ff9c0006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:06:41.525 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:41.524+0000 7ff9b67fc700 1 -- 172.21.15.50:0/511719526 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff9b8025460 con 0x7ff9c40703b0 2023-12-25T20:06:41.714 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:41.713+0000 7ff9ca99f700 1 -- 172.21.15.50:0/511719526 --> [v2:172.21.15.50:6800/4046643232,v1:172.21.15.50:6801/4046643232] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7ff9c410de50 con 0x7ff9ac02fd10 2023-12-25T20:06:41.718 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:41.718+0000 7ff9b67fc700 1 -- 172.21.15.50:0/511719526 <== mgr.14114 v2:172.21.15.50:6800/4046643232 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7ff9c410de50 con 0x7ff9ac02fd10 2023-12-25T20:06:41.723 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:41.720+0000 7ff9ca99f700 1 -- 172.21.15.50:0/511719526 >> [v2:172.21.15.50:6800/4046643232,v1:172.21.15.50:6801/4046643232] conn(0x7ff9ac02fd10 msgr2=0x7ff9ac0321d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:06:41.723 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:41.720+0000 7ff9ca99f700 1 --2- 172.21.15.50:0/511719526 >> [v2:172.21.15.50:6800/4046643232,v1:172.21.15.50:6801/4046643232] conn(0x7ff9ac02fd10 0x7ff9ac0321d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7ff9c00096a0 tx=0x7ff9c0006b40).stop 2023-12-25T20:06:41.723 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:41.720+0000 7ff9ca99f700 1 -- 172.21.15.50:0/511719526 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff9c40703b0 msgr2=0x7ff9c410ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:06:41.723 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:41.720+0000 7ff9ca99f700 1 --2- 172.21.15.50:0/511719526 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff9c40703b0 0x7ff9c410ca70 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7ff9b800e040 tx=0x7ff9b8006ee0).stop 2023-12-25T20:06:41.723 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:41.720+0000 7ff9ca99f700 1 -- 172.21.15.50:0/511719526 shutdown_connections 2023-12-25T20:06:41.723 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:41.720+0000 7ff9ca99f700 1 --2- 172.21.15.50:0/511719526 >> [v2:172.21.15.50:6800/4046643232,v1:172.21.15.50:6801/4046643232] conn(0x7ff9ac02fd10 0x7ff9ac0321d0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:06:41.723 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:41.720+0000 7ff9ca99f700 1 --2- 172.21.15.50:0/511719526 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff9c40703b0 0x7ff9c410ca70 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:06:41.724 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:41.720+0000 7ff9ca99f700 1 -- 172.21.15.50:0/511719526 >> 172.21.15.50:0/511719526 conn(0x7ff9c40f6210 msgr2=0x7ff9c40f7d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:06:41.724 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:41.720+0000 7ff9ca99f700 1 -- 172.21.15.50:0/511719526 shutdown_connections 2023-12-25T20:06:41.724 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:41.720+0000 7ff9ca99f700 1 -- 172.21.15.50:0/511719526 wait complete. 2023-12-25T20:06:41.881 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:41 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:06:40.573628+0000 mgr.smithi050.nbzhun (mgr.14114) 1 : audit [DBG] from='client.14118 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2023-12-25T20:06:41.881 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:41 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:06:40.573883+0000 mon.smithi050 (mon.0) 48 : cluster [DBG] mgrmap e7: smithi050.nbzhun(active, since 1.05525s) 2023-12-25T20:06:41.882 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:41 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:06:40.580382+0000 mgr.smithi050.nbzhun (mgr.14114) 2 : audit [DBG] from='client.14118 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2023-12-25T20:06:42.066 INFO:teuthology.orchestra.run.smithi050.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi050 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/02c8f25c-a361-11ee-95a5-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmps7kf4jlu:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpu2g3lwcx:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm set-user root 2023-12-25T20:06:42.411 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:42.403+0000 7f96da17d700 1 Processor -- start 2023-12-25T20:06:42.411 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:42.403+0000 7f96da17d700 1 -- start start 2023-12-25T20:06:42.411 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:42.403+0000 7f96da17d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f96d4070430 0x7f96d40fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-25T20:06:42.411 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:42.403+0000 7f96da17d700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96d40fedd0 con 0x7f96d4070430 2023-12-25T20:06:42.411 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:42.404+0000 7f96d37fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f96d4070430 0x7f96d40fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:06:42.411 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:42.404+0000 7f96d37fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f96d4070430 0x7f96d40fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:37602/0 (socket says 172.21.15.50:37602) 2023-12-25T20:06:42.412 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:42.404+0000 7f96d37fe700 1 -- 172.21.15.50:0/2564862738 learned_addr learned my addr 172.21.15.50:0/2564862738 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-25T20:06:42.412 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:42.404+0000 7f96d37fe700 1 -- 172.21.15.50:0/2564862738 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96d40fef10 con 0x7f96d4070430 2023-12-25T20:06:42.412 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:42.404+0000 7f96d37fe700 1 --2- 172.21.15.50:0/2564862738 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f96d4070430 0x7f96d40fe8e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f96c800b5f0 tx=0x7f96c8006ee0).ready entity=mon.0 client_cookie=eedbdd4d0dd02667 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:06:42.412 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:42.404+0000 7f96d27fc700 1 -- 172.21.15.50:0/2564862738 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f96c800d600 con 0x7f96d4070430 2023-12-25T20:06:42.412 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:42.404+0000 7f96d27fc700 1 -- 172.21.15.50:0/2564862738 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f96c800dbd0 con 0x7f96d4070430 2023-12-25T20:06:42.412 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:42.404+0000 7f96d27fc700 1 -- 172.21.15.50:0/2564862738 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f96c8011c00 con 0x7f96d4070430 2023-12-25T20:06:42.413 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:42.405+0000 7f96da17d700 1 -- 172.21.15.50:0/2564862738 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f96d4070430 msgr2=0x7f96d40fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:06:42.413 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:42.405+0000 7f96da17d700 1 --2- 172.21.15.50:0/2564862738 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f96d4070430 0x7f96d40fe8e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f96c800b5f0 tx=0x7f96c8006ee0).stop 2023-12-25T20:06:42.413 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:42.405+0000 7f96da17d700 1 -- 172.21.15.50:0/2564862738 shutdown_connections 2023-12-25T20:06:42.413 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:42.405+0000 7f96da17d700 1 --2- 172.21.15.50:0/2564862738 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f96d4070430 0x7f96d40fe8e0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:06:42.413 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:42.405+0000 7f96da17d700 1 -- 172.21.15.50:0/2564862738 >> 172.21.15.50:0/2564862738 conn(0x7f96d40f63c0 msgr2=0x7f96d40f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:06:42.413 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:42.405+0000 7f96da17d700 1 -- 172.21.15.50:0/2564862738 shutdown_connections 2023-12-25T20:06:42.413 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:42.405+0000 7f96da17d700 1 -- 172.21.15.50:0/2564862738 wait complete. 2023-12-25T20:06:42.414 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:42.405+0000 7f96da17d700 1 Processor -- start 2023-12-25T20:06:42.414 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:42.406+0000 7f96da17d700 1 -- start start 2023-12-25T20:06:42.414 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:42.406+0000 7f96da17d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f96d4070430 0x7f96d410cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:06:42.414 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:42.406+0000 7f96da17d700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96d410d0d0 con 0x7f96d4070430 2023-12-25T20:06:42.414 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:42.406+0000 7f96d37fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f96d4070430 0x7f96d410cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:06:42.414 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:42.406+0000 7f96d37fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f96d4070430 0x7f96d410cbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:37612/0 (socket says 172.21.15.50:37612) 2023-12-25T20:06:42.414 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:42.406+0000 7f96d37fe700 1 -- 172.21.15.50:0/2096957377 learned_addr learned my addr 172.21.15.50:0/2096957377 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-25T20:06:42.414 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:42.406+0000 7f96d37fe700 1 -- 172.21.15.50:0/2096957377 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96c800b200 con 0x7f96d4070430 2023-12-25T20:06:42.414 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:42.406+0000 7f96d37fe700 1 --2- 172.21.15.50:0/2096957377 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f96d4070430 0x7f96d410cbe0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f96c800b190 tx=0x7f96c8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:06:42.414 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:42.407+0000 7f96bbfff700 1 -- 172.21.15.50:0/2096957377 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f96c801b410 con 0x7f96d4070430 2023-12-25T20:06:42.414 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:42.407+0000 7f96bbfff700 1 -- 172.21.15.50:0/2096957377 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f96c801b9e0 con 0x7f96d4070430 2023-12-25T20:06:42.415 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:42.407+0000 7f96da17d700 1 -- 172.21.15.50:0/2096957377 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96d410d2d0 con 0x7f96d4070430 2023-12-25T20:06:42.415 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:42.407+0000 7f96bbfff700 1 -- 172.21.15.50:0/2096957377 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f96c801a880 con 0x7f96d4070430 2023-12-25T20:06:42.415 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:42.407+0000 7f96da17d700 1 -- 172.21.15.50:0/2096957377 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96d410d6e0 con 0x7f96d4070430 2023-12-25T20:06:42.415 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:42.407+0000 7f96bbfff700 1 -- 172.21.15.50:0/2096957377 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f96c800f070 con 0x7f96d4070430 2023-12-25T20:06:42.415 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:42.408+0000 7f96da17d700 1 -- 172.21.15.50:0/2096957377 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96d4045ad0 con 0x7f96d4070430 2023-12-25T20:06:42.415 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:42.408+0000 7f96bbfff700 1 --2- 172.21.15.50:0/2096957377 >> [v2:172.21.15.50:6800/4046643232,v1:172.21.15.50:6801/4046643232] conn(0x7f96c002fdb0 0x7f96c0032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:06:42.415 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:42.408+0000 7f96bbfff700 1 -- 172.21.15.50:0/2096957377 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f96c80411c0 con 0x7f96d4070430 2023-12-25T20:06:42.416 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:42.408+0000 7f96d2ffd700 1 --2- 172.21.15.50:0/2096957377 >> [v2:172.21.15.50:6800/4046643232,v1:172.21.15.50:6801/4046643232] conn(0x7f96c002fdb0 0x7f96c0032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:06:42.416 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:42.408+0000 7f96d2ffd700 1 --2- 172.21.15.50:0/2096957377 >> [v2:172.21.15.50:6800/4046643232,v1:172.21.15.50:6801/4046643232] conn(0x7f96c002fdb0 0x7f96c0032270 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f96c40096a0 tx=0x7f96c4006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:06:42.416 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:42.412+0000 7f96bbfff700 1 -- 172.21.15.50:0/2096957377 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f96c8022d20 con 0x7f96d4070430 2023-12-25T20:06:42.574 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:42.573+0000 7f96da17d700 1 -- 172.21.15.50:0/2096957377 --> [v2:172.21.15.50:6800/4046643232,v1:172.21.15.50:6801/4046643232] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f96d410dfc0 con 0x7f96c002fdb0 2023-12-25T20:06:42.574 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:42.573+0000 7f96bbfff700 1 -- 172.21.15.50:0/2096957377 <== mgr.14114 v2:172.21.15.50:6800/4046643232 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f96d410dfc0 con 0x7f96c002fdb0 2023-12-25T20:06:42.574 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: value unchanged 2023-12-25T20:06:42.577 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:42.575+0000 7f96da17d700 1 -- 172.21.15.50:0/2096957377 >> [v2:172.21.15.50:6800/4046643232,v1:172.21.15.50:6801/4046643232] conn(0x7f96c002fdb0 msgr2=0x7f96c0032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:06:42.577 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:42.575+0000 7f96da17d700 1 --2- 172.21.15.50:0/2096957377 >> [v2:172.21.15.50:6800/4046643232,v1:172.21.15.50:6801/4046643232] conn(0x7f96c002fdb0 0x7f96c0032270 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f96c40096a0 tx=0x7f96c4006b40).stop 2023-12-25T20:06:42.577 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:42.575+0000 7f96da17d700 1 -- 172.21.15.50:0/2096957377 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f96d4070430 msgr2=0x7f96d410cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:06:42.577 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:42.575+0000 7f96da17d700 1 --2- 172.21.15.50:0/2096957377 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f96d4070430 0x7f96d410cbe0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f96c800b190 tx=0x7f96c8006ee0).stop 2023-12-25T20:06:42.577 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:42.575+0000 7f96da17d700 1 -- 172.21.15.50:0/2096957377 shutdown_connections 2023-12-25T20:06:42.577 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:42.575+0000 7f96da17d700 1 --2- 172.21.15.50:0/2096957377 >> [v2:172.21.15.50:6800/4046643232,v1:172.21.15.50:6801/4046643232] conn(0x7f96c002fdb0 0x7f96c0032270 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:06:42.577 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:42.575+0000 7f96da17d700 1 --2- 172.21.15.50:0/2096957377 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f96d4070430 0x7f96d410cbe0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:06:42.577 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:42.575+0000 7f96da17d700 1 -- 172.21.15.50:0/2096957377 >> 172.21.15.50:0/2096957377 conn(0x7f96d40f63c0 msgr2=0x7f96d40f7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:06:42.577 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:42.576+0000 7f96da17d700 1 -- 172.21.15.50:0/2096957377 shutdown_connections 2023-12-25T20:06:42.578 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:42.576+0000 7f96da17d700 1 -- 172.21.15.50:0/2096957377 wait complete. 2023-12-25T20:06:42.901 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:42 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:06:41.715211+0000 mgr.smithi050.nbzhun (mgr.14114) 3 : audit [DBG] from='client.14124 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2023-12-25T20:06:42.902 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:42 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:06:41.718962+0000 mon.smithi050 (mon.0) 49 : audit [INF] from='mgr.14114 172.21.15.50:0/3381094233' entity='mgr.smithi050.nbzhun' 2023-12-25T20:06:42.902 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:42 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:06:41.722498+0000 mon.smithi050 (mon.0) 50 : audit [DBG] from='mgr.14114 172.21.15.50:0/3381094233' entity='mgr.smithi050.nbzhun' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-25T20:06:42.902 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:42 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:06:41.738831+0000 mon.smithi050 (mon.0) 51 : cluster [DBG] mgrmap e8: smithi050.nbzhun(active, since 2s) 2023-12-25T20:06:42.941 INFO:teuthology.orchestra.run.smithi050.stderr:Generating ssh key... 2023-12-25T20:06:42.942 INFO:teuthology.orchestra.run.smithi050.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi050 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/02c8f25c-a361-11ee-95a5-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmps7kf4jlu:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpu2g3lwcx:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm generate-key 2023-12-25T20:06:43.272 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:43.268+0000 7f2699b86700 1 Processor -- start 2023-12-25T20:06:43.272 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:43.268+0000 7f2699b86700 1 -- start start 2023-12-25T20:06:43.272 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:43.269+0000 7f2699b86700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f26940fd360 0x7f26940fd780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-25T20:06:43.272 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:43.269+0000 7f2699b86700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26940fdc70 con 0x7f26940fd360 2023-12-25T20:06:43.272 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:43.269+0000 7f26937fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f26940fd360 0x7f26940fd780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:06:43.272 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:43.269+0000 7f26937fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f26940fd360 0x7f26940fd780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:37622/0 (socket says 172.21.15.50:37622) 2023-12-25T20:06:43.272 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:43.269+0000 7f26937fe700 1 -- 172.21.15.50:0/1862405192 learned_addr learned my addr 172.21.15.50:0/1862405192 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-25T20:06:43.273 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:43.269+0000 7f26937fe700 1 -- 172.21.15.50:0/1862405192 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26940fddb0 con 0x7f26940fd360 2023-12-25T20:06:43.273 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:43.269+0000 7f26937fe700 1 --2- 172.21.15.50:0/1862405192 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f26940fd360 0x7f26940fd780 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f268400b5f0 tx=0x7f2684006ee0).ready entity=mon.0 client_cookie=cc76b9315b193f07 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:06:43.273 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:43.269+0000 7f26927fc700 1 -- 172.21.15.50:0/1862405192 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f268400d600 con 0x7f26940fd360 2023-12-25T20:06:43.273 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:43.269+0000 7f26927fc700 1 -- 172.21.15.50:0/1862405192 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f268400dbd0 con 0x7f26940fd360 2023-12-25T20:06:43.273 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:43.269+0000 7f26927fc700 1 -- 172.21.15.50:0/1862405192 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2684011c00 con 0x7f26940fd360 2023-12-25T20:06:43.273 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:43.270+0000 7f2699b86700 1 -- 172.21.15.50:0/1862405192 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f26940fd360 msgr2=0x7f26940fd780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:06:43.273 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:43.270+0000 7f2699b86700 1 --2- 172.21.15.50:0/1862405192 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f26940fd360 0x7f26940fd780 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f268400b5f0 tx=0x7f2684006ee0).stop 2023-12-25T20:06:43.273 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:43.270+0000 7f2699b86700 1 -- 172.21.15.50:0/1862405192 shutdown_connections 2023-12-25T20:06:43.273 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:43.270+0000 7f2699b86700 1 --2- 172.21.15.50:0/1862405192 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f26940fd360 0x7f26940fd780 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:06:43.273 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:43.270+0000 7f2699b86700 1 -- 172.21.15.50:0/1862405192 >> 172.21.15.50:0/1862405192 conn(0x7f26940f5b40 msgr2=0x7f26940f7f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:06:43.273 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:43.270+0000 7f2699b86700 1 -- 172.21.15.50:0/1862405192 shutdown_connections 2023-12-25T20:06:43.273 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:43.270+0000 7f2699b86700 1 -- 172.21.15.50:0/1862405192 wait complete. 2023-12-25T20:06:43.273 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:43.270+0000 7f2699b86700 1 Processor -- start 2023-12-25T20:06:43.273 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:43.271+0000 7f2699b86700 1 -- start start 2023-12-25T20:06:43.273 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:43.271+0000 7f2699b86700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f26940fd360 0x7f269410aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:06:43.274 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:43.271+0000 7f2699b86700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f269410af10 con 0x7f26940fd360 2023-12-25T20:06:43.274 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:43.271+0000 7f26937fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f26940fd360 0x7f269410aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:06:43.274 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:43.271+0000 7f26937fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f26940fd360 0x7f269410aa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:37630/0 (socket says 172.21.15.50:37630) 2023-12-25T20:06:43.274 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:43.271+0000 7f26937fe700 1 -- 172.21.15.50:0/1624030704 learned_addr learned my addr 172.21.15.50:0/1624030704 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-25T20:06:43.274 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:43.271+0000 7f26937fe700 1 -- 172.21.15.50:0/1624030704 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f268400b200 con 0x7f26940fd360 2023-12-25T20:06:43.274 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:43.271+0000 7f26937fe700 1 --2- 172.21.15.50:0/1624030704 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f26940fd360 0x7f269410aa20 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f2684002f40 tx=0x7f2684006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:06:43.275 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:43.272+0000 7f267bfff700 1 -- 172.21.15.50:0/1624030704 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f268401b410 con 0x7f26940fd360 2023-12-25T20:06:43.275 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:43.272+0000 7f267bfff700 1 -- 172.21.15.50:0/1624030704 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f268401b9e0 con 0x7f26940fd360 2023-12-25T20:06:43.275 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:43.272+0000 7f2699b86700 1 -- 172.21.15.50:0/1624030704 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f269410b110 con 0x7f26940fd360 2023-12-25T20:06:43.275 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:43.272+0000 7f267bfff700 1 -- 172.21.15.50:0/1624030704 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f268401a8b0 con 0x7f26940fd360 2023-12-25T20:06:43.275 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:43.272+0000 7f2699b86700 1 -- 172.21.15.50:0/1624030704 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f269410b520 con 0x7f26940fd360 2023-12-25T20:06:43.276 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:43.272+0000 7f267bfff700 1 -- 172.21.15.50:0/1624030704 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f268400f070 con 0x7f26940fd360 2023-12-25T20:06:43.276 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:43.273+0000 7f267bfff700 1 --2- 172.21.15.50:0/1624030704 >> [v2:172.21.15.50:6800/4046643232,v1:172.21.15.50:6801/4046643232] conn(0x7f268002fd50 0x7f2680032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:06:43.276 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:43.273+0000 7f267bfff700 1 -- 172.21.15.50:0/1624030704 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f26840411d0 con 0x7f26940fd360 2023-12-25T20:06:43.276 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:43.273+0000 7f2699b86700 1 -- 172.21.15.50:0/1624030704 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2694045ad0 con 0x7f26940fd360 2023-12-25T20:06:43.276 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:43.273+0000 7f2692ffd700 1 --2- 172.21.15.50:0/1624030704 >> [v2:172.21.15.50:6800/4046643232,v1:172.21.15.50:6801/4046643232] conn(0x7f268002fd50 0x7f2680032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:06:43.276 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:43.273+0000 7f2692ffd700 1 --2- 172.21.15.50:0/1624030704 >> [v2:172.21.15.50:6800/4046643232,v1:172.21.15.50:6801/4046643232] conn(0x7f268002fd50 0x7f2680032210 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f26880096a0 tx=0x7f2688006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:06:43.278 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:43.277+0000 7f267bfff700 1 -- 172.21.15.50:0/1624030704 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2684016070 con 0x7f26940fd360 2023-12-25T20:06:43.437 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:43.436+0000 7f2699b86700 1 -- 172.21.15.50:0/1624030704 --> [v2:172.21.15.50:6800/4046643232,v1:172.21.15.50:6801/4046643232] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f269410be60 con 0x7f268002fd50 2023-12-25T20:06:43.661 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:43.661+0000 7f267bfff700 1 -- 172.21.15.50:0/1624030704 <== mgr.14114 v2:172.21.15.50:6800/4046643232 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f269410be60 con 0x7f268002fd50 2023-12-25T20:06:43.664 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:43.662+0000 7f2699b86700 1 -- 172.21.15.50:0/1624030704 >> [v2:172.21.15.50:6800/4046643232,v1:172.21.15.50:6801/4046643232] conn(0x7f268002fd50 msgr2=0x7f2680032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:06:43.664 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:43.662+0000 7f2699b86700 1 --2- 172.21.15.50:0/1624030704 >> [v2:172.21.15.50:6800/4046643232,v1:172.21.15.50:6801/4046643232] conn(0x7f268002fd50 0x7f2680032210 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f26880096a0 tx=0x7f2688006b40).stop 2023-12-25T20:06:43.664 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:43.662+0000 7f2699b86700 1 -- 172.21.15.50:0/1624030704 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f26940fd360 msgr2=0x7f269410aa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:06:43.664 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:43.663+0000 7f2699b86700 1 --2- 172.21.15.50:0/1624030704 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f26940fd360 0x7f269410aa20 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f2684002f40 tx=0x7f2684006ee0).stop 2023-12-25T20:06:43.664 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:43.663+0000 7f2699b86700 1 -- 172.21.15.50:0/1624030704 shutdown_connections 2023-12-25T20:06:43.664 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:43.663+0000 7f2699b86700 1 --2- 172.21.15.50:0/1624030704 >> [v2:172.21.15.50:6800/4046643232,v1:172.21.15.50:6801/4046643232] conn(0x7f268002fd50 0x7f2680032210 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:06:43.664 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:43.663+0000 7f2699b86700 1 --2- 172.21.15.50:0/1624030704 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f26940fd360 0x7f269410aa20 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:06:43.664 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:43.663+0000 7f2699b86700 1 -- 172.21.15.50:0/1624030704 >> 172.21.15.50:0/1624030704 conn(0x7f26940f5b40 msgr2=0x7f26940ff260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:06:43.665 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:43.664+0000 7f2699b86700 1 -- 172.21.15.50:0/1624030704 shutdown_connections 2023-12-25T20:06:43.665 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:43.664+0000 7f2699b86700 1 -- 172.21.15.50:0/1624030704 wait complete. 2023-12-25T20:06:43.968 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:43 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:06:42.575043+0000 mgr.smithi050.nbzhun (mgr.14114) 4 : audit [DBG] from='client.14126 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2023-12-25T20:06:43.969 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:43 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:06:43.659698+0000 mon.smithi050 (mon.0) 52 : audit [INF] from='mgr.14114 172.21.15.50:0/3381094233' entity='mgr.smithi050.nbzhun' 2023-12-25T20:06:43.969 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:43 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:06:43.661433+0000 mon.smithi050 (mon.0) 53 : audit [INF] from='mgr.14114 172.21.15.50:0/3381094233' entity='mgr.smithi050.nbzhun' 2023-12-25T20:06:44.007 INFO:teuthology.orchestra.run.smithi050.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi050 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/02c8f25c-a361-11ee-95a5-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmps7kf4jlu:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpu2g3lwcx:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm get-pub-key 2023-12-25T20:06:44.357 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:44.352+0000 7fab5f59e700 1 Processor -- start 2023-12-25T20:06:44.358 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:44.352+0000 7fab5f59e700 1 -- start start 2023-12-25T20:06:44.358 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:44.352+0000 7fab5f59e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fab600703f0 0x7fab600fe770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-25T20:06:44.358 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:44.352+0000 7fab5f59e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab600fec60 con 0x7fab600703f0 2023-12-25T20:06:44.358 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:44.352+0000 7fab5e59c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fab600703f0 0x7fab600fe770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:06:44.358 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:44.352+0000 7fab5e59c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fab600703f0 0x7fab600fe770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:37638/0 (socket says 172.21.15.50:37638) 2023-12-25T20:06:44.358 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:44.352+0000 7fab5e59c700 1 -- 172.21.15.50:0/3753479343 learned_addr learned my addr 172.21.15.50:0/3753479343 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-25T20:06:44.358 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:44.352+0000 7fab5e59c700 1 -- 172.21.15.50:0/3753479343 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab600feda0 con 0x7fab600703f0 2023-12-25T20:06:44.358 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:44.353+0000 7fab5e59c700 1 --2- 172.21.15.50:0/3753479343 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fab600703f0 0x7fab600fe770 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7fab5000b5f0 tx=0x7fab50006ee0).ready entity=mon.0 client_cookie=313d853fb6f8b8a9 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:06:44.358 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:44.353+0000 7fab5d59a700 1 -- 172.21.15.50:0/3753479343 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fab5000d600 con 0x7fab600703f0 2023-12-25T20:06:44.359 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:44.353+0000 7fab5d59a700 1 -- 172.21.15.50:0/3753479343 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fab5000dbd0 con 0x7fab600703f0 2023-12-25T20:06:44.359 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:44.353+0000 7fab5d59a700 1 -- 172.21.15.50:0/3753479343 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fab50011c00 con 0x7fab600703f0 2023-12-25T20:06:44.359 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:44.353+0000 7fab5f59e700 1 -- 172.21.15.50:0/3753479343 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fab600703f0 msgr2=0x7fab600fe770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:06:44.359 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:44.353+0000 7fab5f59e700 1 --2- 172.21.15.50:0/3753479343 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fab600703f0 0x7fab600fe770 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7fab5000b5f0 tx=0x7fab50006ee0).stop 2023-12-25T20:06:44.359 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:44.353+0000 7fab5f59e700 1 -- 172.21.15.50:0/3753479343 shutdown_connections 2023-12-25T20:06:44.359 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:44.353+0000 7fab5f59e700 1 --2- 172.21.15.50:0/3753479343 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fab600703f0 0x7fab600fe770 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:06:44.359 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:44.353+0000 7fab5f59e700 1 -- 172.21.15.50:0/3753479343 >> 172.21.15.50:0/3753479343 conn(0x7fab600f6210 msgr2=0x7fab600f8630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:06:44.359 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:44.353+0000 7fab5f59e700 1 -- 172.21.15.50:0/3753479343 shutdown_connections 2023-12-25T20:06:44.360 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:44.353+0000 7fab5f59e700 1 -- 172.21.15.50:0/3753479343 wait complete. 2023-12-25T20:06:44.360 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:44.354+0000 7fab5f59e700 1 Processor -- start 2023-12-25T20:06:44.360 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:44.354+0000 7fab5f59e700 1 -- start start 2023-12-25T20:06:44.360 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:44.354+0000 7fab5f59e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fab600703f0 0x7fab600fc1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:06:44.360 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:44.354+0000 7fab5f59e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab600fa670 con 0x7fab600703f0 2023-12-25T20:06:44.360 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:44.354+0000 7fab5e59c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fab600703f0 0x7fab600fc1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:06:44.360 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:44.354+0000 7fab5e59c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fab600703f0 0x7fab600fc1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:37650/0 (socket says 172.21.15.50:37650) 2023-12-25T20:06:44.360 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:44.354+0000 7fab5e59c700 1 -- 172.21.15.50:0/1988745039 learned_addr learned my addr 172.21.15.50:0/1988745039 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-25T20:06:44.360 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:44.355+0000 7fab5e59c700 1 -- 172.21.15.50:0/1988745039 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab5000b200 con 0x7fab600703f0 2023-12-25T20:06:44.360 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:44.355+0000 7fab5e59c700 1 --2- 172.21.15.50:0/1988745039 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fab600703f0 0x7fab600fc1f0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7fab5000b190 tx=0x7fab50006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:06:44.361 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:44.355+0000 7fab4effd700 1 -- 172.21.15.50:0/1988745039 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fab5001b410 con 0x7fab600703f0 2023-12-25T20:06:44.361 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:44.355+0000 7fab4effd700 1 -- 172.21.15.50:0/1988745039 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fab5001b9e0 con 0x7fab600703f0 2023-12-25T20:06:44.361 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:44.355+0000 7fab5f59e700 1 -- 172.21.15.50:0/1988745039 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab600fa810 con 0x7fab600703f0 2023-12-25T20:06:44.361 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:44.355+0000 7fab4effd700 1 -- 172.21.15.50:0/1988745039 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fab5001a880 con 0x7fab600703f0 2023-12-25T20:06:44.361 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:44.355+0000 7fab5f59e700 1 -- 172.21.15.50:0/1988745039 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab600fac20 con 0x7fab600703f0 2023-12-25T20:06:44.361 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:44.356+0000 7fab5f59e700 1 -- 172.21.15.50:0/1988745039 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab60045ad0 con 0x7fab600703f0 2023-12-25T20:06:44.361 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:44.356+0000 7fab4effd700 1 -- 172.21.15.50:0/1988745039 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fab5000f070 con 0x7fab600703f0 2023-12-25T20:06:44.361 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:44.356+0000 7fab4effd700 1 --2- 172.21.15.50:0/1988745039 >> [v2:172.21.15.50:6800/4046643232,v1:172.21.15.50:6801/4046643232] conn(0x7fab4402fdb0 0x7fab44032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:06:44.361 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:44.356+0000 7fab4effd700 1 -- 172.21.15.50:0/1988745039 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fab50041170 con 0x7fab600703f0 2023-12-25T20:06:44.361 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:44.356+0000 7fab5dd9b700 1 --2- 172.21.15.50:0/1988745039 >> [v2:172.21.15.50:6800/4046643232,v1:172.21.15.50:6801/4046643232] conn(0x7fab4402fdb0 0x7fab44032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:06:44.361 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:44.357+0000 7fab5dd9b700 1 --2- 172.21.15.50:0/1988745039 >> [v2:172.21.15.50:6800/4046643232,v1:172.21.15.50:6801/4046643232] conn(0x7fab4402fdb0 0x7fab44032270 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7fab540096a0 tx=0x7fab54006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:06:44.362 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:44.360+0000 7fab4effd700 1 -- 172.21.15.50:0/1988745039 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fab50019950 con 0x7fab600703f0 2023-12-25T20:06:44.526 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:44.521+0000 7fab5f59e700 1 -- 172.21.15.50:0/1988745039 --> [v2:172.21.15.50:6800/4046643232,v1:172.21.15.50:6801/4046643232] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7fab600fae60 con 0x7fab4402fdb0 2023-12-25T20:06:44.527 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:44.522+0000 7fab4effd700 1 -- 172.21.15.50:0/1988745039 <== mgr.14114 v2:172.21.15.50:6800/4046643232 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7fab600fae60 con 0x7fab4402fdb0 2023-12-25T20:06:44.527 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDBcLPR/YSyydiXtXm47+k2ciQm4QF77G2lYmwHiS+H61PQs1KnHmbDj55yK1iabJYJLy0AfqaAEgXGKJXCuA4Ma5pjRu9hD3k7zotzb3matENxJJ+k0ubYvVij7DWjUTJborwswJy3/hySM0S6mPxul05CI7dThlnxIRonWJCvriSyiQm4o+OMUWB3M09lyEbVIw3oM2PtYCNGcUw8CWPPtawiSZKaUvE0K/nEr1jBLOOYCHEx9grHI9GVJxyMJc6rVzmNm4Ls9QdQWTN73CJlVFDs37pZF6j+i9w0cbvtDFf+pCiFQOkPyWb0tvbXqbKssU7ajwOXa3dgOtDX/2ZafblngS1XnrduJtJXDE0y7mvyya8JPjO/Sb934kK73wtuwUZ8JrtrHXzy81Gt/9Eb0jBMs72VPlPCsKhK3Jzobw1G11iftwoS9kIMwO9xEBG4lCEYqCpDuPaiHoVAUxpRGygXOYSAPWZF4rJ1yf/PaGTi2UrApG5mzenjTXAiifk= ceph-02c8f25c-a361-11ee-95a5-87774f69a715 2023-12-25T20:06:44.527 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:44.524+0000 7fab5f59e700 1 -- 172.21.15.50:0/1988745039 >> [v2:172.21.15.50:6800/4046643232,v1:172.21.15.50:6801/4046643232] conn(0x7fab4402fdb0 msgr2=0x7fab44032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:06:44.527 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:44.524+0000 7fab5f59e700 1 --2- 172.21.15.50:0/1988745039 >> [v2:172.21.15.50:6800/4046643232,v1:172.21.15.50:6801/4046643232] conn(0x7fab4402fdb0 0x7fab44032270 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7fab540096a0 tx=0x7fab54006b40).stop 2023-12-25T20:06:44.527 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:44.524+0000 7fab5f59e700 1 -- 172.21.15.50:0/1988745039 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fab600703f0 msgr2=0x7fab600fc1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:06:44.527 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:44.524+0000 7fab5f59e700 1 --2- 172.21.15.50:0/1988745039 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fab600703f0 0x7fab600fc1f0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7fab5000b190 tx=0x7fab50006ee0).stop 2023-12-25T20:06:44.527 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:44.524+0000 7fab5f59e700 1 -- 172.21.15.50:0/1988745039 shutdown_connections 2023-12-25T20:06:44.527 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:44.524+0000 7fab5f59e700 1 --2- 172.21.15.50:0/1988745039 >> [v2:172.21.15.50:6800/4046643232,v1:172.21.15.50:6801/4046643232] conn(0x7fab4402fdb0 0x7fab44032270 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:06:44.527 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:44.524+0000 7fab5f59e700 1 --2- 172.21.15.50:0/1988745039 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fab600703f0 0x7fab600fc1f0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:06:44.527 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:44.524+0000 7fab5f59e700 1 -- 172.21.15.50:0/1988745039 >> 172.21.15.50:0/1988745039 conn(0x7fab600f6210 msgr2=0x7fab600f7ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:06:44.527 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:44.524+0000 7fab5f59e700 1 -- 172.21.15.50:0/1988745039 shutdown_connections 2023-12-25T20:06:44.527 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:44.524+0000 7fab5f59e700 1 -- 172.21.15.50:0/1988745039 wait complete. 2023-12-25T20:06:44.843 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:44 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cephadm 2023-12-25T20:06:43.438085+0000 mgr.smithi050.nbzhun (mgr.14114) 6 : cephadm [INF] Generating ssh key... 2023-12-25T20:06:44.882 INFO:teuthology.orchestra.run.smithi050.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2023-12-25T20:06:44.883 INFO:teuthology.orchestra.run.smithi050.stderr:Adding key to root@localhost authorized_keys... 2023-12-25T20:06:44.883 INFO:teuthology.orchestra.run.smithi050.stderr:Adding host smithi050... 2023-12-25T20:06:44.883 INFO:teuthology.orchestra.run.smithi050.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi050 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/02c8f25c-a361-11ee-95a5-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmps7kf4jlu:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpu2g3lwcx:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch host add smithi050 172.21.15.50 2023-12-25T20:06:45.219 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:45.216+0000 7fbc48903700 1 Processor -- start 2023-12-25T20:06:45.220 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:45.216+0000 7fbc48903700 1 -- start start 2023-12-25T20:06:45.220 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:45.217+0000 7fbc48903700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbc440fab90 0x7fbc440fafb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-25T20:06:45.220 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:45.217+0000 7fbc48903700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc440fb4a0 con 0x7fbc440fab90 2023-12-25T20:06:45.220 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:45.217+0000 7fbc4259c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbc440fab90 0x7fbc440fafb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:06:45.220 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:45.217+0000 7fbc4259c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbc440fab90 0x7fbc440fafb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:37652/0 (socket says 172.21.15.50:37652) 2023-12-25T20:06:45.220 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:45.217+0000 7fbc4259c700 1 -- 172.21.15.50:0/3886326355 learned_addr learned my addr 172.21.15.50:0/3886326355 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-25T20:06:45.220 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:45.217+0000 7fbc4259c700 1 -- 172.21.15.50:0/3886326355 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc440fb5e0 con 0x7fbc440fab90 2023-12-25T20:06:45.220 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:45.217+0000 7fbc4259c700 1 --2- 172.21.15.50:0/3886326355 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbc440fab90 0x7fbc440fafb0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7fbc3400b5f0 tx=0x7fbc34006ee0).ready entity=mon.0 client_cookie=e5c6e6d9b3a3c533 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:06:45.220 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:45.217+0000 7fbc4159a700 1 -- 172.21.15.50:0/3886326355 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbc3400d600 con 0x7fbc440fab90 2023-12-25T20:06:45.220 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:45.217+0000 7fbc4159a700 1 -- 172.21.15.50:0/3886326355 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fbc3400dbd0 con 0x7fbc440fab90 2023-12-25T20:06:45.220 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:45.217+0000 7fbc4159a700 1 -- 172.21.15.50:0/3886326355 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbc34011c00 con 0x7fbc440fab90 2023-12-25T20:06:45.220 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:45.218+0000 7fbc48903700 1 -- 172.21.15.50:0/3886326355 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbc440fab90 msgr2=0x7fbc440fafb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:06:45.220 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:45.218+0000 7fbc48903700 1 --2- 172.21.15.50:0/3886326355 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbc440fab90 0x7fbc440fafb0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7fbc3400b5f0 tx=0x7fbc34006ee0).stop 2023-12-25T20:06:45.221 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:45.218+0000 7fbc48903700 1 -- 172.21.15.50:0/3886326355 shutdown_connections 2023-12-25T20:06:45.221 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:45.218+0000 7fbc48903700 1 --2- 172.21.15.50:0/3886326355 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbc440fab90 0x7fbc440fafb0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:06:45.221 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:45.218+0000 7fbc48903700 1 -- 172.21.15.50:0/3886326355 >> 172.21.15.50:0/3886326355 conn(0x7fbc440f63b0 msgr2=0x7fbc440f87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:06:45.221 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:45.218+0000 7fbc48903700 1 -- 172.21.15.50:0/3886326355 shutdown_connections 2023-12-25T20:06:45.222 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:45.218+0000 7fbc48903700 1 -- 172.21.15.50:0/3886326355 wait complete. 2023-12-25T20:06:45.222 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:45.219+0000 7fbc48903700 1 Processor -- start 2023-12-25T20:06:45.222 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:45.219+0000 7fbc48903700 1 -- start start 2023-12-25T20:06:45.222 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:45.219+0000 7fbc48903700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbc440fab90 0x7fbc4410aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:06:45.223 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:45.219+0000 7fbc48903700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc4410aef0 con 0x7fbc440fab90 2023-12-25T20:06:45.223 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:45.219+0000 7fbc4259c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbc440fab90 0x7fbc4410aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:06:45.223 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:45.219+0000 7fbc4259c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbc440fab90 0x7fbc4410aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:37658/0 (socket says 172.21.15.50:37658) 2023-12-25T20:06:45.223 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:45.219+0000 7fbc4259c700 1 -- 172.21.15.50:0/509283328 learned_addr learned my addr 172.21.15.50:0/509283328 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-25T20:06:45.223 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:45.219+0000 7fbc4259c700 1 -- 172.21.15.50:0/509283328 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc3400b200 con 0x7fbc440fab90 2023-12-25T20:06:45.223 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:45.219+0000 7fbc4259c700 1 --2- 172.21.15.50:0/509283328 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbc440fab90 0x7fbc4410aa00 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7fbc3400b1d0 tx=0x7fbc34006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:06:45.223 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:45.219+0000 7fbc3affd700 1 -- 172.21.15.50:0/509283328 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbc3401b410 con 0x7fbc440fab90 2023-12-25T20:06:45.223 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:45.220+0000 7fbc3affd700 1 -- 172.21.15.50:0/509283328 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fbc3401b9e0 con 0x7fbc440fab90 2023-12-25T20:06:45.223 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:45.220+0000 7fbc3affd700 1 -- 172.21.15.50:0/509283328 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbc3401a840 con 0x7fbc440fab90 2023-12-25T20:06:45.223 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:45.220+0000 7fbc48903700 1 -- 172.21.15.50:0/509283328 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc4410b0f0 con 0x7fbc440fab90 2023-12-25T20:06:45.223 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:45.220+0000 7fbc48903700 1 -- 172.21.15.50:0/509283328 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc4410b500 con 0x7fbc440fab90 2023-12-25T20:06:45.224 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:45.220+0000 7fbc3affd700 1 -- 172.21.15.50:0/509283328 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fbc3400f070 con 0x7fbc440fab90 2023-12-25T20:06:45.224 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:45.220+0000 7fbc3affd700 1 --2- 172.21.15.50:0/509283328 >> [v2:172.21.15.50:6800/4046643232,v1:172.21.15.50:6801/4046643232] conn(0x7fbc2402fdb0 0x7fbc24032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:06:45.224 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:45.220+0000 7fbc3affd700 1 -- 172.21.15.50:0/509283328 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fbc34041210 con 0x7fbc440fab90 2023-12-25T20:06:45.224 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:45.221+0000 7fbc41d9b700 1 --2- 172.21.15.50:0/509283328 >> [v2:172.21.15.50:6800/4046643232,v1:172.21.15.50:6801/4046643232] conn(0x7fbc2402fdb0 0x7fbc24032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:06:45.224 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:45.221+0000 7fbc41d9b700 1 --2- 172.21.15.50:0/509283328 >> [v2:172.21.15.50:6800/4046643232,v1:172.21.15.50:6801/4046643232] conn(0x7fbc2402fdb0 0x7fbc24032270 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fbc2c0096a0 tx=0x7fbc2c006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:06:45.224 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:45.221+0000 7fbc48903700 1 -- 172.21.15.50:0/509283328 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc44059100 con 0x7fbc440fab90 2023-12-25T20:06:45.226 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:45.225+0000 7fbc3affd700 1 -- 172.21.15.50:0/509283328 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbc34016070 con 0x7fbc440fab90 2023-12-25T20:06:45.406 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:45.405+0000 7fbc48903700 1 -- 172.21.15.50:0/509283328 --> [v2:172.21.15.50:6800/4046643232,v1:172.21.15.50:6801/4046643232] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi050", "addr": "172.21.15.50", "target": ["mon-mgr", ""]}) v1 -- 0x7fbc44104a10 con 0x7fbc2402fdb0 2023-12-25T20:06:46.031 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: Added host 'smithi050' 2023-12-25T20:06:46.032 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:46.029+0000 7fbc3affd700 1 -- 172.21.15.50:0/509283328 <== mgr.14114 v2:172.21.15.50:6800/4046643232 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+23 (secure 0 0 0) 0x7fbc44104a10 con 0x7fbc2402fdb0 2023-12-25T20:06:46.034 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:46.031+0000 7fbc48903700 1 -- 172.21.15.50:0/509283328 >> [v2:172.21.15.50:6800/4046643232,v1:172.21.15.50:6801/4046643232] conn(0x7fbc2402fdb0 msgr2=0x7fbc24032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:06:46.034 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:46.031+0000 7fbc48903700 1 --2- 172.21.15.50:0/509283328 >> [v2:172.21.15.50:6800/4046643232,v1:172.21.15.50:6801/4046643232] conn(0x7fbc2402fdb0 0x7fbc24032270 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fbc2c0096a0 tx=0x7fbc2c006b40).stop 2023-12-25T20:06:46.034 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:46.031+0000 7fbc48903700 1 -- 172.21.15.50:0/509283328 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbc440fab90 msgr2=0x7fbc4410aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:06:46.035 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:46.031+0000 7fbc48903700 1 --2- 172.21.15.50:0/509283328 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbc440fab90 0x7fbc4410aa00 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7fbc3400b1d0 tx=0x7fbc34006ee0).stop 2023-12-25T20:06:46.035 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:46.032+0000 7fbc48903700 1 -- 172.21.15.50:0/509283328 shutdown_connections 2023-12-25T20:06:46.035 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:46.032+0000 7fbc48903700 1 --2- 172.21.15.50:0/509283328 >> [v2:172.21.15.50:6800/4046643232,v1:172.21.15.50:6801/4046643232] conn(0x7fbc2402fdb0 0x7fbc24032270 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:06:46.035 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:46.032+0000 7fbc48903700 1 --2- 172.21.15.50:0/509283328 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbc440fab90 0x7fbc4410aa00 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:06:46.035 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:46.032+0000 7fbc48903700 1 -- 172.21.15.50:0/509283328 >> 172.21.15.50:0/509283328 conn(0x7fbc440f63b0 msgr2=0x7fbc44070100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:06:46.035 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:46.032+0000 7fbc48903700 1 -- 172.21.15.50:0/509283328 shutdown_connections 2023-12-25T20:06:46.036 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:46.032+0000 7fbc48903700 1 -- 172.21.15.50:0/509283328 wait complete. 2023-12-25T20:06:46.065 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:45 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:06:44.523596+0000 mgr.smithi050.nbzhun (mgr.14114) 7 : audit [DBG] from='client.14130 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2023-12-25T20:06:47.469 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:47 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:06:45.407522+0000 mgr.smithi050.nbzhun (mgr.14114) 8 : audit [DBG] from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi050", "addr": "172.21.15.50", "target": ["mon-mgr", ""]}]: dispatch 2023-12-25T20:06:47.469 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:47 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:06:46.029789+0000 mon.smithi050 (mon.0) 54 : audit [INF] from='mgr.14114 172.21.15.50:0/3381094233' entity='mgr.smithi050.nbzhun' 2023-12-25T20:06:47.469 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:47 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cephadm 2023-12-25T20:06:46.030087+0000 mgr.smithi050.nbzhun (mgr.14114) 9 : cephadm [INF] Added host smithi050 2023-12-25T20:06:47.469 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:47 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:06:46.030310+0000 mon.smithi050 (mon.0) 55 : audit [DBG] from='mgr.14114 172.21.15.50:0/3381094233' entity='mgr.smithi050.nbzhun' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-25T20:06:47.824 INFO:teuthology.orchestra.run.smithi050.stderr:Deploying mon service with default placement... 2023-12-25T20:06:47.824 INFO:teuthology.orchestra.run.smithi050.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi050 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/02c8f25c-a361-11ee-95a5-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmps7kf4jlu:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpu2g3lwcx:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply mon 2023-12-25T20:06:49.214 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:49.031+0000 7f72bfae0700 1 Processor -- start 2023-12-25T20:06:49.214 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:49.031+0000 7f72bfae0700 1 -- start start 2023-12-25T20:06:49.214 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:49.031+0000 7f72bfae0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f72b80fd340 0x7f72b80fd760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-25T20:06:49.214 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:49.031+0000 7f72bfae0700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72b80fdc50 con 0x7f72b80fd340 2023-12-25T20:06:49.214 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:49.032+0000 7f72bd87c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f72b80fd340 0x7f72b80fd760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:06:49.215 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:49.032+0000 7f72bd87c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f72b80fd340 0x7f72b80fd760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:47748/0 (socket says 172.21.15.50:47748) 2023-12-25T20:06:49.215 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:49.032+0000 7f72bd87c700 1 -- 172.21.15.50:0/1731949637 learned_addr learned my addr 172.21.15.50:0/1731949637 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-25T20:06:49.215 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:49.032+0000 7f72bd87c700 1 -- 172.21.15.50:0/1731949637 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72b80fdd90 con 0x7f72b80fd340 2023-12-25T20:06:49.215 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:49.032+0000 7f72bd87c700 1 --2- 172.21.15.50:0/1731949637 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f72b80fd340 0x7f72b80fd760 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f72a800b5f0 tx=0x7f72a8006ee0).ready entity=mon.0 client_cookie=1d259e63aced5475 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:06:49.215 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:49.032+0000 7f72bc87a700 1 -- 172.21.15.50:0/1731949637 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f72a800d600 con 0x7f72b80fd340 2023-12-25T20:06:49.215 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:49.032+0000 7f72bc87a700 1 -- 172.21.15.50:0/1731949637 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f72a800dbd0 con 0x7f72b80fd340 2023-12-25T20:06:49.216 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:49.032+0000 7f72bc87a700 1 -- 172.21.15.50:0/1731949637 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f72a8011c00 con 0x7f72b80fd340 2023-12-25T20:06:49.216 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:49.033+0000 7f72bfae0700 1 -- 172.21.15.50:0/1731949637 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f72b80fd340 msgr2=0x7f72b80fd760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:06:49.216 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:49.033+0000 7f72bfae0700 1 --2- 172.21.15.50:0/1731949637 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f72b80fd340 0x7f72b80fd760 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f72a800b5f0 tx=0x7f72a8006ee0).stop 2023-12-25T20:06:49.216 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:49.033+0000 7f72bfae0700 1 -- 172.21.15.50:0/1731949637 shutdown_connections 2023-12-25T20:06:49.216 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:49.033+0000 7f72bfae0700 1 --2- 172.21.15.50:0/1731949637 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f72b80fd340 0x7f72b80fd760 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:06:49.216 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:49.033+0000 7f72bfae0700 1 -- 172.21.15.50:0/1731949637 >> 172.21.15.50:0/1731949637 conn(0x7f72b80f5b40 msgr2=0x7f72b80f7f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:06:49.217 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:49.033+0000 7f72bfae0700 1 -- 172.21.15.50:0/1731949637 shutdown_connections 2023-12-25T20:06:49.217 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:49.033+0000 7f72bfae0700 1 -- 172.21.15.50:0/1731949637 wait complete. 2023-12-25T20:06:49.217 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:49.034+0000 7f72bfae0700 1 Processor -- start 2023-12-25T20:06:49.217 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:49.034+0000 7f72bfae0700 1 -- start start 2023-12-25T20:06:49.217 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:49.034+0000 7f72bfae0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f72b80fd340 0x7f72b810aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:06:49.217 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:49.034+0000 7f72bfae0700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72b810af50 con 0x7f72b80fd340 2023-12-25T20:06:49.218 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:49.034+0000 7f72bd87c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f72b80fd340 0x7f72b810aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:06:49.218 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:49.034+0000 7f72bd87c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f72b80fd340 0x7f72b810aa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:47760/0 (socket says 172.21.15.50:47760) 2023-12-25T20:06:49.220 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:49.034+0000 7f72bd87c700 1 -- 172.21.15.50:0/4155171744 learned_addr learned my addr 172.21.15.50:0/4155171744 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-25T20:06:49.221 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:49.034+0000 7f72bd87c700 1 -- 172.21.15.50:0/4155171744 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72a800b200 con 0x7f72b80fd340 2023-12-25T20:06:49.221 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:49.034+0000 7f72bd87c700 1 --2- 172.21.15.50:0/4155171744 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f72b80fd340 0x7f72b810aa60 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f72a8002f40 tx=0x7f72a8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:06:49.221 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:49.035+0000 7f72a67fc700 1 -- 172.21.15.50:0/4155171744 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f72a801b410 con 0x7f72b80fd340 2023-12-25T20:06:49.221 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:49.035+0000 7f72a67fc700 1 -- 172.21.15.50:0/4155171744 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f72a801b9e0 con 0x7f72b80fd340 2023-12-25T20:06:49.221 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:49.035+0000 7f72bfae0700 1 -- 172.21.15.50:0/4155171744 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72b810b150 con 0x7f72b80fd340 2023-12-25T20:06:49.221 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:49.035+0000 7f72a67fc700 1 -- 172.21.15.50:0/4155171744 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f72a801a8b0 con 0x7f72b80fd340 2023-12-25T20:06:49.221 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:49.035+0000 7f72bfae0700 1 -- 172.21.15.50:0/4155171744 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72b810b560 con 0x7f72b80fd340 2023-12-25T20:06:49.221 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:49.035+0000 7f72a67fc700 1 -- 172.21.15.50:0/4155171744 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f72a800f070 con 0x7f72b80fd340 2023-12-25T20:06:49.221 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:49.035+0000 7f72bfae0700 1 -- 172.21.15.50:0/4155171744 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72b8045ad0 con 0x7f72b80fd340 2023-12-25T20:06:49.221 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:49.036+0000 7f72a67fc700 1 --2- 172.21.15.50:0/4155171744 >> [v2:172.21.15.50:6800/4046643232,v1:172.21.15.50:6801/4046643232] conn(0x7f72a002fdb0 0x7f72a0032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:06:49.221 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:49.036+0000 7f72a67fc700 1 -- 172.21.15.50:0/4155171744 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f72a80411d0 con 0x7f72b80fd340 2023-12-25T20:06:49.221 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:49.036+0000 7f72bd07b700 1 --2- 172.21.15.50:0/4155171744 >> [v2:172.21.15.50:6800/4046643232,v1:172.21.15.50:6801/4046643232] conn(0x7f72a002fdb0 0x7f72a0032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:06:49.222 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:49.036+0000 7f72bd07b700 1 --2- 172.21.15.50:0/4155171744 >> [v2:172.21.15.50:6800/4046643232,v1:172.21.15.50:6801/4046643232] conn(0x7f72a002fdb0 0x7f72a0032270 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f72b40096a0 tx=0x7f72b4006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:06:49.222 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:49.040+0000 7f72a67fc700 1 -- 172.21.15.50:0/4155171744 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f72a8022310 con 0x7f72b80fd340 2023-12-25T20:06:49.233 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:49.232+0000 7f72bfae0700 1 -- 172.21.15.50:0/4155171744 --> [v2:172.21.15.50:6800/4046643232,v1:172.21.15.50:6801/4046643232] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}) v1 -- 0x7f72b810bea0 con 0x7f72a002fdb0 2023-12-25T20:06:49.237 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:49.236+0000 7f72a67fc700 1 -- 172.21.15.50:0/4155171744 <== mgr.14114 v2:172.21.15.50:6800/4046643232 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f72b810bea0 con 0x7f72a002fdb0 2023-12-25T20:06:49.237 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: Scheduled mon update... 2023-12-25T20:06:49.239 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:49.238+0000 7f72bfae0700 1 -- 172.21.15.50:0/4155171744 >> [v2:172.21.15.50:6800/4046643232,v1:172.21.15.50:6801/4046643232] conn(0x7f72a002fdb0 msgr2=0x7f72a0032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:06:49.239 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:49.238+0000 7f72bfae0700 1 --2- 172.21.15.50:0/4155171744 >> [v2:172.21.15.50:6800/4046643232,v1:172.21.15.50:6801/4046643232] conn(0x7f72a002fdb0 0x7f72a0032270 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f72b40096a0 tx=0x7f72b4006b40).stop 2023-12-25T20:06:49.240 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:49.238+0000 7f72bfae0700 1 -- 172.21.15.50:0/4155171744 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f72b80fd340 msgr2=0x7f72b810aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:06:49.240 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:49.238+0000 7f72bfae0700 1 --2- 172.21.15.50:0/4155171744 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f72b80fd340 0x7f72b810aa60 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f72a8002f40 tx=0x7f72a8006ee0).stop 2023-12-25T20:06:49.240 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:49.238+0000 7f72bfae0700 1 -- 172.21.15.50:0/4155171744 shutdown_connections 2023-12-25T20:06:49.240 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:49.238+0000 7f72bfae0700 1 --2- 172.21.15.50:0/4155171744 >> [v2:172.21.15.50:6800/4046643232,v1:172.21.15.50:6801/4046643232] conn(0x7f72a002fdb0 0x7f72a0032270 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:06:49.240 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:49.238+0000 7f72bfae0700 1 --2- 172.21.15.50:0/4155171744 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f72b80fd340 0x7f72b810aa60 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:06:49.240 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:49.238+0000 7f72bfae0700 1 -- 172.21.15.50:0/4155171744 >> 172.21.15.50:0/4155171744 conn(0x7f72b80f5b40 msgr2=0x7f72b80ff2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:06:49.240 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:49.238+0000 7f72bfae0700 1 -- 172.21.15.50:0/4155171744 shutdown_connections 2023-12-25T20:06:49.240 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:49.238+0000 7f72bfae0700 1 -- 172.21.15.50:0/4155171744 wait complete. 2023-12-25T20:06:50.719 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:50 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:06:49.234324+0000 mgr.smithi050.nbzhun (mgr.14114) 10 : audit [DBG] from='client.14134 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2023-12-25T20:06:50.719 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:50 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cephadm 2023-12-25T20:06:49.235082+0000 mgr.smithi050.nbzhun (mgr.14114) 11 : cephadm [INF] Saving service mon spec with placement count:5 2023-12-25T20:06:50.719 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:50 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:06:49.237278+0000 mon.smithi050 (mon.0) 56 : audit [INF] from='mgr.14114 172.21.15.50:0/3381094233' entity='mgr.smithi050.nbzhun' 2023-12-25T20:06:51.107 INFO:teuthology.orchestra.run.smithi050.stderr:Deploying mgr service with default placement... 2023-12-25T20:06:51.107 INFO:teuthology.orchestra.run.smithi050.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi050 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/02c8f25c-a361-11ee-95a5-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmps7kf4jlu:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpu2g3lwcx:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply mgr 2023-12-25T20:06:52.429 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:52.256+0000 7f5931d1a700 1 Processor -- start 2023-12-25T20:06:52.429 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:52.256+0000 7f5931d1a700 1 -- start start 2023-12-25T20:06:52.429 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:52.256+0000 7f5931d1a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f592c070430 0x7f592c0fe730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-25T20:06:52.429 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:52.256+0000 7f5931d1a700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f592c0fec20 con 0x7f592c070430 2023-12-25T20:06:52.429 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:52.256+0000 7f5930d18700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f592c070430 0x7f592c0fe730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:06:52.430 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:52.257+0000 7f5930d18700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f592c070430 0x7f592c0fe730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:47776/0 (socket says 172.21.15.50:47776) 2023-12-25T20:06:52.430 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:52.257+0000 7f5930d18700 1 -- 172.21.15.50:0/2189835884 learned_addr learned my addr 172.21.15.50:0/2189835884 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-25T20:06:52.430 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:52.257+0000 7f5930d18700 1 -- 172.21.15.50:0/2189835884 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f592c0fed60 con 0x7f592c070430 2023-12-25T20:06:52.430 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:52.257+0000 7f5930d18700 1 --2- 172.21.15.50:0/2189835884 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f592c070430 0x7f592c0fe730 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f592000b5f0 tx=0x7f5920006ee0).ready entity=mon.0 client_cookie=81283f96ee2eb507 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:06:52.430 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:52.257+0000 7f592b7fe700 1 -- 172.21.15.50:0/2189835884 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f592000d600 con 0x7f592c070430 2023-12-25T20:06:52.430 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:52.257+0000 7f592b7fe700 1 -- 172.21.15.50:0/2189835884 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f592000dbd0 con 0x7f592c070430 2023-12-25T20:06:52.430 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:52.257+0000 7f592b7fe700 1 -- 172.21.15.50:0/2189835884 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5920011c00 con 0x7f592c070430 2023-12-25T20:06:52.430 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:52.258+0000 7f5931d1a700 1 -- 172.21.15.50:0/2189835884 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f592c070430 msgr2=0x7f592c0fe730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:06:52.430 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:52.258+0000 7f5931d1a700 1 --2- 172.21.15.50:0/2189835884 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f592c070430 0x7f592c0fe730 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f592000b5f0 tx=0x7f5920006ee0).stop 2023-12-25T20:06:52.430 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:52.258+0000 7f5931d1a700 1 -- 172.21.15.50:0/2189835884 shutdown_connections 2023-12-25T20:06:52.430 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:52.258+0000 7f5931d1a700 1 --2- 172.21.15.50:0/2189835884 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f592c070430 0x7f592c0fe730 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:06:52.430 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:52.258+0000 7f5931d1a700 1 -- 172.21.15.50:0/2189835884 >> 172.21.15.50:0/2189835884 conn(0x7f592c0f6210 msgr2=0x7f592c0f8630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:06:52.430 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:52.258+0000 7f5931d1a700 1 -- 172.21.15.50:0/2189835884 shutdown_connections 2023-12-25T20:06:52.430 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:52.258+0000 7f5931d1a700 1 -- 172.21.15.50:0/2189835884 wait complete. 2023-12-25T20:06:52.431 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:52.258+0000 7f5931d1a700 1 Processor -- start 2023-12-25T20:06:52.431 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:52.258+0000 7f5931d1a700 1 -- start start 2023-12-25T20:06:52.431 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:52.259+0000 7f5931d1a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f592c070430 0x7f592c10a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:06:52.431 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:52.259+0000 7f5931d1a700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f592c10ad20 con 0x7f592c070430 2023-12-25T20:06:52.431 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:52.259+0000 7f5930d18700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f592c070430 0x7f592c10a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:06:52.431 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:52.259+0000 7f5930d18700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f592c070430 0x7f592c10a830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:47786/0 (socket says 172.21.15.50:47786) 2023-12-25T20:06:52.431 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:52.259+0000 7f5930d18700 1 -- 172.21.15.50:0/751588311 learned_addr learned my addr 172.21.15.50:0/751588311 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-25T20:06:52.431 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:52.259+0000 7f5930d18700 1 -- 172.21.15.50:0/751588311 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f592000b200 con 0x7f592c070430 2023-12-25T20:06:52.431 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:52.259+0000 7f5930d18700 1 --2- 172.21.15.50:0/751588311 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f592c070430 0x7f592c10a830 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f592000b190 tx=0x7f5920006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:06:52.431 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:52.259+0000 7f59297fa700 1 -- 172.21.15.50:0/751588311 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f592001b410 con 0x7f592c070430 2023-12-25T20:06:52.431 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:52.259+0000 7f5931d1a700 1 -- 172.21.15.50:0/751588311 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f592c10af20 con 0x7f592c070430 2023-12-25T20:06:52.431 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:52.259+0000 7f59297fa700 1 -- 172.21.15.50:0/751588311 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f592001b9e0 con 0x7f592c070430 2023-12-25T20:06:52.431 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:52.259+0000 7f59297fa700 1 -- 172.21.15.50:0/751588311 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f592001a880 con 0x7f592c070430 2023-12-25T20:06:52.431 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:52.259+0000 7f5931d1a700 1 -- 172.21.15.50:0/751588311 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f592c10b330 con 0x7f592c070430 2023-12-25T20:06:52.432 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:52.260+0000 7f59297fa700 1 -- 172.21.15.50:0/751588311 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f592000f070 con 0x7f592c070430 2023-12-25T20:06:52.432 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:52.260+0000 7f59297fa700 1 --2- 172.21.15.50:0/751588311 >> [v2:172.21.15.50:6800/4046643232,v1:172.21.15.50:6801/4046643232] conn(0x7f591002fd50 0x7f5910032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:06:52.432 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:52.260+0000 7f5931d1a700 1 -- 172.21.15.50:0/751588311 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f592c045ad0 con 0x7f592c070430 2023-12-25T20:06:52.432 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:52.260+0000 7f59297fa700 1 -- 172.21.15.50:0/751588311 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f5920041290 con 0x7f592c070430 2023-12-25T20:06:52.432 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:52.261+0000 7f592bfff700 1 --2- 172.21.15.50:0/751588311 >> [v2:172.21.15.50:6800/4046643232,v1:172.21.15.50:6801/4046643232] conn(0x7f591002fd50 0x7f5910032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:06:52.432 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:52.261+0000 7f592bfff700 1 --2- 172.21.15.50:0/751588311 >> [v2:172.21.15.50:6800/4046643232,v1:172.21.15.50:6801/4046643232] conn(0x7f591002fd50 0x7f5910032210 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f59180096a0 tx=0x7f5918006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:06:52.432 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:52.265+0000 7f59297fa700 1 -- 172.21.15.50:0/751588311 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5920016070 con 0x7f592c070430 2023-12-25T20:06:52.454 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:52.453+0000 7f5931d1a700 1 -- 172.21.15.50:0/751588311 --> [v2:172.21.15.50:6800/4046643232,v1:172.21.15.50:6801/4046643232] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7f592c10bc70 con 0x7f591002fd50 2023-12-25T20:06:52.457 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:52.457+0000 7f59297fa700 1 -- 172.21.15.50:0/751588311 <== mgr.14114 v2:172.21.15.50:6800/4046643232 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f592c10bc70 con 0x7f591002fd50 2023-12-25T20:06:52.457 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: Scheduled mgr update... 2023-12-25T20:06:52.460 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:52.458+0000 7f5931d1a700 1 -- 172.21.15.50:0/751588311 >> [v2:172.21.15.50:6800/4046643232,v1:172.21.15.50:6801/4046643232] conn(0x7f591002fd50 msgr2=0x7f5910032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:06:52.460 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:52.458+0000 7f5931d1a700 1 --2- 172.21.15.50:0/751588311 >> [v2:172.21.15.50:6800/4046643232,v1:172.21.15.50:6801/4046643232] conn(0x7f591002fd50 0x7f5910032210 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f59180096a0 tx=0x7f5918006b40).stop 2023-12-25T20:06:52.460 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:52.458+0000 7f5931d1a700 1 -- 172.21.15.50:0/751588311 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f592c070430 msgr2=0x7f592c10a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:06:52.460 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:52.458+0000 7f5931d1a700 1 --2- 172.21.15.50:0/751588311 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f592c070430 0x7f592c10a830 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f592000b190 tx=0x7f5920006ee0).stop 2023-12-25T20:06:52.460 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:52.458+0000 7f5931d1a700 1 -- 172.21.15.50:0/751588311 shutdown_connections 2023-12-25T20:06:52.460 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:52.458+0000 7f5931d1a700 1 --2- 172.21.15.50:0/751588311 >> [v2:172.21.15.50:6800/4046643232,v1:172.21.15.50:6801/4046643232] conn(0x7f591002fd50 0x7f5910032210 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:06:52.460 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:52.458+0000 7f5931d1a700 1 --2- 172.21.15.50:0/751588311 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f592c070430 0x7f592c10a830 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:06:52.460 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:52.458+0000 7f5931d1a700 1 -- 172.21.15.50:0/751588311 >> 172.21.15.50:0/751588311 conn(0x7f592c0f6210 msgr2=0x7f592c0f7cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:06:52.461 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:52.459+0000 7f5931d1a700 1 -- 172.21.15.50:0/751588311 shutdown_connections 2023-12-25T20:06:52.461 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:52.459+0000 7f5931d1a700 1 -- 172.21.15.50:0/751588311 wait complete. 2023-12-25T20:06:53.719 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:53 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:06:52.455217+0000 mgr.smithi050.nbzhun (mgr.14114) 12 : audit [DBG] from='client.14136 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2023-12-25T20:06:53.719 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:53 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cephadm 2023-12-25T20:06:52.455763+0000 mgr.smithi050.nbzhun (mgr.14114) 13 : cephadm [INF] Saving service mgr spec with placement count:2 2023-12-25T20:06:53.719 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:53 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:06:52.457991+0000 mon.smithi050 (mon.0) 57 : audit [INF] from='mgr.14114 172.21.15.50:0/3381094233' entity='mgr.smithi050.nbzhun' 2023-12-25T20:06:54.487 INFO:teuthology.orchestra.run.smithi050.stderr:Deploying crash service with default placement... 2023-12-25T20:06:54.487 INFO:teuthology.orchestra.run.smithi050.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi050 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/02c8f25c-a361-11ee-95a5-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmps7kf4jlu:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpu2g3lwcx:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply crash 2023-12-25T20:06:54.958 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:54.956+0000 7f9946c62700 1 Processor -- start 2023-12-25T20:06:54.958 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:54.956+0000 7f9946c62700 1 -- start start 2023-12-25T20:06:54.959 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:54.957+0000 7f9946c62700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f99400fab90 0x7f99400fafb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-25T20:06:54.959 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:54.957+0000 7f9946c62700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99400fb4a0 con 0x7f99400fab90 2023-12-25T20:06:54.959 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:54.957+0000 7f99449fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f99400fab90 0x7f99400fafb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:06:54.959 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:54.957+0000 7f99449fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f99400fab90 0x7f99400fafb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:47794/0 (socket says 172.21.15.50:47794) 2023-12-25T20:06:54.959 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:54.957+0000 7f99449fe700 1 -- 172.21.15.50:0/2942779100 learned_addr learned my addr 172.21.15.50:0/2942779100 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-25T20:06:54.959 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:54.957+0000 7f99449fe700 1 -- 172.21.15.50:0/2942779100 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99400fb5e0 con 0x7f99400fab90 2023-12-25T20:06:54.959 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:54.957+0000 7f99449fe700 1 --2- 172.21.15.50:0/2942779100 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f99400fab90 0x7f99400fafb0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f993400b870 tx=0x7f9934006f70).ready entity=mon.0 client_cookie=f99e18dc0e6ed0e server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:06:54.959 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:54.957+0000 7f993f7fe700 1 -- 172.21.15.50:0/2942779100 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f993400da20 con 0x7f99400fab90 2023-12-25T20:06:54.959 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:54.957+0000 7f993f7fe700 1 -- 172.21.15.50:0/2942779100 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f9934010470 con 0x7f99400fab90 2023-12-25T20:06:54.959 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:54.957+0000 7f993f7fe700 1 -- 172.21.15.50:0/2942779100 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f993400f5b0 con 0x7f99400fab90 2023-12-25T20:06:54.961 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:54.958+0000 7f9946c62700 1 -- 172.21.15.50:0/2942779100 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f99400fab90 msgr2=0x7f99400fafb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:06:54.961 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:54.958+0000 7f9946c62700 1 --2- 172.21.15.50:0/2942779100 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f99400fab90 0x7f99400fafb0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f993400b870 tx=0x7f9934006f70).stop 2023-12-25T20:06:54.961 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:54.958+0000 7f9946c62700 1 -- 172.21.15.50:0/2942779100 shutdown_connections 2023-12-25T20:06:54.961 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:54.958+0000 7f9946c62700 1 --2- 172.21.15.50:0/2942779100 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f99400fab90 0x7f99400fafb0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:06:54.962 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:54.958+0000 7f9946c62700 1 -- 172.21.15.50:0/2942779100 >> 172.21.15.50:0/2942779100 conn(0x7f99400f63b0 msgr2=0x7f99400f87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:06:54.962 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:54.958+0000 7f9946c62700 1 -- 172.21.15.50:0/2942779100 shutdown_connections 2023-12-25T20:06:54.962 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:54.958+0000 7f9946c62700 1 -- 172.21.15.50:0/2942779100 wait complete. 2023-12-25T20:06:54.962 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:54.959+0000 7f9946c62700 1 Processor -- start 2023-12-25T20:06:54.962 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:54.959+0000 7f9946c62700 1 -- start start 2023-12-25T20:06:54.962 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:54.959+0000 7f9946c62700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f99400fab90 0x7f9940108730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:06:54.962 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:54.959+0000 7f9946c62700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9940108c20 con 0x7f99400fab90 2023-12-25T20:06:54.962 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:54.959+0000 7f99449fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f99400fab90 0x7f9940108730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:06:54.962 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:54.960+0000 7f99449fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f99400fab90 0x7f9940108730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:47796/0 (socket says 172.21.15.50:47796) 2023-12-25T20:06:54.962 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:54.960+0000 7f99449fe700 1 -- 172.21.15.50:0/4159774699 learned_addr learned my addr 172.21.15.50:0/4159774699 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-25T20:06:54.962 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:54.960+0000 7f99449fe700 1 -- 172.21.15.50:0/4159774699 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f993400b480 con 0x7f99400fab90 2023-12-25T20:06:54.962 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:54.960+0000 7f99449fe700 1 --2- 172.21.15.50:0/4159774699 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f99400fab90 0x7f9940108730 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f9934006800 tx=0x7f9934006f70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:06:54.962 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:54.960+0000 7f993d7fa700 1 -- 172.21.15.50:0/4159774699 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f993400f820 con 0x7f99400fab90 2023-12-25T20:06:54.962 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:54.960+0000 7f993d7fa700 1 -- 172.21.15.50:0/4159774699 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f993400fdf0 con 0x7f99400fab90 2023-12-25T20:06:54.963 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:54.960+0000 7f993d7fa700 1 -- 172.21.15.50:0/4159774699 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9934019d50 con 0x7f99400fab90 2023-12-25T20:06:54.963 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:54.960+0000 7f9946c62700 1 -- 172.21.15.50:0/4159774699 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9940108e20 con 0x7f99400fab90 2023-12-25T20:06:54.963 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:54.960+0000 7f9946c62700 1 -- 172.21.15.50:0/4159774699 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9940109230 con 0x7f99400fab90 2023-12-25T20:06:54.963 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:54.961+0000 7f993d7fa700 1 -- 172.21.15.50:0/4159774699 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f9934016030 con 0x7f99400fab90 2023-12-25T20:06:54.964 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:54.961+0000 7f993d7fa700 1 --2- 172.21.15.50:0/4159774699 >> [v2:172.21.15.50:6800/4046643232,v1:172.21.15.50:6801/4046643232] conn(0x7f992402fdb0 0x7f9924032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:06:54.964 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:54.961+0000 7f993d7fa700 1 -- 172.21.15.50:0/4159774699 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f99340440f0 con 0x7f99400fab90 2023-12-25T20:06:54.964 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:54.961+0000 7f993ffff700 1 --2- 172.21.15.50:0/4159774699 >> [v2:172.21.15.50:6800/4046643232,v1:172.21.15.50:6801/4046643232] conn(0x7f992402fdb0 0x7f9924032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:06:54.964 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:54.962+0000 7f993ffff700 1 --2- 172.21.15.50:0/4159774699 >> [v2:172.21.15.50:6800/4046643232,v1:172.21.15.50:6801/4046643232] conn(0x7f992402fdb0 0x7f9924032270 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f992c0096a0 tx=0x7f992c006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:06:54.964 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:54.962+0000 7f9946c62700 1 -- 172.21.15.50:0/4159774699 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9940059100 con 0x7f99400fab90 2023-12-25T20:06:54.967 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:54.966+0000 7f993d7fa700 1 -- 172.21.15.50:0/4159774699 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9934014020 con 0x7f99400fab90 2023-12-25T20:06:55.155 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:55.154+0000 7f9946c62700 1 -- 172.21.15.50:0/4159774699 --> [v2:172.21.15.50:6800/4046643232,v1:172.21.15.50:6801/4046643232] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}) v1 -- 0x7f99401026c0 con 0x7f992402fdb0 2023-12-25T20:06:55.159 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:55.158+0000 7f993d7fa700 1 -- 172.21.15.50:0/4159774699 <== mgr.14114 v2:172.21.15.50:6800/4046643232 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7f99401026c0 con 0x7f992402fdb0 2023-12-25T20:06:55.159 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: Scheduled crash update... 2023-12-25T20:06:55.161 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:55.160+0000 7f9946c62700 1 -- 172.21.15.50:0/4159774699 >> [v2:172.21.15.50:6800/4046643232,v1:172.21.15.50:6801/4046643232] conn(0x7f992402fdb0 msgr2=0x7f9924032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:06:55.161 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:55.160+0000 7f9946c62700 1 --2- 172.21.15.50:0/4159774699 >> [v2:172.21.15.50:6800/4046643232,v1:172.21.15.50:6801/4046643232] conn(0x7f992402fdb0 0x7f9924032270 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f992c0096a0 tx=0x7f992c006b40).stop 2023-12-25T20:06:55.161 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:55.160+0000 7f9946c62700 1 -- 172.21.15.50:0/4159774699 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f99400fab90 msgr2=0x7f9940108730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:06:55.161 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:55.160+0000 7f9946c62700 1 --2- 172.21.15.50:0/4159774699 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f99400fab90 0x7f9940108730 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f9934006800 tx=0x7f9934006f70).stop 2023-12-25T20:06:55.162 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:55.160+0000 7f9946c62700 1 -- 172.21.15.50:0/4159774699 shutdown_connections 2023-12-25T20:06:55.162 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:55.160+0000 7f9946c62700 1 --2- 172.21.15.50:0/4159774699 >> [v2:172.21.15.50:6800/4046643232,v1:172.21.15.50:6801/4046643232] conn(0x7f992402fdb0 0x7f9924032270 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:06:55.162 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:55.160+0000 7f9946c62700 1 --2- 172.21.15.50:0/4159774699 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f99400fab90 0x7f9940108730 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:06:55.162 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:55.160+0000 7f9946c62700 1 -- 172.21.15.50:0/4159774699 >> 172.21.15.50:0/4159774699 conn(0x7f99400f63b0 msgr2=0x7f99400ff920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:06:55.162 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:55.160+0000 7f9946c62700 1 -- 172.21.15.50:0/4159774699 shutdown_connections 2023-12-25T20:06:55.162 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:55.160+0000 7f9946c62700 1 -- 172.21.15.50:0/4159774699 wait complete. 2023-12-25T20:06:55.706 INFO:teuthology.orchestra.run.smithi050.stderr:Enabling mgr prometheus module... 2023-12-25T20:06:55.706 INFO:teuthology.orchestra.run.smithi050.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi050 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/02c8f25c-a361-11ee-95a5-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmps7kf4jlu:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpu2g3lwcx:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable prometheus 2023-12-25T20:06:56.162 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:56.161+0000 7f21f2dd8700 1 Processor -- start 2023-12-25T20:06:56.162 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:56.161+0000 7f21f2dd8700 1 -- start start 2023-12-25T20:06:56.162 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:56.161+0000 7f21f2dd8700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f21ec070430 0x7f21ec0fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-25T20:06:56.162 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:56.161+0000 7f21f2dd8700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21ec0fedd0 con 0x7f21ec070430 2023-12-25T20:06:56.163 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:56.162+0000 7f21f0b74700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f21ec070430 0x7f21ec0fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:06:56.163 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:56.162+0000 7f21f0b74700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f21ec070430 0x7f21ec0fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:47810/0 (socket says 172.21.15.50:47810) 2023-12-25T20:06:56.163 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:56.162+0000 7f21f0b74700 1 -- 172.21.15.50:0/1465308853 learned_addr learned my addr 172.21.15.50:0/1465308853 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-25T20:06:56.163 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:56.162+0000 7f21f0b74700 1 -- 172.21.15.50:0/1465308853 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21ec0fef10 con 0x7f21ec070430 2023-12-25T20:06:56.164 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:56.162+0000 7f21f0b74700 1 --2- 172.21.15.50:0/1465308853 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f21ec070430 0x7f21ec0fe8e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f21e000b5f0 tx=0x7f21e0006ee0).ready entity=mon.0 client_cookie=dca911524cbfac2e server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:06:56.164 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:56.162+0000 7f21eb7fe700 1 -- 172.21.15.50:0/1465308853 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f21e000d600 con 0x7f21ec070430 2023-12-25T20:06:56.164 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:56.162+0000 7f21eb7fe700 1 -- 172.21.15.50:0/1465308853 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f21e000dc10 con 0x7f21ec070430 2023-12-25T20:06:56.164 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:56.162+0000 7f21eb7fe700 1 -- 172.21.15.50:0/1465308853 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f21e0011cd0 con 0x7f21ec070430 2023-12-25T20:06:56.164 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:56.163+0000 7f21f2dd8700 1 -- 172.21.15.50:0/1465308853 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f21ec070430 msgr2=0x7f21ec0fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:06:56.164 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:56.163+0000 7f21f2dd8700 1 --2- 172.21.15.50:0/1465308853 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f21ec070430 0x7f21ec0fe8e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f21e000b5f0 tx=0x7f21e0006ee0).stop 2023-12-25T20:06:56.164 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:56.163+0000 7f21f2dd8700 1 -- 172.21.15.50:0/1465308853 shutdown_connections 2023-12-25T20:06:56.165 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:56.163+0000 7f21f2dd8700 1 --2- 172.21.15.50:0/1465308853 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f21ec070430 0x7f21ec0fe8e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:06:56.165 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:56.163+0000 7f21f2dd8700 1 -- 172.21.15.50:0/1465308853 >> 172.21.15.50:0/1465308853 conn(0x7f21ec0f63c0 msgr2=0x7f21ec0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:06:56.165 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:56.163+0000 7f21f2dd8700 1 -- 172.21.15.50:0/1465308853 shutdown_connections 2023-12-25T20:06:56.165 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:56.163+0000 7f21f2dd8700 1 -- 172.21.15.50:0/1465308853 wait complete. 2023-12-25T20:06:56.165 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:56.164+0000 7f21f2dd8700 1 Processor -- start 2023-12-25T20:06:56.165 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:56.164+0000 7f21f2dd8700 1 -- start start 2023-12-25T20:06:56.166 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:56.164+0000 7f21f2dd8700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f21ec070430 0x7f21ec0fc360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:06:56.166 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:56.164+0000 7f21f2dd8700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21ec0fa7e0 con 0x7f21ec070430 2023-12-25T20:06:56.166 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:56.164+0000 7f21f0b74700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f21ec070430 0x7f21ec0fc360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:06:56.166 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:56.164+0000 7f21f0b74700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f21ec070430 0x7f21ec0fc360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:47818/0 (socket says 172.21.15.50:47818) 2023-12-25T20:06:56.166 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:56.165+0000 7f21f0b74700 1 -- 172.21.15.50:0/3167729383 learned_addr learned my addr 172.21.15.50:0/3167729383 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-25T20:06:56.166 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:56.165+0000 7f21f0b74700 1 -- 172.21.15.50:0/3167729383 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21e000b200 con 0x7f21ec070430 2023-12-25T20:06:56.166 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:56.165+0000 7f21f0b74700 1 --2- 172.21.15.50:0/3167729383 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f21ec070430 0x7f21ec0fc360 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f21e0006800 tx=0x7f21e0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:06:56.166 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:56.165+0000 7f21e97fa700 1 -- 172.21.15.50:0/3167729383 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f21e0011df0 con 0x7f21ec070430 2023-12-25T20:06:56.166 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:56.165+0000 7f21f2dd8700 1 -- 172.21.15.50:0/3167729383 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21ec0fa980 con 0x7f21ec070430 2023-12-25T20:06:56.167 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:56.165+0000 7f21f2dd8700 1 -- 172.21.15.50:0/3167729383 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21ec0fad90 con 0x7f21ec070430 2023-12-25T20:06:56.167 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:56.167+0000 7f21e97fa700 1 -- 172.21.15.50:0/3167729383 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f21e001b8c0 con 0x7f21ec070430 2023-12-25T20:06:56.167 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:56.166+0000 7f21f2dd8700 1 -- 172.21.15.50:0/3167729383 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21ec045ad0 con 0x7f21ec070430 2023-12-25T20:06:56.168 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:56.167+0000 7f21e97fa700 1 -- 172.21.15.50:0/3167729383 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f21e001a830 con 0x7f21ec070430 2023-12-25T20:06:56.168 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:56.167+0000 7f21e97fa700 1 -- 172.21.15.50:0/3167729383 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f21e000f070 con 0x7f21ec070430 2023-12-25T20:06:56.168 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:56.167+0000 7f21e97fa700 1 --2- 172.21.15.50:0/3167729383 >> [v2:172.21.15.50:6800/4046643232,v1:172.21.15.50:6801/4046643232] conn(0x7f21d002fe50 0x7f21d0032310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:06:56.168 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:56.167+0000 7f21e97fa700 1 -- 172.21.15.50:0/3167729383 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f21e0041ea0 con 0x7f21ec070430 2023-12-25T20:06:56.170 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:56.170+0000 7f21ebfff700 1 --2- 172.21.15.50:0/3167729383 >> [v2:172.21.15.50:6800/4046643232,v1:172.21.15.50:6801/4046643232] conn(0x7f21d002fe50 0x7f21d0032310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:06:56.171 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:56.170+0000 7f21ebfff700 1 --2- 172.21.15.50:0/3167729383 >> [v2:172.21.15.50:6800/4046643232,v1:172.21.15.50:6801/4046643232] conn(0x7f21d002fe50 0x7f21d0032310 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f21d80096a0 tx=0x7f21d8006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:06:56.171 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:56.171+0000 7f21e97fa700 1 -- 172.21.15.50:0/3167729383 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f21e0010bc0 con 0x7f21ec070430 2023-12-25T20:06:56.376 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:56.375+0000 7f21f2dd8700 1 -- 172.21.15.50:0/3167729383 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "prometheus"} v 0) v1 -- 0x7f21ec1049e0 con 0x7f21ec070430 2023-12-25T20:06:56.469 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:56 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:06:55.156590+0000 mgr.smithi050.nbzhun (mgr.14114) 14 : audit [DBG] from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2023-12-25T20:06:56.469 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:56 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cephadm 2023-12-25T20:06:55.157275+0000 mgr.smithi050.nbzhun (mgr.14114) 15 : cephadm [INF] Saving service crash spec with placement * 2023-12-25T20:06:56.469 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:56 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:06:55.159524+0000 mon.smithi050 (mon.0) 58 : audit [INF] from='mgr.14114 172.21.15.50:0/3381094233' entity='mgr.smithi050.nbzhun' 2023-12-25T20:06:56.469 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:56 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:06:55.561113+0000 mon.smithi050 (mon.0) 59 : audit [INF] from='mgr.14114 172.21.15.50:0/3381094233' entity='mgr.smithi050.nbzhun' 2023-12-25T20:06:56.469 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:56 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:06:55.788658+0000 mon.smithi050 (mon.0) 60 : audit [INF] from='mgr.14114 172.21.15.50:0/3381094233' entity='mgr.smithi050.nbzhun' 2023-12-25T20:06:57.161 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:57.160+0000 7f21e97fa700 1 -- 172.21.15.50:0/3167729383 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mgrmap(e 9) v1 ==== 36134+0+0 (secure 0 0 0) 0x7f21e0044020 con 0x7f21ec070430 2023-12-25T20:06:57.162 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:57.162+0000 7f21e97fa700 1 -- 172.21.15.50:0/3167729383 <== mon.0 v2:172.21.15.50:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "prometheus"}]=0 v9) v1 ==== 89+0+0 (secure 0 0 0) 0x7f21e0047510 con 0x7f21ec070430 2023-12-25T20:06:57.166 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:57.164+0000 7f21f2dd8700 1 -- 172.21.15.50:0/3167729383 >> [v2:172.21.15.50:6800/4046643232,v1:172.21.15.50:6801/4046643232] conn(0x7f21d002fe50 msgr2=0x7f21d0032310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:06:57.166 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:57.164+0000 7f21f2dd8700 1 --2- 172.21.15.50:0/3167729383 >> [v2:172.21.15.50:6800/4046643232,v1:172.21.15.50:6801/4046643232] conn(0x7f21d002fe50 0x7f21d0032310 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f21d80096a0 tx=0x7f21d8006b40).stop 2023-12-25T20:06:57.166 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:57.164+0000 7f21f2dd8700 1 -- 172.21.15.50:0/3167729383 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f21ec070430 msgr2=0x7f21ec0fc360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:06:57.166 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:57.164+0000 7f21f2dd8700 1 --2- 172.21.15.50:0/3167729383 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f21ec070430 0x7f21ec0fc360 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f21e0006800 tx=0x7f21e0006ee0).stop 2023-12-25T20:06:57.166 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:57.164+0000 7f21f2dd8700 1 -- 172.21.15.50:0/3167729383 shutdown_connections 2023-12-25T20:06:57.166 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:57.164+0000 7f21f2dd8700 1 --2- 172.21.15.50:0/3167729383 >> [v2:172.21.15.50:6800/4046643232,v1:172.21.15.50:6801/4046643232] conn(0x7f21d002fe50 0x7f21d0032310 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:06:57.166 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:57.164+0000 7f21f2dd8700 1 --2- 172.21.15.50:0/3167729383 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f21ec070430 0x7f21ec0fc360 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:06:57.166 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:57.164+0000 7f21f2dd8700 1 -- 172.21.15.50:0/3167729383 >> 172.21.15.50:0/3167729383 conn(0x7f21ec0f63c0 msgr2=0x7f21ec0f7f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:06:57.166 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:57.164+0000 7f21f2dd8700 1 -- 172.21.15.50:0/3167729383 shutdown_connections 2023-12-25T20:06:57.167 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:57.164+0000 7f21f2dd8700 1 -- 172.21.15.50:0/3167729383 wait complete. 2023-12-25T20:06:57.469 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:57 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:06:56.377419+0000 mon.smithi050 (mon.0) 61 : audit [INF] from='client.? 172.21.15.50:0/3167729383' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2023-12-25T20:06:57.469 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:57 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:06:56.535422+0000 mon.smithi050 (mon.0) 62 : audit [INF] from='mgr.14114 172.21.15.50:0/3381094233' entity='mgr.smithi050.nbzhun' 2023-12-25T20:06:57.657 INFO:teuthology.orchestra.run.smithi050.stderr:Deploying prometheus service with default placement... 2023-12-25T20:06:57.657 INFO:teuthology.orchestra.run.smithi050.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi050 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/02c8f25c-a361-11ee-95a5-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmps7kf4jlu:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpu2g3lwcx:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply prometheus 2023-12-25T20:06:58.054 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:58.053+0000 7f40a179a700 1 Processor -- start 2023-12-25T20:06:58.054 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:58.053+0000 7f40a179a700 1 -- start start 2023-12-25T20:06:58.055 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:58.053+0000 7f40a179a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f409c0703d0 0x7f409c100d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-25T20:06:58.055 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:58.053+0000 7f40a179a700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f409c0fc730 con 0x7f409c0703d0 2023-12-25T20:06:58.055 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:58.053+0000 7f409affd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f409c0703d0 0x7f409c100d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:06:58.055 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:58.053+0000 7f409affd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f409c0703d0 0x7f409c100d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:44460/0 (socket says 172.21.15.50:44460) 2023-12-25T20:06:58.057 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:58.053+0000 7f409affd700 1 -- 172.21.15.50:0/2757259352 learned_addr learned my addr 172.21.15.50:0/2757259352 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-25T20:06:58.057 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:58.053+0000 7f409affd700 1 -- 172.21.15.50:0/2757259352 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f409c101230 con 0x7f409c0703d0 2023-12-25T20:06:58.057 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:58.054+0000 7f409affd700 1 --2- 172.21.15.50:0/2757259352 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f409c0703d0 0x7f409c100d40 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f408c00b5f0 tx=0x7f408c006ee0).ready entity=mon.0 client_cookie=ce7490ebd3de0c7b server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:06:58.057 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:58.054+0000 7f4099ffb700 1 -- 172.21.15.50:0/2757259352 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f408c00d600 con 0x7f409c0703d0 2023-12-25T20:06:58.057 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:58.054+0000 7f4099ffb700 1 -- 172.21.15.50:0/2757259352 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f408c00dc10 con 0x7f409c0703d0 2023-12-25T20:06:58.057 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:58.054+0000 7f4099ffb700 1 -- 172.21.15.50:0/2757259352 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f408c011cd0 con 0x7f409c0703d0 2023-12-25T20:06:58.058 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:58.054+0000 7f40a179a700 1 -- 172.21.15.50:0/2757259352 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f409c0703d0 msgr2=0x7f409c100d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:06:58.058 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:58.054+0000 7f40a179a700 1 --2- 172.21.15.50:0/2757259352 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f409c0703d0 0x7f409c100d40 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f408c00b5f0 tx=0x7f408c006ee0).stop 2023-12-25T20:06:58.058 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:58.054+0000 7f40a179a700 1 -- 172.21.15.50:0/2757259352 shutdown_connections 2023-12-25T20:06:58.058 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:58.054+0000 7f40a179a700 1 --2- 172.21.15.50:0/2757259352 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f409c0703d0 0x7f409c100d40 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:06:58.058 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:58.054+0000 7f40a179a700 1 -- 172.21.15.50:0/2757259352 >> 172.21.15.50:0/2757259352 conn(0x7f409c0f63c0 msgr2=0x7f409c0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:06:58.058 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:58.054+0000 7f40a179a700 1 -- 172.21.15.50:0/2757259352 shutdown_connections 2023-12-25T20:06:58.058 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:58.054+0000 7f40a179a700 1 -- 172.21.15.50:0/2757259352 wait complete. 2023-12-25T20:06:58.058 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:58.055+0000 7f40a179a700 1 Processor -- start 2023-12-25T20:06:58.058 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:58.055+0000 7f40a179a700 1 -- start start 2023-12-25T20:06:58.058 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:58.055+0000 7f40a179a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f409c0703d0 0x7f409c10cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:06:58.058 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:58.055+0000 7f40a179a700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f409c10d220 con 0x7f409c0703d0 2023-12-25T20:06:58.058 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:58.055+0000 7f409affd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f409c0703d0 0x7f409c10cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:06:58.060 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:58.055+0000 7f409affd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f409c0703d0 0x7f409c10cd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:44474/0 (socket says 172.21.15.50:44474) 2023-12-25T20:06:58.060 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:58.055+0000 7f409affd700 1 -- 172.21.15.50:0/3670368901 learned_addr learned my addr 172.21.15.50:0/3670368901 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-25T20:06:58.060 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:58.056+0000 7f409affd700 1 -- 172.21.15.50:0/3670368901 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f408c00b200 con 0x7f409c0703d0 2023-12-25T20:06:58.060 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:58.056+0000 7f409affd700 1 --2- 172.21.15.50:0/3670368901 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f409c0703d0 0x7f409c10cd30 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f408c00df90 tx=0x7f408c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:06:58.060 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:58.056+0000 7f40837fe700 1 -- 172.21.15.50:0/3670368901 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f408c011da0 con 0x7f409c0703d0 2023-12-25T20:06:58.060 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:58.056+0000 7f40837fe700 1 -- 172.21.15.50:0/3670368901 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f408c01b8c0 con 0x7f409c0703d0 2023-12-25T20:06:58.060 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:58.056+0000 7f40a179a700 1 -- 172.21.15.50:0/3670368901 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f409c10d420 con 0x7f409c0703d0 2023-12-25T20:06:58.060 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:58.056+0000 7f40837fe700 1 -- 172.21.15.50:0/3670368901 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f408c01a930 con 0x7f409c0703d0 2023-12-25T20:06:58.060 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:58.056+0000 7f40a179a700 1 -- 172.21.15.50:0/3670368901 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f409c10d830 con 0x7f409c0703d0 2023-12-25T20:06:58.060 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:58.057+0000 7f40837fe700 1 -- 172.21.15.50:0/3670368901 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 9) v1 ==== 36134+0+0 (secure 0 0 0) 0x7f408c00f070 con 0x7f409c0703d0 2023-12-25T20:06:58.061 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:58.057+0000 7f40837fe700 1 --2- 172.21.15.50:0/3670368901 >> [v2:172.21.15.50:6800/4046643232,v1:172.21.15.50:6801/4046643232] conn(0x7f408802fe00 0x7f40880322c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:06:58.061 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:58.057+0000 7f40837fe700 1 -- 172.21.15.50:0/3670368901 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f408c040eb0 con 0x7f409c0703d0 2023-12-25T20:06:58.061 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:58.057+0000 7f409a7fc700 1 -- 172.21.15.50:0/3670368901 >> [v2:172.21.15.50:6800/4046643232,v1:172.21.15.50:6801/4046643232] conn(0x7f408802fe00 msgr2=0x7f40880322c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:6800/4046643232 2023-12-25T20:06:58.061 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:58.057+0000 7f409a7fc700 1 --2- 172.21.15.50:0/3670368901 >> [v2:172.21.15.50:6800/4046643232,v1:172.21.15.50:6801/4046643232] conn(0x7f408802fe00 0x7f40880322c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-12-25T20:06:58.061 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:58.057+0000 7f40a179a700 1 -- 172.21.15.50:0/3670368901 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f409c045ad0 con 0x7f409c0703d0 2023-12-25T20:06:58.062 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:58.062+0000 7f40837fe700 1 -- 172.21.15.50:0/3670368901 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f408c016070 con 0x7f409c0703d0 2023-12-25T20:06:58.252 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:58.250+0000 7f40a179a700 1 -- 172.21.15.50:0/3670368901 --> [v2:172.21.15.50:6800/4046643232,v1:172.21.15.50:6801/4046643232] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7f409c106bd0 con 0x7f408802fe00 2023-12-25T20:06:58.258 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:58.258+0000 7f409a7fc700 1 -- 172.21.15.50:0/3670368901 >> [v2:172.21.15.50:6800/4046643232,v1:172.21.15.50:6801/4046643232] conn(0x7f408802fe00 msgr2=0x7f40880322c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:6800/4046643232 2023-12-25T20:06:58.258 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:58.258+0000 7f409a7fc700 1 --2- 172.21.15.50:0/3670368901 >> [v2:172.21.15.50:6800/4046643232,v1:172.21.15.50:6801/4046643232] conn(0x7f408802fe00 0x7f40880322c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-12-25T20:06:58.469 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:58 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:06:57.163295+0000 mon.smithi050 (mon.0) 63 : audit [INF] from='client.? 172.21.15.50:0/3167729383' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2023-12-25T20:06:58.469 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:06:58 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:06:57.163345+0000 mon.smithi050 (mon.0) 64 : cluster [DBG] mgrmap e9: smithi050.nbzhun(active, since 17s) 2023-12-25T20:06:58.660 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:58.659+0000 7f409a7fc700 1 -- 172.21.15.50:0/3670368901 >> [v2:172.21.15.50:6800/4046643232,v1:172.21.15.50:6801/4046643232] conn(0x7f408802fe00 msgr2=0x7f40880322c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:6800/4046643232 2023-12-25T20:06:58.660 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:58.659+0000 7f409a7fc700 1 --2- 172.21.15.50:0/3670368901 >> [v2:172.21.15.50:6800/4046643232,v1:172.21.15.50:6801/4046643232] conn(0x7f408802fe00 0x7f40880322c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-12-25T20:06:59.461 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:59.460+0000 7f409a7fc700 1 -- 172.21.15.50:0/3670368901 >> [v2:172.21.15.50:6800/4046643232,v1:172.21.15.50:6801/4046643232] conn(0x7f408802fe00 msgr2=0x7f40880322c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:6800/4046643232 2023-12-25T20:06:59.461 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:06:59.460+0000 7f409a7fc700 1 --2- 172.21.15.50:0/3670368901 >> [v2:172.21.15.50:6800/4046643232,v1:172.21.15.50:6801/4046643232] conn(0x7f408802fe00 0x7f40880322c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-12-25T20:07:01.063 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:01.061+0000 7f409a7fc700 1 -- 172.21.15.50:0/3670368901 >> [v2:172.21.15.50:6800/4046643232,v1:172.21.15.50:6801/4046643232] conn(0x7f408802fe00 msgr2=0x7f40880322c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:6800/4046643232 2023-12-25T20:07:01.063 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:01.061+0000 7f409a7fc700 1 --2- 172.21.15.50:0/3670368901 >> [v2:172.21.15.50:6800/4046643232,v1:172.21.15.50:6801/4046643232] conn(0x7f408802fe00 0x7f40880322c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 3.200000 2023-12-25T20:07:02.407 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:02.406+0000 7f40837fe700 1 -- 172.21.15.50:0/3670368901 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mgrmap(e 10) v1 ==== 35984+0+0 (secure 0 0 0) 0x7f408c010dd0 con 0x7f409c0703d0 2023-12-25T20:07:02.407 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:02.406+0000 7f40837fe700 1 -- 172.21.15.50:0/3670368901 >> [v2:172.21.15.50:6800/4046643232,v1:172.21.15.50:6801/4046643232] conn(0x7f408802fe00 msgr2=0x7f40880322c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-25T20:07:02.407 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:02.406+0000 7f40837fe700 1 --2- 172.21.15.50:0/3670368901 >> [v2:172.21.15.50:6800/4046643232,v1:172.21.15.50:6801/4046643232] conn(0x7f408802fe00 0x7f40880322c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-12-25T20:07:03.409 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:03.407+0000 7f40837fe700 1 -- 172.21.15.50:0/3670368901 <== mon.0 v2:172.21.15.50:3300/0 8 ==== mgrmap(e 11) v1 ==== 36054+0+0 (secure 0 0 0) 0x7f408c005d20 con 0x7f409c0703d0 2023-12-25T20:07:03.409 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:03.407+0000 7f40837fe700 1 --2- 172.21.15.50:0/3670368901 >> [v2:172.21.15.50:6800/2637522512,v1:172.21.15.50:6801/2637522512] conn(0x7f4088033c20 0x7f4088036010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:07:03.409 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:03.407+0000 7f40837fe700 1 -- 172.21.15.50:0/3670368901 --> [v2:172.21.15.50:6800/2637522512,v1:172.21.15.50:6801/2637522512] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7f409c106bd0 con 0x7f4088033c20 2023-12-25T20:07:03.409 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:03.407+0000 7f409a7fc700 1 --2- 172.21.15.50:0/3670368901 >> [v2:172.21.15.50:6800/2637522512,v1:172.21.15.50:6801/2637522512] conn(0x7f4088033c20 0x7f4088036010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:07:03.409 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:03.408+0000 7f409a7fc700 1 --2- 172.21.15.50:0/3670368901 >> [v2:172.21.15.50:6800/2637522512,v1:172.21.15.50:6801/2637522512] conn(0x7f4088033c20 0x7f4088036010 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f4090006fd0 tx=0x7f4090006d30).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:07:03.412 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:03.412+0000 7f40837fe700 1 -- 172.21.15.50:0/3670368901 <== mgr.14142 v2:172.21.15.50:6800/2637522512 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7f409c106bd0 con 0x7f4088033c20 2023-12-25T20:07:03.412 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: Scheduled prometheus update... 2023-12-25T20:07:03.414 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:03.413+0000 7f40a179a700 1 -- 172.21.15.50:0/3670368901 >> [v2:172.21.15.50:6800/2637522512,v1:172.21.15.50:6801/2637522512] conn(0x7f4088033c20 msgr2=0x7f4088036010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:07:03.414 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:03.413+0000 7f40a179a700 1 --2- 172.21.15.50:0/3670368901 >> [v2:172.21.15.50:6800/2637522512,v1:172.21.15.50:6801/2637522512] conn(0x7f4088033c20 0x7f4088036010 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f4090006fd0 tx=0x7f4090006d30).stop 2023-12-25T20:07:03.414 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:03.413+0000 7f40a179a700 1 -- 172.21.15.50:0/3670368901 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f409c0703d0 msgr2=0x7f409c10cd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:07:03.414 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:03.413+0000 7f40a179a700 1 --2- 172.21.15.50:0/3670368901 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f409c0703d0 0x7f409c10cd30 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f408c00df90 tx=0x7f408c006ee0).stop 2023-12-25T20:07:03.415 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:03.414+0000 7f40a179a700 1 -- 172.21.15.50:0/3670368901 shutdown_connections 2023-12-25T20:07:03.415 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:03.414+0000 7f40a179a700 1 --2- 172.21.15.50:0/3670368901 >> [v2:172.21.15.50:6800/2637522512,v1:172.21.15.50:6801/2637522512] conn(0x7f4088033c20 0x7f4088036010 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:07:03.415 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:03.414+0000 7f40a179a700 1 --2- 172.21.15.50:0/3670368901 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f409c0703d0 0x7f409c10cd30 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:07:03.415 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:03.414+0000 7f40a179a700 1 --2- 172.21.15.50:0/3670368901 >> [v2:172.21.15.50:6800/4046643232,v1:172.21.15.50:6801/4046643232] conn(0x7f408802fe00 0x7f40880322c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-12-25T20:07:03.416 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:03.414+0000 7f40a179a700 1 -- 172.21.15.50:0/3670368901 >> 172.21.15.50:0/3670368901 conn(0x7f409c0f63c0 msgr2=0x7f409c0f7ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:07:03.416 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:03.414+0000 7f40a179a700 1 -- 172.21.15.50:0/3670368901 shutdown_connections 2023-12-25T20:07:03.416 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:03.414+0000 7f40a179a700 1 -- 172.21.15.50:0/3670368901 wait complete. 2023-12-25T20:07:03.620 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:07:03 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:07:02.354974+0000 mon.smithi050 (mon.0) 65 : cluster [INF] Active manager daemon smithi050.nbzhun restarted 2023-12-25T20:07:03.621 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:07:03 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:07:02.355361+0000 mon.smithi050 (mon.0) 66 : cluster [INF] Activating manager daemon smithi050.nbzhun 2023-12-25T20:07:03.621 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:07:03 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:07:02.357020+0000 mon.smithi050 (mon.0) 67 : cluster [DBG] osdmap e3: 0 total, 0 up, 0 in 2023-12-25T20:07:03.621 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:07:03 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:07:02.407866+0000 mon.smithi050 (mon.0) 68 : cluster [DBG] mgrmap e10: smithi050.nbzhun(active, starting, since 0.0525866s) 2023-12-25T20:07:03.621 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:07:03 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:07:02.409028+0000 mon.smithi050 (mon.0) 69 : audit [DBG] from='mgr.14142 172.21.15.50:0/4184813742' entity='mgr.smithi050.nbzhun' cmd=[{"prefix": "mon metadata", "id": "smithi050"}]: dispatch 2023-12-25T20:07:03.621 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:07:03 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:07:02.409590+0000 mon.smithi050 (mon.0) 70 : audit [DBG] from='mgr.14142 172.21.15.50:0/4184813742' entity='mgr.smithi050.nbzhun' cmd=[{"prefix": "mgr metadata", "who": "smithi050.nbzhun", "id": "smithi050.nbzhun"}]: dispatch 2023-12-25T20:07:03.621 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:07:03 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:07:02.410249+0000 mon.smithi050 (mon.0) 71 : audit [DBG] from='mgr.14142 172.21.15.50:0/4184813742' entity='mgr.smithi050.nbzhun' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-12-25T20:07:03.621 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:07:03 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:07:02.410403+0000 mon.smithi050 (mon.0) 72 : audit [DBG] from='mgr.14142 172.21.15.50:0/4184813742' entity='mgr.smithi050.nbzhun' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-12-25T20:07:03.621 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:07:03 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:07:02.410551+0000 mon.smithi050 (mon.0) 73 : audit [DBG] from='mgr.14142 172.21.15.50:0/4184813742' entity='mgr.smithi050.nbzhun' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-12-25T20:07:03.621 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:07:03 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:07:02.423804+0000 mon.smithi050 (mon.0) 74 : cluster [INF] Manager daemon smithi050.nbzhun is now available 2023-12-25T20:07:03.621 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:07:03 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:07:02.529040+0000 mon.smithi050 (mon.0) 75 : audit [DBG] from='mgr.14142 172.21.15.50:0/4184813742' entity='mgr.smithi050.nbzhun' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-25T20:07:03.621 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:07:03 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:07:02.530487+0000 mon.smithi050 (mon.0) 76 : audit [DBG] from='mgr.14142 172.21.15.50:0/4184813742' entity='mgr.smithi050.nbzhun' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-25T20:07:03.622 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:07:03 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:07:02.545857+0000 mon.smithi050 (mon.0) 77 : audit [INF] from='mgr.14142 172.21.15.50:0/4184813742' entity='mgr.smithi050.nbzhun' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi050.nbzhun/mirror_snapshot_schedule"}]: dispatch 2023-12-25T20:07:03.622 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:07:03 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:07:02.548474+0000 mon.smithi050 (mon.0) 78 : audit [INF] from='mgr.14142 172.21.15.50:0/4184813742' entity='mgr.smithi050.nbzhun' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi050.nbzhun/trash_purge_schedule"}]: dispatch 2023-12-25T20:07:03.796 INFO:teuthology.orchestra.run.smithi050.stderr:Deploying grafana service with default placement... 2023-12-25T20:07:03.796 INFO:teuthology.orchestra.run.smithi050.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi050 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/02c8f25c-a361-11ee-95a5-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmps7kf4jlu:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpu2g3lwcx:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply grafana 2023-12-25T20:07:04.310 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:04.303+0000 7f2351a81700 1 Processor -- start 2023-12-25T20:07:04.310 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:04.303+0000 7f2351a81700 1 -- start start 2023-12-25T20:07:04.310 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:04.303+0000 7f2351a81700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f234c0fa9f0 0x7f234c0fae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-25T20:07:04.310 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:04.303+0000 7f2351a81700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f234c0fb300 con 0x7f234c0fa9f0 2023-12-25T20:07:04.310 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:04.303+0000 7f2350a7f700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f234c0fa9f0 0x7f234c0fae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:07:04.311 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:04.303+0000 7f2350a7f700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f234c0fa9f0 0x7f234c0fae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:44522/0 (socket says 172.21.15.50:44522) 2023-12-25T20:07:04.311 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:04.303+0000 7f2350a7f700 1 -- 172.21.15.50:0/2229160513 learned_addr learned my addr 172.21.15.50:0/2229160513 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-25T20:07:04.311 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:04.303+0000 7f2350a7f700 1 -- 172.21.15.50:0/2229160513 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f234c0fb440 con 0x7f234c0fa9f0 2023-12-25T20:07:04.311 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:04.304+0000 7f2350a7f700 1 --2- 172.21.15.50:0/2229160513 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f234c0fa9f0 0x7f234c0fae10 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f234000b5f0 tx=0x7f2340006ee0).ready entity=mon.0 client_cookie=b2314b8bbbeb3e71 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:07:04.311 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:04.304+0000 7f234b7fe700 1 -- 172.21.15.50:0/2229160513 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f234000d600 con 0x7f234c0fa9f0 2023-12-25T20:07:04.311 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:04.304+0000 7f234b7fe700 1 -- 172.21.15.50:0/2229160513 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f234000dc10 con 0x7f234c0fa9f0 2023-12-25T20:07:04.311 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:04.304+0000 7f234b7fe700 1 -- 172.21.15.50:0/2229160513 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2340011cd0 con 0x7f234c0fa9f0 2023-12-25T20:07:04.312 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:04.304+0000 7f2351a81700 1 -- 172.21.15.50:0/2229160513 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f234c0fa9f0 msgr2=0x7f234c0fae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:07:04.312 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:04.304+0000 7f2351a81700 1 --2- 172.21.15.50:0/2229160513 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f234c0fa9f0 0x7f234c0fae10 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f234000b5f0 tx=0x7f2340006ee0).stop 2023-12-25T20:07:04.312 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:04.304+0000 7f2351a81700 1 -- 172.21.15.50:0/2229160513 shutdown_connections 2023-12-25T20:07:04.312 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:04.304+0000 7f2351a81700 1 --2- 172.21.15.50:0/2229160513 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f234c0fa9f0 0x7f234c0fae10 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:07:04.312 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:04.304+0000 7f2351a81700 1 -- 172.21.15.50:0/2229160513 >> 172.21.15.50:0/2229160513 conn(0x7f234c0f6250 msgr2=0x7f234c0f8670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:07:04.312 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:04.305+0000 7f2351a81700 1 -- 172.21.15.50:0/2229160513 shutdown_connections 2023-12-25T20:07:04.312 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:04.305+0000 7f2351a81700 1 -- 172.21.15.50:0/2229160513 wait complete. 2023-12-25T20:07:04.313 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:04.305+0000 7f2351a81700 1 Processor -- start 2023-12-25T20:07:04.313 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:04.305+0000 7f2351a81700 1 -- start start 2023-12-25T20:07:04.313 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:04.305+0000 7f2351a81700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f234c0fa9f0 0x7f234c10d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:07:04.313 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:04.305+0000 7f2351a81700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f234c070520 con 0x7f234c0fa9f0 2023-12-25T20:07:04.313 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:04.306+0000 7f2350a7f700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f234c0fa9f0 0x7f234c10d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:07:04.313 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:04.306+0000 7f2350a7f700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f234c0fa9f0 0x7f234c10d5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:44528/0 (socket says 172.21.15.50:44528) 2023-12-25T20:07:04.313 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:04.306+0000 7f2350a7f700 1 -- 172.21.15.50:0/297641047 learned_addr learned my addr 172.21.15.50:0/297641047 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-25T20:07:04.314 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:04.306+0000 7f2350a7f700 1 -- 172.21.15.50:0/297641047 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f234000b200 con 0x7f234c0fa9f0 2023-12-25T20:07:04.314 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:04.306+0000 7f2350a7f700 1 --2- 172.21.15.50:0/297641047 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f234c0fa9f0 0x7f234c10d5a0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f2340006cb0 tx=0x7f2340006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:07:04.314 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:04.306+0000 7f23497fa700 1 -- 172.21.15.50:0/297641047 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2340011df0 con 0x7f234c0fa9f0 2023-12-25T20:07:04.314 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:04.306+0000 7f23497fa700 1 -- 172.21.15.50:0/297641047 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f234001b8c0 con 0x7f234c0fa9f0 2023-12-25T20:07:04.314 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:04.306+0000 7f2351a81700 1 -- 172.21.15.50:0/297641047 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f234c070720 con 0x7f234c0fa9f0 2023-12-25T20:07:04.314 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:04.306+0000 7f23497fa700 1 -- 172.21.15.50:0/297641047 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f234001a930 con 0x7f234c0fa9f0 2023-12-25T20:07:04.314 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:04.306+0000 7f2351a81700 1 -- 172.21.15.50:0/297641047 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f234c10dc50 con 0x7f234c0fa9f0 2023-12-25T20:07:04.315 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:04.307+0000 7f23497fa700 1 -- 172.21.15.50:0/297641047 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 11) v1 ==== 36054+0+0 (secure 0 0 0) 0x7f234000f070 con 0x7f234c0fa9f0 2023-12-25T20:07:04.315 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:04.307+0000 7f23497fa700 1 --2- 172.21.15.50:0/297641047 >> [v2:172.21.15.50:6800/2637522512,v1:172.21.15.50:6801/2637522512] conn(0x7f233002fd70 0x7f2330032230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:07:04.315 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:04.307+0000 7f23497fa700 1 -- 172.21.15.50:0/297641047 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f2340040de0 con 0x7f234c0fa9f0 2023-12-25T20:07:04.315 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:04.307+0000 7f2351a81700 1 -- 172.21.15.50:0/297641047 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f234c045ad0 con 0x7f234c0fa9f0 2023-12-25T20:07:04.315 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:04.307+0000 7f234bfff700 1 --2- 172.21.15.50:0/297641047 >> [v2:172.21.15.50:6800/2637522512,v1:172.21.15.50:6801/2637522512] conn(0x7f233002fd70 0x7f2330032230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:07:04.315 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:04.308+0000 7f234bfff700 1 --2- 172.21.15.50:0/297641047 >> [v2:172.21.15.50:6800/2637522512,v1:172.21.15.50:6801/2637522512] conn(0x7f233002fd70 0x7f2330032230 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f23380096a0 tx=0x7f2338006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:07:04.316 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:04.312+0000 7f23497fa700 1 -- 172.21.15.50:0/297641047 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2340024410 con 0x7f234c0fa9f0 2023-12-25T20:07:04.514 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:04.513+0000 7f2351a81700 1 -- 172.21.15.50:0/297641047 --> [v2:172.21.15.50:6800/2637522512,v1:172.21.15.50:6801/2637522512] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}) v1 -- 0x7f234c057f10 con 0x7f233002fd70 2023-12-25T20:07:04.517 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:04.517+0000 7f23497fa700 1 -- 172.21.15.50:0/297641047 <== mgr.14142 v2:172.21.15.50:6800/2637522512 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f234c057f10 con 0x7f233002fd70 2023-12-25T20:07:04.517 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: Scheduled grafana update... 2023-12-25T20:07:04.520 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:04.518+0000 7f2351a81700 1 -- 172.21.15.50:0/297641047 >> [v2:172.21.15.50:6800/2637522512,v1:172.21.15.50:6801/2637522512] conn(0x7f233002fd70 msgr2=0x7f2330032230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:07:04.520 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:04.518+0000 7f2351a81700 1 --2- 172.21.15.50:0/297641047 >> [v2:172.21.15.50:6800/2637522512,v1:172.21.15.50:6801/2637522512] conn(0x7f233002fd70 0x7f2330032230 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f23380096a0 tx=0x7f2338006b40).stop 2023-12-25T20:07:04.520 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:04.518+0000 7f2351a81700 1 -- 172.21.15.50:0/297641047 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f234c0fa9f0 msgr2=0x7f234c10d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:07:04.520 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:04.518+0000 7f2351a81700 1 --2- 172.21.15.50:0/297641047 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f234c0fa9f0 0x7f234c10d5a0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f2340006cb0 tx=0x7f2340006ee0).stop 2023-12-25T20:07:04.520 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:04.518+0000 7f2351a81700 1 -- 172.21.15.50:0/297641047 shutdown_connections 2023-12-25T20:07:04.520 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:04.518+0000 7f2351a81700 1 --2- 172.21.15.50:0/297641047 >> [v2:172.21.15.50:6800/2637522512,v1:172.21.15.50:6801/2637522512] conn(0x7f233002fd70 0x7f2330032230 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:07:04.520 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:04.518+0000 7f2351a81700 1 --2- 172.21.15.50:0/297641047 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f234c0fa9f0 0x7f234c10d5a0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:07:04.520 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:04.518+0000 7f2351a81700 1 -- 172.21.15.50:0/297641047 >> 172.21.15.50:0/297641047 conn(0x7f234c0f6250 msgr2=0x7f234c0f7d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:07:04.520 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:04.519+0000 7f2351a81700 1 -- 172.21.15.50:0/297641047 shutdown_connections 2023-12-25T20:07:04.521 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:04.519+0000 7f2351a81700 1 -- 172.21.15.50:0/297641047 wait complete. 2023-12-25T20:07:04.719 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:07:04 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:07:03.409566+0000 mon.smithi050 (mon.0) 79 : cluster [DBG] mgrmap e11: smithi050.nbzhun(active, since 1.05429s) 2023-12-25T20:07:04.719 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:07:04 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cephadm 2023-12-25T20:07:03.410972+0000 mgr.smithi050.nbzhun (mgr.14142) 2 : cephadm [INF] Saving service prometheus spec with placement count:1 2023-12-25T20:07:04.719 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:07:04 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:07:03.412920+0000 mon.smithi050 (mon.0) 80 : audit [INF] from='mgr.14142 172.21.15.50:0/4184813742' entity='mgr.smithi050.nbzhun' 2023-12-25T20:07:04.719 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:07:04 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:07:03.845017+0000 mon.smithi050 (mon.0) 81 : audit [INF] from='mgr.14142 172.21.15.50:0/4184813742' entity='mgr.smithi050.nbzhun' 2023-12-25T20:07:05.047 INFO:teuthology.orchestra.run.smithi050.stderr:Deploying node-exporter service with default placement... 2023-12-25T20:07:05.047 INFO:teuthology.orchestra.run.smithi050.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi050 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/02c8f25c-a361-11ee-95a5-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmps7kf4jlu:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpu2g3lwcx:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply node-exporter 2023-12-25T20:07:05.458 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:05.455+0000 7f8d9bfff700 1 Processor -- start 2023-12-25T20:07:05.458 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:05.455+0000 7f8d9bfff700 1 -- start start 2023-12-25T20:07:05.458 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:05.455+0000 7f8d9bfff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8d9c070430 0x7f8d9c0fe730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-25T20:07:05.458 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:05.455+0000 7f8d9bfff700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d9c0fec20 con 0x7f8d9c070430 2023-12-25T20:07:05.458 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:05.455+0000 7f8d9affd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8d9c070430 0x7f8d9c0fe730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:07:05.458 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:05.455+0000 7f8d9affd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8d9c070430 0x7f8d9c0fe730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:44544/0 (socket says 172.21.15.50:44544) 2023-12-25T20:07:05.458 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:05.455+0000 7f8d9affd700 1 -- 172.21.15.50:0/1263773522 learned_addr learned my addr 172.21.15.50:0/1263773522 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-25T20:07:05.458 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:05.456+0000 7f8d9affd700 1 -- 172.21.15.50:0/1263773522 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d9c0fed60 con 0x7f8d9c070430 2023-12-25T20:07:05.458 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:05.456+0000 7f8d9affd700 1 --2- 172.21.15.50:0/1263773522 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8d9c070430 0x7f8d9c0fe730 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f8d8c00b5f0 tx=0x7f8d8c006ee0).ready entity=mon.0 client_cookie=55f15c7dda1a952d server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:07:05.458 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:05.456+0000 7f8d99ffb700 1 -- 172.21.15.50:0/1263773522 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8d8c00d600 con 0x7f8d9c070430 2023-12-25T20:07:05.458 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:05.456+0000 7f8d99ffb700 1 -- 172.21.15.50:0/1263773522 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8d8c00dc10 con 0x7f8d9c070430 2023-12-25T20:07:05.458 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:05.456+0000 7f8d99ffb700 1 -- 172.21.15.50:0/1263773522 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8d8c011cd0 con 0x7f8d9c070430 2023-12-25T20:07:05.459 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:05.456+0000 7f8d9bfff700 1 -- 172.21.15.50:0/1263773522 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8d9c070430 msgr2=0x7f8d9c0fe730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:07:05.459 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:05.456+0000 7f8d9bfff700 1 --2- 172.21.15.50:0/1263773522 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8d9c070430 0x7f8d9c0fe730 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f8d8c00b5f0 tx=0x7f8d8c006ee0).stop 2023-12-25T20:07:05.459 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:05.456+0000 7f8d9bfff700 1 -- 172.21.15.50:0/1263773522 shutdown_connections 2023-12-25T20:07:05.459 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:05.457+0000 7f8d9bfff700 1 --2- 172.21.15.50:0/1263773522 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8d9c070430 0x7f8d9c0fe730 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:07:05.459 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:05.457+0000 7f8d9bfff700 1 -- 172.21.15.50:0/1263773522 >> 172.21.15.50:0/1263773522 conn(0x7f8d9c0f6210 msgr2=0x7f8d9c0f8630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:07:05.459 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:05.457+0000 7f8d9bfff700 1 -- 172.21.15.50:0/1263773522 shutdown_connections 2023-12-25T20:07:05.459 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:05.457+0000 7f8d9bfff700 1 -- 172.21.15.50:0/1263773522 wait complete. 2023-12-25T20:07:05.459 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:05.457+0000 7f8d9bfff700 1 Processor -- start 2023-12-25T20:07:05.460 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:05.457+0000 7f8d9bfff700 1 -- start start 2023-12-25T20:07:05.460 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:05.457+0000 7f8d9bfff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8d9c070430 0x7f8d9c0fc1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:07:05.460 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:05.457+0000 7f8d9bfff700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d9c0fa630 con 0x7f8d9c070430 2023-12-25T20:07:05.460 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:05.458+0000 7f8d9affd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8d9c070430 0x7f8d9c0fc1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:07:05.460 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:05.458+0000 7f8d9affd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8d9c070430 0x7f8d9c0fc1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:44552/0 (socket says 172.21.15.50:44552) 2023-12-25T20:07:05.461 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:05.458+0000 7f8d9affd700 1 -- 172.21.15.50:0/3637595507 learned_addr learned my addr 172.21.15.50:0/3637595507 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-25T20:07:05.461 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:05.458+0000 7f8d9affd700 1 -- 172.21.15.50:0/3637595507 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d8c00b200 con 0x7f8d9c070430 2023-12-25T20:07:05.461 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:05.458+0000 7f8d9affd700 1 --2- 172.21.15.50:0/3637595507 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8d9c070430 0x7f8d9c0fc1b0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f8d8c00df90 tx=0x7f8d8c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:07:05.461 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:05.458+0000 7f8d837fe700 1 -- 172.21.15.50:0/3637595507 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8d8c011da0 con 0x7f8d9c070430 2023-12-25T20:07:05.461 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:05.458+0000 7f8d837fe700 1 -- 172.21.15.50:0/3637595507 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8d8c01b8c0 con 0x7f8d9c070430 2023-12-25T20:07:05.461 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:05.458+0000 7f8d9bfff700 1 -- 172.21.15.50:0/3637595507 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d9c0fa7d0 con 0x7f8d9c070430 2023-12-25T20:07:05.462 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:05.458+0000 7f8d837fe700 1 -- 172.21.15.50:0/3637595507 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8d8c01a930 con 0x7f8d9c070430 2023-12-25T20:07:05.462 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:05.458+0000 7f8d9bfff700 1 -- 172.21.15.50:0/3637595507 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d9c0fabe0 con 0x7f8d9c070430 2023-12-25T20:07:05.462 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:05.459+0000 7f8d837fe700 1 -- 172.21.15.50:0/3637595507 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f8d8c00f070 con 0x7f8d9c070430 2023-12-25T20:07:05.462 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:05.459+0000 7f8d837fe700 1 --2- 172.21.15.50:0/3637595507 >> [v2:172.21.15.50:6800/2637522512,v1:172.21.15.50:6801/2637522512] conn(0x7f8d8802fe60 0x7f8d88032320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:07:05.462 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:05.459+0000 7f8d837fe700 1 -- 172.21.15.50:0/3637595507 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f8d8c041090 con 0x7f8d9c070430 2023-12-25T20:07:05.462 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:05.459+0000 7f8d9a7fc700 1 --2- 172.21.15.50:0/3637595507 >> [v2:172.21.15.50:6800/2637522512,v1:172.21.15.50:6801/2637522512] conn(0x7f8d8802fe60 0x7f8d88032320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:07:05.462 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:05.459+0000 7f8d9bfff700 1 -- 172.21.15.50:0/3637595507 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d9c045ad0 con 0x7f8d9c070430 2023-12-25T20:07:05.462 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:05.460+0000 7f8d9a7fc700 1 --2- 172.21.15.50:0/3637595507 >> [v2:172.21.15.50:6800/2637522512,v1:172.21.15.50:6801/2637522512] conn(0x7f8d8802fe60 0x7f8d88032320 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f8d900096a0 tx=0x7f8d90006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:07:05.464 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:05.464+0000 7f8d837fe700 1 -- 172.21.15.50:0/3637595507 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8d8c016070 con 0x7f8d9c070430 2023-12-25T20:07:05.656 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:05.654+0000 7f8d9bfff700 1 -- 172.21.15.50:0/3637595507 --> [v2:172.21.15.50:6800/2637522512,v1:172.21.15.50:6801/2637522512] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7f8d9c104750 con 0x7f8d8802fe60 2023-12-25T20:07:05.659 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:05.658+0000 7f8d837fe700 1 -- 172.21.15.50:0/3637595507 <== mgr.14142 v2:172.21.15.50:6800/2637522512 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7f8d9c104750 con 0x7f8d8802fe60 2023-12-25T20:07:05.659 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: Scheduled node-exporter update... 2023-12-25T20:07:05.662 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:05.660+0000 7f8d9bfff700 1 -- 172.21.15.50:0/3637595507 >> [v2:172.21.15.50:6800/2637522512,v1:172.21.15.50:6801/2637522512] conn(0x7f8d8802fe60 msgr2=0x7f8d88032320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:07:05.662 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:05.660+0000 7f8d9bfff700 1 --2- 172.21.15.50:0/3637595507 >> [v2:172.21.15.50:6800/2637522512,v1:172.21.15.50:6801/2637522512] conn(0x7f8d8802fe60 0x7f8d88032320 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f8d900096a0 tx=0x7f8d90006b40).stop 2023-12-25T20:07:05.662 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:05.660+0000 7f8d9bfff700 1 -- 172.21.15.50:0/3637595507 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8d9c070430 msgr2=0x7f8d9c0fc1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:07:05.662 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:05.660+0000 7f8d9bfff700 1 --2- 172.21.15.50:0/3637595507 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8d9c070430 0x7f8d9c0fc1b0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f8d8c00df90 tx=0x7f8d8c006ee0).stop 2023-12-25T20:07:05.662 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:05.660+0000 7f8d9bfff700 1 -- 172.21.15.50:0/3637595507 shutdown_connections 2023-12-25T20:07:05.662 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:05.660+0000 7f8d9bfff700 1 --2- 172.21.15.50:0/3637595507 >> [v2:172.21.15.50:6800/2637522512,v1:172.21.15.50:6801/2637522512] conn(0x7f8d8802fe60 0x7f8d88032320 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:07:05.662 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:05.660+0000 7f8d9bfff700 1 --2- 172.21.15.50:0/3637595507 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8d9c070430 0x7f8d9c0fc1b0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:07:05.662 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:05.660+0000 7f8d9bfff700 1 -- 172.21.15.50:0/3637595507 >> 172.21.15.50:0/3637595507 conn(0x7f8d9c0f6210 msgr2=0x7f8d9c0f7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:07:05.662 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:05.660+0000 7f8d9bfff700 1 -- 172.21.15.50:0/3637595507 shutdown_connections 2023-12-25T20:07:05.662 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:05.660+0000 7f8d9bfff700 1 -- 172.21.15.50:0/3637595507 wait complete. 2023-12-25T20:07:05.837 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:07:05 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:07:04.515159+0000 mgr.smithi050.nbzhun (mgr.14142) 3 : audit [DBG] from='client.14150 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2023-12-25T20:07:05.837 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:07:05 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cephadm 2023-12-25T20:07:04.515688+0000 mgr.smithi050.nbzhun (mgr.14142) 4 : cephadm [INF] Saving service grafana spec with placement count:1 2023-12-25T20:07:05.837 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:07:05 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:07:04.517911+0000 mon.smithi050 (mon.0) 82 : audit [INF] from='mgr.14142 172.21.15.50:0/4184813742' entity='mgr.smithi050.nbzhun' 2023-12-25T20:07:05.837 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:07:05 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:07:04.846078+0000 mon.smithi050 (mon.0) 83 : cluster [DBG] mgrmap e12: smithi050.nbzhun(active, since 2s) 2023-12-25T20:07:05.996 INFO:teuthology.orchestra.run.smithi050.stderr:Deploying alertmanager service with default placement... 2023-12-25T20:07:05.997 INFO:teuthology.orchestra.run.smithi050.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi050 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/02c8f25c-a361-11ee-95a5-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmps7kf4jlu:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpu2g3lwcx:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply alertmanager 2023-12-25T20:07:06.382 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:06.381+0000 7fbaa57d8700 1 Processor -- start 2023-12-25T20:07:06.382 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:06.381+0000 7fbaa57d8700 1 -- start start 2023-12-25T20:07:06.383 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:06.381+0000 7fbaa57d8700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbaa00704a0 0x7fbaa00fe8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-25T20:07:06.383 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:06.381+0000 7fbaa57d8700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaa00fede0 con 0x7fbaa00704a0 2023-12-25T20:07:06.383 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:06.381+0000 7fba9effd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbaa00704a0 0x7fbaa00fe8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:07:06.383 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:06.381+0000 7fba9effd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbaa00704a0 0x7fbaa00fe8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:44554/0 (socket says 172.21.15.50:44554) 2023-12-25T20:07:06.383 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:06.381+0000 7fba9effd700 1 -- 172.21.15.50:0/2260487400 learned_addr learned my addr 172.21.15.50:0/2260487400 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-25T20:07:06.383 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:06.381+0000 7fba9effd700 1 -- 172.21.15.50:0/2260487400 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaa00fef20 con 0x7fbaa00704a0 2023-12-25T20:07:06.385 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:06.381+0000 7fba9effd700 1 --2- 172.21.15.50:0/2260487400 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbaa00704a0 0x7fbaa00fe8f0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7fba9000b5f0 tx=0x7fba90006ee0).ready entity=mon.0 client_cookie=87c18aa2c7e59144 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:07:06.385 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:06.382+0000 7fba9dffb700 1 -- 172.21.15.50:0/2260487400 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fba9000d600 con 0x7fbaa00704a0 2023-12-25T20:07:06.385 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:06.382+0000 7fba9dffb700 1 -- 172.21.15.50:0/2260487400 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fba9000dc10 con 0x7fbaa00704a0 2023-12-25T20:07:06.385 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:06.382+0000 7fba9dffb700 1 -- 172.21.15.50:0/2260487400 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fba90011cd0 con 0x7fbaa00704a0 2023-12-25T20:07:06.385 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:06.382+0000 7fbaa57d8700 1 -- 172.21.15.50:0/2260487400 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbaa00704a0 msgr2=0x7fbaa00fe8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:07:06.385 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:06.382+0000 7fbaa57d8700 1 --2- 172.21.15.50:0/2260487400 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbaa00704a0 0x7fbaa00fe8f0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7fba9000b5f0 tx=0x7fba90006ee0).stop 2023-12-25T20:07:06.385 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:06.382+0000 7fbaa57d8700 1 -- 172.21.15.50:0/2260487400 shutdown_connections 2023-12-25T20:07:06.385 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:06.382+0000 7fbaa57d8700 1 --2- 172.21.15.50:0/2260487400 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbaa00704a0 0x7fbaa00fe8f0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:07:06.385 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:06.382+0000 7fbaa57d8700 1 -- 172.21.15.50:0/2260487400 >> 172.21.15.50:0/2260487400 conn(0x7fbaa00f63b0 msgr2=0x7fbaa00f87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:07:06.385 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:06.382+0000 7fbaa57d8700 1 -- 172.21.15.50:0/2260487400 shutdown_connections 2023-12-25T20:07:06.386 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:06.382+0000 7fbaa57d8700 1 -- 172.21.15.50:0/2260487400 wait complete. 2023-12-25T20:07:06.386 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:06.383+0000 7fbaa57d8700 1 Processor -- start 2023-12-25T20:07:06.386 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:06.383+0000 7fbaa57d8700 1 -- start start 2023-12-25T20:07:06.386 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:06.383+0000 7fbaa57d8700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbaa00704a0 0x7fbaa010aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:07:06.386 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:06.383+0000 7fbaa57d8700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaa010af30 con 0x7fbaa00704a0 2023-12-25T20:07:06.386 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:06.383+0000 7fba9effd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbaa00704a0 0x7fbaa010aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:07:06.386 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:06.383+0000 7fba9effd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbaa00704a0 0x7fbaa010aa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:44568/0 (socket says 172.21.15.50:44568) 2023-12-25T20:07:06.387 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:06.383+0000 7fba9effd700 1 -- 172.21.15.50:0/1992858362 learned_addr learned my addr 172.21.15.50:0/1992858362 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-25T20:07:06.387 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:06.384+0000 7fba9effd700 1 -- 172.21.15.50:0/1992858362 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba9000b200 con 0x7fbaa00704a0 2023-12-25T20:07:06.387 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:06.384+0000 7fba9effd700 1 --2- 172.21.15.50:0/1992858362 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbaa00704a0 0x7fbaa010aa40 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7fba90006a40 tx=0x7fba90006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:07:06.387 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:06.384+0000 7fba877fe700 1 -- 172.21.15.50:0/1992858362 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fba90011c90 con 0x7fbaa00704a0 2023-12-25T20:07:06.387 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:06.384+0000 7fbaa57d8700 1 -- 172.21.15.50:0/1992858362 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbaa010b130 con 0x7fbaa00704a0 2023-12-25T20:07:06.387 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:06.384+0000 7fba877fe700 1 -- 172.21.15.50:0/1992858362 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fba9001b8c0 con 0x7fbaa00704a0 2023-12-25T20:07:06.387 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:06.384+0000 7fba877fe700 1 -- 172.21.15.50:0/1992858362 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fba9001a8c0 con 0x7fbaa00704a0 2023-12-25T20:07:06.387 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:06.384+0000 7fbaa57d8700 1 -- 172.21.15.50:0/1992858362 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbaa010b540 con 0x7fbaa00704a0 2023-12-25T20:07:06.388 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:06.385+0000 7fba877fe700 1 -- 172.21.15.50:0/1992858362 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7fba9000f070 con 0x7fbaa00704a0 2023-12-25T20:07:06.388 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:06.385+0000 7fba877fe700 1 --2- 172.21.15.50:0/1992858362 >> [v2:172.21.15.50:6800/2637522512,v1:172.21.15.50:6801/2637522512] conn(0x7fba8c02fe60 0x7fba8c032320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:07:06.388 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:06.385+0000 7fba877fe700 1 -- 172.21.15.50:0/1992858362 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7fba90040a70 con 0x7fbaa00704a0 2023-12-25T20:07:06.388 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:06.385+0000 7fba9e7fc700 1 --2- 172.21.15.50:0/1992858362 >> [v2:172.21.15.50:6800/2637522512,v1:172.21.15.50:6801/2637522512] conn(0x7fba8c02fe60 0x7fba8c032320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:07:06.388 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:06.385+0000 7fba9e7fc700 1 --2- 172.21.15.50:0/1992858362 >> [v2:172.21.15.50:6800/2637522512,v1:172.21.15.50:6801/2637522512] conn(0x7fba8c02fe60 0x7fba8c032320 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fba940096a0 tx=0x7fba94006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:07:06.388 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:06.385+0000 7fbaa57d8700 1 -- 172.21.15.50:0/1992858362 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbaa0059100 con 0x7fbaa00704a0 2023-12-25T20:07:06.390 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:06.390+0000 7fba877fe700 1 -- 172.21.15.50:0/1992858362 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fba90016070 con 0x7fbaa00704a0 2023-12-25T20:07:06.578 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:06.577+0000 7fbaa57d8700 1 -- 172.21.15.50:0/1992858362 --> [v2:172.21.15.50:6800/2637522512,v1:172.21.15.50:6801/2637522512] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}) v1 -- 0x7fbaa0104950 con 0x7fba8c02fe60 2023-12-25T20:07:06.581 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:06.580+0000 7fba877fe700 1 -- 172.21.15.50:0/1992858362 <== mgr.14142 v2:172.21.15.50:6800/2637522512 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7fbaa0104950 con 0x7fba8c02fe60 2023-12-25T20:07:06.581 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: Scheduled alertmanager update... 2023-12-25T20:07:06.584 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:06.582+0000 7fbaa57d8700 1 -- 172.21.15.50:0/1992858362 >> [v2:172.21.15.50:6800/2637522512,v1:172.21.15.50:6801/2637522512] conn(0x7fba8c02fe60 msgr2=0x7fba8c032320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:07:06.584 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:06.582+0000 7fbaa57d8700 1 --2- 172.21.15.50:0/1992858362 >> [v2:172.21.15.50:6800/2637522512,v1:172.21.15.50:6801/2637522512] conn(0x7fba8c02fe60 0x7fba8c032320 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fba940096a0 tx=0x7fba94006b40).stop 2023-12-25T20:07:06.584 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:06.582+0000 7fbaa57d8700 1 -- 172.21.15.50:0/1992858362 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbaa00704a0 msgr2=0x7fbaa010aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:07:06.584 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:06.582+0000 7fbaa57d8700 1 --2- 172.21.15.50:0/1992858362 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbaa00704a0 0x7fbaa010aa40 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7fba90006a40 tx=0x7fba90006ee0).stop 2023-12-25T20:07:06.584 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:06.582+0000 7fbaa57d8700 1 -- 172.21.15.50:0/1992858362 shutdown_connections 2023-12-25T20:07:06.584 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:06.582+0000 7fbaa57d8700 1 --2- 172.21.15.50:0/1992858362 >> [v2:172.21.15.50:6800/2637522512,v1:172.21.15.50:6801/2637522512] conn(0x7fba8c02fe60 0x7fba8c032320 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:07:06.584 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:06.582+0000 7fbaa57d8700 1 --2- 172.21.15.50:0/1992858362 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbaa00704a0 0x7fbaa010aa40 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:07:06.584 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:06.582+0000 7fbaa57d8700 1 -- 172.21.15.50:0/1992858362 >> 172.21.15.50:0/1992858362 conn(0x7fbaa00f63b0 msgr2=0x7fbaa00f7e20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:07:06.584 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:06.582+0000 7fbaa57d8700 1 -- 172.21.15.50:0/1992858362 shutdown_connections 2023-12-25T20:07:06.584 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:06.582+0000 7fbaa57d8700 1 -- 172.21.15.50:0/1992858362 wait complete. 2023-12-25T20:07:06.938 INFO:teuthology.orchestra.run.smithi050.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi050 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/02c8f25c-a361-11ee-95a5-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmps7kf4jlu:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpu2g3lwcx:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mgr mgr/cephadm/container_init True --force 2023-12-25T20:07:06.940 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:07:06 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:07:05.656888+0000 mgr.smithi050.nbzhun (mgr.14142) 5 : audit [DBG] from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2023-12-25T20:07:06.940 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:07:06 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cephadm 2023-12-25T20:07:05.657528+0000 mgr.smithi050.nbzhun (mgr.14142) 6 : cephadm [INF] Saving service node-exporter spec with placement * 2023-12-25T20:07:06.940 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:07:06 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:07:05.659732+0000 mon.smithi050 (mon.0) 84 : audit [INF] from='mgr.14142 172.21.15.50:0/4184813742' entity='mgr.smithi050.nbzhun' 2023-12-25T20:07:06.940 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:07:06 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:07:06.581737+0000 mon.smithi050 (mon.0) 85 : audit [INF] from='mgr.14142 172.21.15.50:0/4184813742' entity='mgr.smithi050.nbzhun' 2023-12-25T20:07:07.313 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:07.311+0000 7f9cac971700 1 Processor -- start 2023-12-25T20:07:07.313 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:07.311+0000 7f9cac971700 1 -- start start 2023-12-25T20:07:07.313 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:07.311+0000 7f9cac971700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9ca80fab80 0x7f9ca80fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-25T20:07:07.313 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:07.312+0000 7f9cac971700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ca80fb490 con 0x7f9ca80fab80 2023-12-25T20:07:07.313 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:07.312+0000 7f9ca659c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9ca80fab80 0x7f9ca80fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:07:07.313 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:07.312+0000 7f9ca659c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9ca80fab80 0x7f9ca80fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:40956/0 (socket says 172.21.15.50:40956) 2023-12-25T20:07:07.314 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:07.312+0000 7f9ca659c700 1 -- 172.21.15.50:0/544064833 learned_addr learned my addr 172.21.15.50:0/544064833 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-25T20:07:07.314 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:07.312+0000 7f9ca659c700 1 -- 172.21.15.50:0/544064833 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ca80fb5d0 con 0x7f9ca80fab80 2023-12-25T20:07:07.316 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:07.312+0000 7f9ca659c700 1 --2- 172.21.15.50:0/544064833 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9ca80fab80 0x7f9ca80fafa0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f9c9000b5f0 tx=0x7f9c90006ee0).ready entity=mon.0 client_cookie=5f5c6064d608f49c server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:07:07.316 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:07.312+0000 7f9ca559a700 1 -- 172.21.15.50:0/544064833 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9c9000d600 con 0x7f9ca80fab80 2023-12-25T20:07:07.316 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:07.312+0000 7f9ca559a700 1 -- 172.21.15.50:0/544064833 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9c9000dc10 con 0x7f9ca80fab80 2023-12-25T20:07:07.316 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:07.312+0000 7f9ca559a700 1 -- 172.21.15.50:0/544064833 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9c90011cd0 con 0x7f9ca80fab80 2023-12-25T20:07:07.316 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:07.313+0000 7f9cac971700 1 -- 172.21.15.50:0/544064833 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9ca80fab80 msgr2=0x7f9ca80fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:07:07.316 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:07.313+0000 7f9cac971700 1 --2- 172.21.15.50:0/544064833 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9ca80fab80 0x7f9ca80fafa0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f9c9000b5f0 tx=0x7f9c90006ee0).stop 2023-12-25T20:07:07.316 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:07.313+0000 7f9cac971700 1 -- 172.21.15.50:0/544064833 shutdown_connections 2023-12-25T20:07:07.316 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:07.313+0000 7f9cac971700 1 --2- 172.21.15.50:0/544064833 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9ca80fab80 0x7f9ca80fafa0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:07:07.316 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:07.313+0000 7f9cac971700 1 -- 172.21.15.50:0/544064833 >> 172.21.15.50:0/544064833 conn(0x7f9ca80f63c0 msgr2=0x7f9ca80f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:07:07.316 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:07.313+0000 7f9cac971700 1 -- 172.21.15.50:0/544064833 shutdown_connections 2023-12-25T20:07:07.316 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:07.313+0000 7f9cac971700 1 -- 172.21.15.50:0/544064833 wait complete. 2023-12-25T20:07:07.317 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:07.314+0000 7f9cac971700 1 Processor -- start 2023-12-25T20:07:07.317 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:07.314+0000 7f9cac971700 1 -- start start 2023-12-25T20:07:07.317 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:07.314+0000 7f9cac971700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9ca80fab80 0x7f9ca810aac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:07:07.317 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:07.314+0000 7f9cac971700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ca810afb0 con 0x7f9ca80fab80 2023-12-25T20:07:07.317 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:07.314+0000 7f9ca659c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9ca80fab80 0x7f9ca810aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:07:07.317 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:07.314+0000 7f9ca659c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9ca80fab80 0x7f9ca810aac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:40958/0 (socket says 172.21.15.50:40958) 2023-12-25T20:07:07.317 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:07.314+0000 7f9ca659c700 1 -- 172.21.15.50:0/4032756812 learned_addr learned my addr 172.21.15.50:0/4032756812 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-25T20:07:07.317 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:07.315+0000 7f9ca659c700 1 -- 172.21.15.50:0/4032756812 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c9000b200 con 0x7f9ca80fab80 2023-12-25T20:07:07.317 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:07.315+0000 7f9ca659c700 1 --2- 172.21.15.50:0/4032756812 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9ca80fab80 0x7f9ca810aac0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f9c9000dfb0 tx=0x7f9c90006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:07:07.318 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:07.315+0000 7f9c9effd700 1 -- 172.21.15.50:0/4032756812 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9c90011df0 con 0x7f9ca80fab80 2023-12-25T20:07:07.318 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:07.315+0000 7f9c9effd700 1 -- 172.21.15.50:0/4032756812 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9c9001b8c0 con 0x7f9ca80fab80 2023-12-25T20:07:07.318 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:07.315+0000 7f9c9effd700 1 -- 172.21.15.50:0/4032756812 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9c9001a8c0 con 0x7f9ca80fab80 2023-12-25T20:07:07.318 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:07.315+0000 7f9cac971700 1 -- 172.21.15.50:0/4032756812 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ca810b1b0 con 0x7f9ca80fab80 2023-12-25T20:07:07.318 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:07.315+0000 7f9cac971700 1 -- 172.21.15.50:0/4032756812 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ca810b5c0 con 0x7f9ca80fab80 2023-12-25T20:07:07.319 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:07.316+0000 7f9c9effd700 1 -- 172.21.15.50:0/4032756812 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f9c9000f070 con 0x7f9ca80fab80 2023-12-25T20:07:07.319 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:07.316+0000 7f9c9effd700 1 --2- 172.21.15.50:0/4032756812 >> [v2:172.21.15.50:6800/2637522512,v1:172.21.15.50:6801/2637522512] conn(0x7f9c8802fe60 0x7f9c88032320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:07:07.319 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:07.316+0000 7f9c9effd700 1 -- 172.21.15.50:0/4032756812 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f9c90041140 con 0x7f9ca80fab80 2023-12-25T20:07:07.319 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:07.316+0000 7f9ca5d9b700 1 --2- 172.21.15.50:0/4032756812 >> [v2:172.21.15.50:6800/2637522512,v1:172.21.15.50:6801/2637522512] conn(0x7f9c8802fe60 0x7f9c88032320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:07:07.319 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:07.316+0000 7f9cac971700 1 -- 172.21.15.50:0/4032756812 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ca8059100 con 0x7f9ca80fab80 2023-12-25T20:07:07.319 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:07.316+0000 7f9ca5d9b700 1 --2- 172.21.15.50:0/4032756812 >> [v2:172.21.15.50:6800/2637522512,v1:172.21.15.50:6801/2637522512] conn(0x7f9c8802fe60 0x7f9c88032320 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f9c980096a0 tx=0x7f9c98006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:07:07.321 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:07.321+0000 7f9c9effd700 1 -- 172.21.15.50:0/4032756812 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9c90016070 con 0x7f9ca80fab80 2023-12-25T20:07:07.480 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:07.479+0000 7f9cac971700 1 -- 172.21.15.50:0/4032756812 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f9ca81048e0 con 0x7f9ca80fab80 2023-12-25T20:07:07.482 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:07.481+0000 7f9c9effd700 1 -- 172.21.15.50:0/4032756812 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v9)=0 v9) v1 ==== 142+0+0 (secure 0 0 0) 0x7f9c90019c90 con 0x7f9ca80fab80 2023-12-25T20:07:07.485 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:07.483+0000 7f9cac971700 1 -- 172.21.15.50:0/4032756812 >> [v2:172.21.15.50:6800/2637522512,v1:172.21.15.50:6801/2637522512] conn(0x7f9c8802fe60 msgr2=0x7f9c88032320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:07:07.485 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:07.483+0000 7f9cac971700 1 --2- 172.21.15.50:0/4032756812 >> [v2:172.21.15.50:6800/2637522512,v1:172.21.15.50:6801/2637522512] conn(0x7f9c8802fe60 0x7f9c88032320 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f9c980096a0 tx=0x7f9c98006b40).stop 2023-12-25T20:07:07.485 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:07.483+0000 7f9cac971700 1 -- 172.21.15.50:0/4032756812 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9ca80fab80 msgr2=0x7f9ca810aac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:07:07.485 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:07.483+0000 7f9cac971700 1 --2- 172.21.15.50:0/4032756812 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9ca80fab80 0x7f9ca810aac0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f9c9000dfb0 tx=0x7f9c90006ee0).stop 2023-12-25T20:07:07.485 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:07.483+0000 7f9cac971700 1 -- 172.21.15.50:0/4032756812 shutdown_connections 2023-12-25T20:07:07.485 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:07.483+0000 7f9cac971700 1 --2- 172.21.15.50:0/4032756812 >> [v2:172.21.15.50:6800/2637522512,v1:172.21.15.50:6801/2637522512] conn(0x7f9c8802fe60 0x7f9c88032320 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:07:07.485 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:07.483+0000 7f9cac971700 1 --2- 172.21.15.50:0/4032756812 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9ca80fab80 0x7f9ca810aac0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:07:07.485 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:07.483+0000 7f9cac971700 1 -- 172.21.15.50:0/4032756812 >> 172.21.15.50:0/4032756812 conn(0x7f9ca80f63c0 msgr2=0x7f9ca80f6fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:07:07.485 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:07.484+0000 7f9cac971700 1 -- 172.21.15.50:0/4032756812 shutdown_connections 2023-12-25T20:07:07.486 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:07.484+0000 7f9cac971700 1 -- 172.21.15.50:0/4032756812 wait complete. 2023-12-25T20:07:07.832 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:07:07 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:07:06.579045+0000 mgr.smithi050.nbzhun (mgr.14142) 7 : audit [DBG] from='client.14154 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2023-12-25T20:07:07.832 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:07:07 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cephadm 2023-12-25T20:07:06.579595+0000 mgr.smithi050.nbzhun (mgr.14142) 8 : cephadm [INF] Saving service alertmanager spec with placement count:1 2023-12-25T20:07:07.832 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:07:07 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:07:07.483044+0000 mon.smithi050 (mon.0) 86 : audit [INF] from='client.? 172.21.15.50:0/4032756812' entity='client.admin' 2023-12-25T20:07:07.847 INFO:teuthology.orchestra.run.smithi050.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi050 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/02c8f25c-a361-11ee-95a5-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmps7kf4jlu:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpu2g3lwcx:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mgr mgr/dashboard/ssl_server_port 8443 2023-12-25T20:07:08.214 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:08.212+0000 7f283cdf6700 1 Processor -- start 2023-12-25T20:07:08.215 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:08.212+0000 7f283cdf6700 1 -- start start 2023-12-25T20:07:08.215 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:08.213+0000 7f283cdf6700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2838070420 0x7f2838100d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-25T20:07:08.215 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:08.213+0000 7f283cdf6700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28380fc760 con 0x7f2838070420 2023-12-25T20:07:08.215 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:08.213+0000 7f283659c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2838070420 0x7f2838100d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:07:08.215 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:08.213+0000 7f283659c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2838070420 0x7f2838100d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:40960/0 (socket says 172.21.15.50:40960) 2023-12-25T20:07:08.215 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:08.213+0000 7f283659c700 1 -- 172.21.15.50:0/630411015 learned_addr learned my addr 172.21.15.50:0/630411015 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-25T20:07:08.215 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:08.213+0000 7f283659c700 1 -- 172.21.15.50:0/630411015 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2838101260 con 0x7f2838070420 2023-12-25T20:07:08.215 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:08.213+0000 7f283659c700 1 --2- 172.21.15.50:0/630411015 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2838070420 0x7f2838100d70 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f282800b5f0 tx=0x7f2828006ee0).ready entity=mon.0 client_cookie=872deac11268f4fe server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:07:08.215 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:08.214+0000 7f283559a700 1 -- 172.21.15.50:0/630411015 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f282800d600 con 0x7f2838070420 2023-12-25T20:07:08.215 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:08.214+0000 7f283559a700 1 -- 172.21.15.50:0/630411015 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f282800dc10 con 0x7f2838070420 2023-12-25T20:07:08.217 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:08.214+0000 7f283559a700 1 -- 172.21.15.50:0/630411015 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2828011cd0 con 0x7f2838070420 2023-12-25T20:07:08.218 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:08.214+0000 7f283cdf6700 1 -- 172.21.15.50:0/630411015 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2838070420 msgr2=0x7f2838100d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:07:08.218 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:08.214+0000 7f283cdf6700 1 --2- 172.21.15.50:0/630411015 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2838070420 0x7f2838100d70 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f282800b5f0 tx=0x7f2828006ee0).stop 2023-12-25T20:07:08.218 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:08.214+0000 7f283cdf6700 1 -- 172.21.15.50:0/630411015 shutdown_connections 2023-12-25T20:07:08.218 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:08.214+0000 7f283cdf6700 1 --2- 172.21.15.50:0/630411015 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2838070420 0x7f2838100d70 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:07:08.218 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:08.214+0000 7f283cdf6700 1 -- 172.21.15.50:0/630411015 >> 172.21.15.50:0/630411015 conn(0x7f28380f63b0 msgr2=0x7f28380f87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:07:08.218 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:08.214+0000 7f283cdf6700 1 -- 172.21.15.50:0/630411015 shutdown_connections 2023-12-25T20:07:08.218 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:08.214+0000 7f283cdf6700 1 -- 172.21.15.50:0/630411015 wait complete. 2023-12-25T20:07:08.218 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:08.215+0000 7f283cdf6700 1 Processor -- start 2023-12-25T20:07:08.218 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:08.215+0000 7f283cdf6700 1 -- start start 2023-12-25T20:07:08.218 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:08.215+0000 7f283cdf6700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2838070420 0x7f283810ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:07:08.218 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:08.215+0000 7f283cdf6700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f283810d1b0 con 0x7f2838070420 2023-12-25T20:07:08.218 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:08.215+0000 7f283659c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2838070420 0x7f283810ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:07:08.219 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:08.215+0000 7f283659c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2838070420 0x7f283810ccc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:40964/0 (socket says 172.21.15.50:40964) 2023-12-25T20:07:08.219 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:08.215+0000 7f283659c700 1 -- 172.21.15.50:0/3421289954 learned_addr learned my addr 172.21.15.50:0/3421289954 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-25T20:07:08.219 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:08.215+0000 7f283659c700 1 -- 172.21.15.50:0/3421289954 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f282800b200 con 0x7f2838070420 2023-12-25T20:07:08.219 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:08.215+0000 7f283659c700 1 --2- 172.21.15.50:0/3421289954 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2838070420 0x7f283810ccc0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f282800b1d0 tx=0x7f2828006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:07:08.219 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:08.216+0000 7f282effd700 1 -- 172.21.15.50:0/3421289954 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2828011df0 con 0x7f2838070420 2023-12-25T20:07:08.219 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:08.216+0000 7f282effd700 1 -- 172.21.15.50:0/3421289954 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f282801b8c0 con 0x7f2838070420 2023-12-25T20:07:08.219 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:08.216+0000 7f283cdf6700 1 -- 172.21.15.50:0/3421289954 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f283810d3b0 con 0x7f2838070420 2023-12-25T20:07:08.219 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:08.216+0000 7f282effd700 1 -- 172.21.15.50:0/3421289954 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f282801a830 con 0x7f2838070420 2023-12-25T20:07:08.219 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:08.216+0000 7f283cdf6700 1 -- 172.21.15.50:0/3421289954 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f283810d7c0 con 0x7f2838070420 2023-12-25T20:07:08.219 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:08.216+0000 7f282effd700 1 -- 172.21.15.50:0/3421289954 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f282800f070 con 0x7f2838070420 2023-12-25T20:07:08.219 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:08.217+0000 7f283cdf6700 1 -- 172.21.15.50:0/3421289954 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2838045ad0 con 0x7f2838070420 2023-12-25T20:07:08.219 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:08.217+0000 7f282effd700 1 --2- 172.21.15.50:0/3421289954 >> [v2:172.21.15.50:6800/2637522512,v1:172.21.15.50:6801/2637522512] conn(0x7f281802fe60 0x7f2818032320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:07:08.219 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:08.217+0000 7f282effd700 1 -- 172.21.15.50:0/3421289954 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f2828040fc0 con 0x7f2838070420 2023-12-25T20:07:08.219 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:08.217+0000 7f2835d9b700 1 --2- 172.21.15.50:0/3421289954 >> [v2:172.21.15.50:6800/2637522512,v1:172.21.15.50:6801/2637522512] conn(0x7f281802fe60 0x7f2818032320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:07:08.219 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:08.217+0000 7f2835d9b700 1 --2- 172.21.15.50:0/3421289954 >> [v2:172.21.15.50:6800/2637522512,v1:172.21.15.50:6801/2637522512] conn(0x7f281802fe60 0x7f2818032320 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f28200096a0 tx=0x7f2820006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:07:08.221 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:08.221+0000 7f282effd700 1 -- 172.21.15.50:0/3421289954 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2828010540 con 0x7f2838070420 2023-12-25T20:07:08.379 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:08.379+0000 7f283cdf6700 1 -- 172.21.15.50:0/3421289954 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7f2838101e30 con 0x7f2838070420 2023-12-25T20:07:08.382 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:08.381+0000 7f282effd700 1 -- 172.21.15.50:0/3421289954 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v10)=0 v10) v1 ==== 130+0+0 (secure 0 0 0) 0x7f2828016070 con 0x7f2838070420 2023-12-25T20:07:08.385 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:08.383+0000 7f283cdf6700 1 -- 172.21.15.50:0/3421289954 >> [v2:172.21.15.50:6800/2637522512,v1:172.21.15.50:6801/2637522512] conn(0x7f281802fe60 msgr2=0x7f2818032320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:07:08.385 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:08.383+0000 7f283cdf6700 1 --2- 172.21.15.50:0/3421289954 >> [v2:172.21.15.50:6800/2637522512,v1:172.21.15.50:6801/2637522512] conn(0x7f281802fe60 0x7f2818032320 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f28200096a0 tx=0x7f2820006b40).stop 2023-12-25T20:07:08.385 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:08.383+0000 7f283cdf6700 1 -- 172.21.15.50:0/3421289954 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2838070420 msgr2=0x7f283810ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:07:08.385 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:08.383+0000 7f283cdf6700 1 --2- 172.21.15.50:0/3421289954 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2838070420 0x7f283810ccc0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f282800b1d0 tx=0x7f2828006ee0).stop 2023-12-25T20:07:08.385 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:08.383+0000 7f283cdf6700 1 -- 172.21.15.50:0/3421289954 shutdown_connections 2023-12-25T20:07:08.385 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:08.383+0000 7f283cdf6700 1 --2- 172.21.15.50:0/3421289954 >> [v2:172.21.15.50:6800/2637522512,v1:172.21.15.50:6801/2637522512] conn(0x7f281802fe60 0x7f2818032320 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:07:08.385 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:08.383+0000 7f283cdf6700 1 --2- 172.21.15.50:0/3421289954 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2838070420 0x7f283810ccc0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:07:08.385 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:08.383+0000 7f283cdf6700 1 -- 172.21.15.50:0/3421289954 >> 172.21.15.50:0/3421289954 conn(0x7f28380f63b0 msgr2=0x7f28380f7e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:07:08.385 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:08.384+0000 7f283cdf6700 1 -- 172.21.15.50:0/3421289954 shutdown_connections 2023-12-25T20:07:08.385 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:08.384+0000 7f283cdf6700 1 -- 172.21.15.50:0/3421289954 wait complete. 2023-12-25T20:07:08.730 INFO:teuthology.orchestra.run.smithi050.stderr:Enabling the dashboard module... 2023-12-25T20:07:08.730 INFO:teuthology.orchestra.run.smithi050.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi050 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/02c8f25c-a361-11ee-95a5-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmps7kf4jlu:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpu2g3lwcx:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable dashboard 2023-12-25T20:07:09.153 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:09.150+0000 7fef09328700 1 Processor -- start 2023-12-25T20:07:09.153 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:09.150+0000 7fef09328700 1 -- start start 2023-12-25T20:07:09.153 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:09.151+0000 7fef09328700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fef040703d0 0x7fef040fe940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-25T20:07:09.153 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:09.151+0000 7fef09328700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef040fee30 con 0x7fef040703d0 2023-12-25T20:07:09.153 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:09.151+0000 7fef02ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fef040703d0 0x7fef040fe940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:07:09.153 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:09.151+0000 7fef02ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fef040703d0 0x7fef040fe940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:40980/0 (socket says 172.21.15.50:40980) 2023-12-25T20:07:09.153 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:09.151+0000 7fef02ffd700 1 -- 172.21.15.50:0/2577471726 learned_addr learned my addr 172.21.15.50:0/2577471726 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-25T20:07:09.153 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:09.151+0000 7fef02ffd700 1 -- 172.21.15.50:0/2577471726 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef040fef70 con 0x7fef040703d0 2023-12-25T20:07:09.153 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:09.151+0000 7fef02ffd700 1 --2- 172.21.15.50:0/2577471726 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fef040703d0 0x7fef040fe940 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7feef4009960 tx=0x7feef40092d0).ready entity=mon.0 client_cookie=be4d1ca9f7f81788 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:07:09.153 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:09.151+0000 7fef01ffb700 1 -- 172.21.15.50:0/2577471726 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feef400b9c0 con 0x7fef040703d0 2023-12-25T20:07:09.153 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:09.151+0000 7fef01ffb700 1 -- 172.21.15.50:0/2577471726 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7feef4011400 con 0x7fef040703d0 2023-12-25T20:07:09.153 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:09.151+0000 7fef01ffb700 1 -- 172.21.15.50:0/2577471726 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feef4010580 con 0x7fef040703d0 2023-12-25T20:07:09.154 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:09.152+0000 7fef09328700 1 -- 172.21.15.50:0/2577471726 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fef040703d0 msgr2=0x7fef040fe940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:07:09.154 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:09.152+0000 7fef09328700 1 --2- 172.21.15.50:0/2577471726 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fef040703d0 0x7fef040fe940 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7feef4009960 tx=0x7feef40092d0).stop 2023-12-25T20:07:09.156 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:09.152+0000 7fef09328700 1 -- 172.21.15.50:0/2577471726 shutdown_connections 2023-12-25T20:07:09.156 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:09.152+0000 7fef09328700 1 --2- 172.21.15.50:0/2577471726 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fef040703d0 0x7fef040fe940 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:07:09.156 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:09.152+0000 7fef09328700 1 -- 172.21.15.50:0/2577471726 >> 172.21.15.50:0/2577471726 conn(0x7fef040f63c0 msgr2=0x7fef040f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:07:09.156 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:09.152+0000 7fef09328700 1 -- 172.21.15.50:0/2577471726 shutdown_connections 2023-12-25T20:07:09.156 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:09.152+0000 7fef09328700 1 -- 172.21.15.50:0/2577471726 wait complete. 2023-12-25T20:07:09.156 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:09.153+0000 7fef09328700 1 Processor -- start 2023-12-25T20:07:09.157 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:09.153+0000 7fef09328700 1 -- start start 2023-12-25T20:07:09.157 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:09.153+0000 7fef09328700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fef040703d0 0x7fef040fc3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:07:09.157 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:09.153+0000 7fef09328700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef040fa840 con 0x7fef040703d0 2023-12-25T20:07:09.157 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:09.153+0000 7fef02ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fef040703d0 0x7fef040fc3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:07:09.157 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:09.153+0000 7fef02ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fef040703d0 0x7fef040fc3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:40994/0 (socket says 172.21.15.50:40994) 2023-12-25T20:07:09.157 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:09.153+0000 7fef02ffd700 1 -- 172.21.15.50:0/846847057 learned_addr learned my addr 172.21.15.50:0/846847057 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-25T20:07:09.157 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:09.154+0000 7fef02ffd700 1 -- 172.21.15.50:0/846847057 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feef4009570 con 0x7fef040703d0 2023-12-25T20:07:09.157 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:09.154+0000 7fef02ffd700 1 --2- 172.21.15.50:0/846847057 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fef040703d0 0x7fef040fc3c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7feef4009f20 tx=0x7feef40092d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:07:09.157 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:09.154+0000 7feeeb7fe700 1 -- 172.21.15.50:0/846847057 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feef40103a0 con 0x7fef040703d0 2023-12-25T20:07:09.157 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:09.154+0000 7feeeb7fe700 1 -- 172.21.15.50:0/846847057 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7feef4010ca0 con 0x7fef040703d0 2023-12-25T20:07:09.157 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:09.154+0000 7fef09328700 1 -- 172.21.15.50:0/846847057 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef040fa9e0 con 0x7fef040703d0 2023-12-25T20:07:09.157 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:09.154+0000 7feeeb7fe700 1 -- 172.21.15.50:0/846847057 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feef401ada0 con 0x7fef040703d0 2023-12-25T20:07:09.157 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:09.154+0000 7fef09328700 1 -- 172.21.15.50:0/846847057 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef040fadf0 con 0x7fef040703d0 2023-12-25T20:07:09.158 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:09.155+0000 7feeeb7fe700 1 -- 172.21.15.50:0/846847057 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7feef400f070 con 0x7fef040703d0 2023-12-25T20:07:09.158 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:09.155+0000 7feeeb7fe700 1 --2- 172.21.15.50:0/846847057 >> [v2:172.21.15.50:6800/2637522512,v1:172.21.15.50:6801/2637522512] conn(0x7feef002feb0 0x7feef0032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:07:09.158 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:09.155+0000 7feeeb7fe700 1 -- 172.21.15.50:0/846847057 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7feef4042cb0 con 0x7fef040703d0 2023-12-25T20:07:09.158 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:09.155+0000 7fef027fc700 1 --2- 172.21.15.50:0/846847057 >> [v2:172.21.15.50:6800/2637522512,v1:172.21.15.50:6801/2637522512] conn(0x7feef002feb0 0x7feef0032370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:07:09.158 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:09.155+0000 7fef09328700 1 -- 172.21.15.50:0/846847057 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef04059100 con 0x7fef040703d0 2023-12-25T20:07:09.158 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:09.155+0000 7fef027fc700 1 --2- 172.21.15.50:0/846847057 >> [v2:172.21.15.50:6800/2637522512,v1:172.21.15.50:6801/2637522512] conn(0x7feef002feb0 0x7feef0032370 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7feef80096a0 tx=0x7feef8006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:07:09.160 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:09.159+0000 7feeeb7fe700 1 -- 172.21.15.50:0/846847057 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7feef401e070 con 0x7fef040703d0 2023-12-25T20:07:09.361 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:09.361+0000 7fef09328700 1 -- 172.21.15.50:0/846847057 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7fef04045ad0 con 0x7fef040703d0 2023-12-25T20:07:09.384 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:09.383+0000 7feeeb7fe700 1 -- 172.21.15.50:0/846847057 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mgrmap(e 13) v1 ==== 36208+0+0 (secure 0 0 0) 0x7feef4032090 con 0x7fef040703d0 2023-12-25T20:07:09.384 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:09.384+0000 7feeeb7fe700 1 -- 172.21.15.50:0/846847057 <== mon.0 v2:172.21.15.50:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v13) v1 ==== 88+0+0 (secure 0 0 0) 0x7feef40223c0 con 0x7fef040703d0 2023-12-25T20:07:09.387 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:09.386+0000 7fef09328700 1 -- 172.21.15.50:0/846847057 >> [v2:172.21.15.50:6800/2637522512,v1:172.21.15.50:6801/2637522512] conn(0x7feef002feb0 msgr2=0x7feef0032370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:07:09.388 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:09.386+0000 7fef09328700 1 --2- 172.21.15.50:0/846847057 >> [v2:172.21.15.50:6800/2637522512,v1:172.21.15.50:6801/2637522512] conn(0x7feef002feb0 0x7feef0032370 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7feef80096a0 tx=0x7feef8006b40).stop 2023-12-25T20:07:09.388 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:09.386+0000 7fef09328700 1 -- 172.21.15.50:0/846847057 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fef040703d0 msgr2=0x7fef040fc3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:07:09.388 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:09.386+0000 7fef09328700 1 --2- 172.21.15.50:0/846847057 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fef040703d0 0x7fef040fc3c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7feef4009f20 tx=0x7feef40092d0).stop 2023-12-25T20:07:09.388 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:09.386+0000 7fef09328700 1 -- 172.21.15.50:0/846847057 shutdown_connections 2023-12-25T20:07:09.388 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:09.386+0000 7fef09328700 1 --2- 172.21.15.50:0/846847057 >> [v2:172.21.15.50:6800/2637522512,v1:172.21.15.50:6801/2637522512] conn(0x7feef002feb0 0x7feef0032370 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:07:09.388 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:09.386+0000 7fef09328700 1 --2- 172.21.15.50:0/846847057 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fef040703d0 0x7fef040fc3c0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:07:09.388 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:09.386+0000 7fef09328700 1 -- 172.21.15.50:0/846847057 >> 172.21.15.50:0/846847057 conn(0x7fef040f63c0 msgr2=0x7fef040f7ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:07:09.388 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:09.386+0000 7fef09328700 1 -- 172.21.15.50:0/846847057 shutdown_connections 2023-12-25T20:07:09.388 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:09.386+0000 7fef09328700 1 -- 172.21.15.50:0/846847057 wait complete. 2023-12-25T20:07:09.718 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:07:09 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:07:08.382992+0000 mon.smithi050 (mon.0) 87 : audit [INF] from='client.? 172.21.15.50:0/3421289954' entity='client.admin' 2023-12-25T20:07:09.719 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:07:09 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:07:09.362634+0000 mon.smithi050 (mon.0) 88 : audit [INF] from='client.? 172.21.15.50:0/846847057' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2023-12-25T20:07:09.779 INFO:teuthology.orchestra.run.smithi050.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi050 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/02c8f25c-a361-11ee-95a5-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmps7kf4jlu:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpu2g3lwcx:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr stat 2023-12-25T20:07:10.169 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:10.163+0000 7fbd408e8700 1 Processor -- start 2023-12-25T20:07:10.169 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:10.163+0000 7fbd408e8700 1 -- start start 2023-12-25T20:07:10.169 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:10.164+0000 7fbd408e8700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbd3c0fab80 0x7fbd3c0fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-25T20:07:10.169 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:10.164+0000 7fbd408e8700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd3c0fb490 con 0x7fbd3c0fab80 2023-12-25T20:07:10.169 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:10.164+0000 7fbd3a59c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbd3c0fab80 0x7fbd3c0fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:07:10.169 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:10.164+0000 7fbd3a59c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbd3c0fab80 0x7fbd3c0fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:41020/0 (socket says 172.21.15.50:41020) 2023-12-25T20:07:10.169 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:10.164+0000 7fbd3a59c700 1 -- 172.21.15.50:0/2476566582 learned_addr learned my addr 172.21.15.50:0/2476566582 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-25T20:07:10.169 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:10.164+0000 7fbd3a59c700 1 -- 172.21.15.50:0/2476566582 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd3c0fb5d0 con 0x7fbd3c0fab80 2023-12-25T20:07:10.170 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:10.164+0000 7fbd3a59c700 1 --2- 172.21.15.50:0/2476566582 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbd3c0fab80 0x7fbd3c0fafa0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7fbd2c00b5f0 tx=0x7fbd2c006ee0).ready entity=mon.0 client_cookie=cafd783561cb80ac server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:07:10.170 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:10.164+0000 7fbd3959a700 1 -- 172.21.15.50:0/2476566582 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbd2c00d600 con 0x7fbd3c0fab80 2023-12-25T20:07:10.170 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:10.164+0000 7fbd3959a700 1 -- 172.21.15.50:0/2476566582 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbd2c00dc10 con 0x7fbd3c0fab80 2023-12-25T20:07:10.170 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:10.164+0000 7fbd3959a700 1 -- 172.21.15.50:0/2476566582 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbd2c011cd0 con 0x7fbd3c0fab80 2023-12-25T20:07:10.170 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:10.165+0000 7fbd408e8700 1 -- 172.21.15.50:0/2476566582 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbd3c0fab80 msgr2=0x7fbd3c0fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:07:10.170 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:10.165+0000 7fbd408e8700 1 --2- 172.21.15.50:0/2476566582 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbd3c0fab80 0x7fbd3c0fafa0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7fbd2c00b5f0 tx=0x7fbd2c006ee0).stop 2023-12-25T20:07:10.170 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:10.165+0000 7fbd408e8700 1 -- 172.21.15.50:0/2476566582 shutdown_connections 2023-12-25T20:07:10.170 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:10.165+0000 7fbd408e8700 1 --2- 172.21.15.50:0/2476566582 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbd3c0fab80 0x7fbd3c0fafa0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:07:10.170 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:10.165+0000 7fbd408e8700 1 -- 172.21.15.50:0/2476566582 >> 172.21.15.50:0/2476566582 conn(0x7fbd3c0f63c0 msgr2=0x7fbd3c0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:07:10.170 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:10.165+0000 7fbd408e8700 1 -- 172.21.15.50:0/2476566582 shutdown_connections 2023-12-25T20:07:10.170 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:10.165+0000 7fbd408e8700 1 -- 172.21.15.50:0/2476566582 wait complete. 2023-12-25T20:07:10.170 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:10.166+0000 7fbd408e8700 1 Processor -- start 2023-12-25T20:07:10.170 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:10.166+0000 7fbd408e8700 1 -- start start 2023-12-25T20:07:10.171 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:10.166+0000 7fbd408e8700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbd3c0fab80 0x7fbd3c10aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:07:10.171 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:10.166+0000 7fbd408e8700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd3c10af30 con 0x7fbd3c0fab80 2023-12-25T20:07:10.171 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:10.166+0000 7fbd3a59c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbd3c0fab80 0x7fbd3c10aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:07:10.171 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:10.166+0000 7fbd3a59c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbd3c0fab80 0x7fbd3c10aa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:41032/0 (socket says 172.21.15.50:41032) 2023-12-25T20:07:10.171 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:10.166+0000 7fbd3a59c700 1 -- 172.21.15.50:0/2349140916 learned_addr learned my addr 172.21.15.50:0/2349140916 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-25T20:07:10.171 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:10.167+0000 7fbd3a59c700 1 -- 172.21.15.50:0/2349140916 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd2c00b200 con 0x7fbd3c0fab80 2023-12-25T20:07:10.171 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:10.167+0000 7fbd3a59c700 1 --2- 172.21.15.50:0/2349140916 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbd3c0fab80 0x7fbd3c10aa40 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7fbd2c00e040 tx=0x7fbd2c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:07:10.171 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:10.167+0000 7fbd2affd700 1 -- 172.21.15.50:0/2349140916 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbd2c01b410 con 0x7fbd3c0fab80 2023-12-25T20:07:10.171 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:10.167+0000 7fbd2affd700 1 -- 172.21.15.50:0/2349140916 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbd2c01ba20 con 0x7fbd3c0fab80 2023-12-25T20:07:10.171 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:10.167+0000 7fbd408e8700 1 -- 172.21.15.50:0/2349140916 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd3c10b130 con 0x7fbd3c0fab80 2023-12-25T20:07:10.171 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:10.167+0000 7fbd408e8700 1 -- 172.21.15.50:0/2349140916 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd3c10b540 con 0x7fbd3c0fab80 2023-12-25T20:07:10.171 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:10.168+0000 7fbd2affd700 1 -- 172.21.15.50:0/2349140916 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbd2c01a910 con 0x7fbd3c0fab80 2023-12-25T20:07:10.171 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:10.168+0000 7fbd2affd700 1 -- 172.21.15.50:0/2349140916 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 13) v1 ==== 36208+0+0 (secure 0 0 0) 0x7fbd2c00f070 con 0x7fbd3c0fab80 2023-12-25T20:07:10.171 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:10.168+0000 7fbd2affd700 1 --2- 172.21.15.50:0/2349140916 >> [v2:172.21.15.50:6800/2637522512,v1:172.21.15.50:6801/2637522512] conn(0x7fbd2002ff00 0x7fbd200323c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:07:10.172 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:10.168+0000 7fbd2affd700 1 -- 172.21.15.50:0/2349140916 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7fbd2c042070 con 0x7fbd3c0fab80 2023-12-25T20:07:10.172 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:10.168+0000 7fbd39d9b700 1 -- 172.21.15.50:0/2349140916 >> [v2:172.21.15.50:6800/2637522512,v1:172.21.15.50:6801/2637522512] conn(0x7fbd2002ff00 msgr2=0x7fbd200323c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:6800/2637522512 2023-12-25T20:07:10.172 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:10.168+0000 7fbd39d9b700 1 --2- 172.21.15.50:0/2349140916 >> [v2:172.21.15.50:6800/2637522512,v1:172.21.15.50:6801/2637522512] conn(0x7fbd2002ff00 0x7fbd200323c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-12-25T20:07:10.172 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:10.168+0000 7fbd408e8700 1 -- 172.21.15.50:0/2349140916 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd18004fa0 con 0x7fbd3c0fab80 2023-12-25T20:07:10.172 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:10.172+0000 7fbd2affd700 1 -- 172.21.15.50:0/2349140916 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbd2c016070 con 0x7fbd3c0fab80 2023-12-25T20:07:10.369 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:10.369+0000 7fbd39d9b700 1 -- 172.21.15.50:0/2349140916 >> [v2:172.21.15.50:6800/2637522512,v1:172.21.15.50:6801/2637522512] conn(0x7fbd2002ff00 msgr2=0x7fbd200323c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:6800/2637522512 2023-12-25T20:07:10.369 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:10.369+0000 7fbd39d9b700 1 --2- 172.21.15.50:0/2349140916 >> [v2:172.21.15.50:6800/2637522512,v1:172.21.15.50:6801/2637522512] conn(0x7fbd2002ff00 0x7fbd200323c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-12-25T20:07:10.375 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:10.371+0000 7fbd408e8700 1 -- 172.21.15.50:0/2349140916 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fbd18005e80 con 0x7fbd3c0fab80 2023-12-25T20:07:10.375 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:10.371+0000 7fbd2affd700 1 -- 172.21.15.50:0/2349140916 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v13) v1 ==== 56+0+104 (secure 0 0 0) 0x7fbd2c0109b0 con 0x7fbd3c0fab80 2023-12-25T20:07:10.375 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: { 2023-12-25T20:07:10.375 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "epoch": 13, 2023-12-25T20:07:10.375 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "available": true, 2023-12-25T20:07:10.376 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "active_name": "smithi050.nbzhun", 2023-12-25T20:07:10.376 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "num_standby": 0 2023-12-25T20:07:10.376 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: } 2023-12-25T20:07:10.376 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:10.373+0000 7fbd408e8700 1 -- 172.21.15.50:0/2349140916 >> [v2:172.21.15.50:6800/2637522512,v1:172.21.15.50:6801/2637522512] conn(0x7fbd2002ff00 msgr2=0x7fbd200323c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-25T20:07:10.376 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:10.373+0000 7fbd408e8700 1 --2- 172.21.15.50:0/2349140916 >> [v2:172.21.15.50:6800/2637522512,v1:172.21.15.50:6801/2637522512] conn(0x7fbd2002ff00 0x7fbd200323c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-12-25T20:07:10.376 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:10.373+0000 7fbd408e8700 1 -- 172.21.15.50:0/2349140916 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbd3c0fab80 msgr2=0x7fbd3c10aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:07:10.376 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:10.373+0000 7fbd408e8700 1 --2- 172.21.15.50:0/2349140916 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbd3c0fab80 0x7fbd3c10aa40 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7fbd2c00e040 tx=0x7fbd2c006ee0).stop 2023-12-25T20:07:10.376 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:10.373+0000 7fbd408e8700 1 -- 172.21.15.50:0/2349140916 shutdown_connections 2023-12-25T20:07:10.376 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:10.373+0000 7fbd408e8700 1 --2- 172.21.15.50:0/2349140916 >> [v2:172.21.15.50:6800/2637522512,v1:172.21.15.50:6801/2637522512] conn(0x7fbd2002ff00 0x7fbd200323c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-12-25T20:07:10.376 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:10.373+0000 7fbd408e8700 1 --2- 172.21.15.50:0/2349140916 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbd3c0fab80 0x7fbd3c10aa40 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:07:10.376 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:10.373+0000 7fbd408e8700 1 -- 172.21.15.50:0/2349140916 >> 172.21.15.50:0/2349140916 conn(0x7fbd3c0f63c0 msgr2=0x7fbd3c0f7e80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:07:10.376 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:10.373+0000 7fbd408e8700 1 -- 172.21.15.50:0/2349140916 shutdown_connections 2023-12-25T20:07:10.376 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:10.373+0000 7fbd408e8700 1 -- 172.21.15.50:0/2349140916 wait complete. 2023-12-25T20:07:10.685 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:07:10 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:07:09.385593+0000 mon.smithi050 (mon.0) 89 : audit [INF] from='client.? 172.21.15.50:0/846847057' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2023-12-25T20:07:10.686 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:07:10 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:07:09.385646+0000 mon.smithi050 (mon.0) 90 : cluster [DBG] mgrmap e13: smithi050.nbzhun(active, since 7s) 2023-12-25T20:07:10.686 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:07:10 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:07:10.372000+0000 mon.smithi050 (mon.0) 91 : audit [DBG] from='client.? 172.21.15.50:0/2349140916' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-12-25T20:07:10.731 INFO:teuthology.orchestra.run.smithi050.stderr:Waiting for the mgr to restart... 2023-12-25T20:07:10.731 INFO:teuthology.orchestra.run.smithi050.stderr:Waiting for mgr epoch 13... 2023-12-25T20:07:10.731 INFO:teuthology.orchestra.run.smithi050.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi050 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/02c8f25c-a361-11ee-95a5-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmps7kf4jlu:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpu2g3lwcx:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 tell mgr mgr_status 2023-12-25T20:07:11.148 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:11.146+0000 7fdda31fe700 1 Processor -- start 2023-12-25T20:07:11.148 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:11.146+0000 7fdda31fe700 1 -- start start 2023-12-25T20:07:11.148 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:11.146+0000 7fdda31fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdd9c070430 0x7fdd9c0fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-25T20:07:11.148 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:11.147+0000 7fdda31fe700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd9c0fedd0 con 0x7fdd9c070430 2023-12-25T20:07:11.148 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:11.147+0000 7fdda0f9a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdd9c070430 0x7fdd9c0fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:07:11.148 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:11.147+0000 7fdda0f9a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdd9c070430 0x7fdd9c0fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:41038/0 (socket says 172.21.15.50:41038) 2023-12-25T20:07:11.148 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:11.147+0000 7fdda0f9a700 1 -- 172.21.15.50:0/519092874 learned_addr learned my addr 172.21.15.50:0/519092874 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-25T20:07:11.148 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:11.147+0000 7fdda0f9a700 1 -- 172.21.15.50:0/519092874 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd9c0fef10 con 0x7fdd9c070430 2023-12-25T20:07:11.148 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:11.147+0000 7fdda0f9a700 1 --2- 172.21.15.50:0/519092874 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdd9c070430 0x7fdd9c0fe8e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7fdd9000b5f0 tx=0x7fdd90006ee0).ready entity=mon.0 client_cookie=bc1fc2e0a1ea4633 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:07:11.148 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:11.147+0000 7fdd9b7fe700 1 -- 172.21.15.50:0/519092874 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdd9000d600 con 0x7fdd9c070430 2023-12-25T20:07:11.148 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:11.147+0000 7fdd9b7fe700 1 -- 172.21.15.50:0/519092874 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdd9000dc10 con 0x7fdd9c070430 2023-12-25T20:07:11.148 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:11.147+0000 7fdd9b7fe700 1 -- 172.21.15.50:0/519092874 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdd90011cd0 con 0x7fdd9c070430 2023-12-25T20:07:11.149 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:11.148+0000 7fdda31fe700 1 -- 172.21.15.50:0/519092874 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdd9c070430 msgr2=0x7fdd9c0fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:07:11.149 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:11.148+0000 7fdda31fe700 1 --2- 172.21.15.50:0/519092874 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdd9c070430 0x7fdd9c0fe8e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7fdd9000b5f0 tx=0x7fdd90006ee0).stop 2023-12-25T20:07:11.150 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:11.148+0000 7fdda31fe700 1 -- 172.21.15.50:0/519092874 shutdown_connections 2023-12-25T20:07:11.150 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:11.148+0000 7fdda31fe700 1 --2- 172.21.15.50:0/519092874 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdd9c070430 0x7fdd9c0fe8e0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:07:11.150 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:11.148+0000 7fdda31fe700 1 -- 172.21.15.50:0/519092874 >> 172.21.15.50:0/519092874 conn(0x7fdd9c0f63c0 msgr2=0x7fdd9c0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:07:11.150 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:11.148+0000 7fdda31fe700 1 -- 172.21.15.50:0/519092874 shutdown_connections 2023-12-25T20:07:11.150 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:11.148+0000 7fdda31fe700 1 -- 172.21.15.50:0/519092874 wait complete. 2023-12-25T20:07:11.150 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:11.149+0000 7fdda31fe700 1 Processor -- start 2023-12-25T20:07:11.150 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:11.149+0000 7fdda31fe700 1 -- start start 2023-12-25T20:07:11.151 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:11.149+0000 7fdda31fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdd9c070430 0x7fdd9c10ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:07:11.151 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:11.149+0000 7fdda31fe700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd9c10aff0 con 0x7fdd9c070430 2023-12-25T20:07:11.151 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:11.149+0000 7fdda0f9a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdd9c070430 0x7fdd9c10ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:07:11.151 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:11.149+0000 7fdda0f9a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdd9c070430 0x7fdd9c10ab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:41044/0 (socket says 172.21.15.50:41044) 2023-12-25T20:07:11.151 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:11.149+0000 7fdda0f9a700 1 -- 172.21.15.50:0/3713946862 learned_addr learned my addr 172.21.15.50:0/3713946862 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-25T20:07:11.151 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:11.150+0000 7fdda0f9a700 1 -- 172.21.15.50:0/3713946862 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd9000b200 con 0x7fdd9c070430 2023-12-25T20:07:11.152 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:11.150+0000 7fdda0f9a700 1 --2- 172.21.15.50:0/3713946862 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdd9c070430 0x7fdd9c10ab00 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7fdd90006800 tx=0x7fdd90006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:07:11.152 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:11.150+0000 7fdd997fa700 1 -- 172.21.15.50:0/3713946862 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdd90011df0 con 0x7fdd9c070430 2023-12-25T20:07:11.152 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:11.150+0000 7fdda31fe700 1 -- 172.21.15.50:0/3713946862 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd9c10b1f0 con 0x7fdd9c070430 2023-12-25T20:07:11.152 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:11.150+0000 7fdd997fa700 1 -- 172.21.15.50:0/3713946862 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdd9001b8c0 con 0x7fdd9c070430 2023-12-25T20:07:11.152 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:11.150+0000 7fdd997fa700 1 -- 172.21.15.50:0/3713946862 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdd9001a830 con 0x7fdd9c070430 2023-12-25T20:07:11.152 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:11.150+0000 7fdda31fe700 1 -- 172.21.15.50:0/3713946862 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd9c10b600 con 0x7fdd9c070430 2023-12-25T20:07:11.152 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:11.151+0000 7fdd997fa700 1 -- 172.21.15.50:0/3713946862 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 13) v1 ==== 36208+0+0 (secure 0 0 0) 0x7fdd9000f070 con 0x7fdd9c070430 2023-12-25T20:07:11.152 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:11.151+0000 7fdd997fa700 1 --2- 172.21.15.50:0/3713946862 >> [v2:172.21.15.50:6800/2637522512,v1:172.21.15.50:6801/2637522512] conn(0x7fdd8002feb0 0x7fdd80032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:07:11.152 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:11.151+0000 7fdda31fe700 1 -- 172.21.15.50:0/3713946862 --> [v2:172.21.15.50:6800/2637522512,v1:172.21.15.50:6801/2637522512] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdd9c059100 con 0x7fdd8002feb0 2023-12-25T20:07:11.152 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:11.151+0000 7fdd997fa700 1 -- 172.21.15.50:0/3713946862 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7fdd90041100 con 0x7fdd9c070430 2023-12-25T20:07:11.152 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:11.151+0000 7fdd9bfff700 1 -- 172.21.15.50:0/3713946862 >> [v2:172.21.15.50:6800/2637522512,v1:172.21.15.50:6801/2637522512] conn(0x7fdd8002feb0 msgr2=0x7fdd80032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:6800/2637522512 2023-12-25T20:07:11.152 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:11.151+0000 7fdd9bfff700 1 --2- 172.21.15.50:0/3713946862 >> [v2:172.21.15.50:6800/2637522512,v1:172.21.15.50:6801/2637522512] conn(0x7fdd8002feb0 0x7fdd80032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-12-25T20:07:11.352 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:11.352+0000 7fdd9bfff700 1 -- 172.21.15.50:0/3713946862 >> [v2:172.21.15.50:6800/2637522512,v1:172.21.15.50:6801/2637522512] conn(0x7fdd8002feb0 msgr2=0x7fdd80032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:6800/2637522512 2023-12-25T20:07:11.352 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:11.352+0000 7fdd9bfff700 1 --2- 172.21.15.50:0/3713946862 >> [v2:172.21.15.50:6800/2637522512,v1:172.21.15.50:6801/2637522512] conn(0x7fdd8002feb0 0x7fdd80032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-12-25T20:07:11.753 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:11.753+0000 7fdd9bfff700 1 -- 172.21.15.50:0/3713946862 >> [v2:172.21.15.50:6800/2637522512,v1:172.21.15.50:6801/2637522512] conn(0x7fdd8002feb0 msgr2=0x7fdd80032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:6800/2637522512 2023-12-25T20:07:11.753 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:11.753+0000 7fdd9bfff700 1 --2- 172.21.15.50:0/3713946862 >> [v2:172.21.15.50:6800/2637522512,v1:172.21.15.50:6801/2637522512] conn(0x7fdd8002feb0 0x7fdd80032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-12-25T20:07:12.554 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:12.554+0000 7fdd9bfff700 1 -- 172.21.15.50:0/3713946862 >> [v2:172.21.15.50:6800/2637522512,v1:172.21.15.50:6801/2637522512] conn(0x7fdd8002feb0 msgr2=0x7fdd80032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:6800/2637522512 2023-12-25T20:07:12.554 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:12.554+0000 7fdd9bfff700 1 --2- 172.21.15.50:0/3713946862 >> [v2:172.21.15.50:6800/2637522512,v1:172.21.15.50:6801/2637522512] conn(0x7fdd8002feb0 0x7fdd80032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-12-25T20:07:14.155 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:14.155+0000 7fdd9bfff700 1 -- 172.21.15.50:0/3713946862 >> [v2:172.21.15.50:6800/2637522512,v1:172.21.15.50:6801/2637522512] conn(0x7fdd8002feb0 msgr2=0x7fdd80032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:6800/2637522512 2023-12-25T20:07:14.156 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:14.155+0000 7fdd9bfff700 1 --2- 172.21.15.50:0/3713946862 >> [v2:172.21.15.50:6800/2637522512,v1:172.21.15.50:6801/2637522512] conn(0x7fdd8002feb0 0x7fdd80032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 3.200000 2023-12-25T20:07:14.619 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:14.619+0000 7fdd997fa700 1 -- 172.21.15.50:0/3713946862 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mgrmap(e 14) v1 ==== 35997+0+0 (secure 0 0 0) 0x7fdd90024790 con 0x7fdd9c070430 2023-12-25T20:07:14.620 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:14.619+0000 7fdd997fa700 1 -- 172.21.15.50:0/3713946862 >> [v2:172.21.15.50:6800/2637522512,v1:172.21.15.50:6801/2637522512] conn(0x7fdd8002feb0 msgr2=0x7fdd80032370 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-25T20:07:14.620 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:14.619+0000 7fdd997fa700 1 --2- 172.21.15.50:0/3713946862 >> [v2:172.21.15.50:6800/2637522512,v1:172.21.15.50:6801/2637522512] conn(0x7fdd8002feb0 0x7fdd80032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-12-25T20:07:15.621 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:15.620+0000 7fdd997fa700 1 -- 172.21.15.50:0/3713946862 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mgrmap(e 15) v1 ==== 36067+0+0 (secure 0 0 0) 0x7fdd90041e50 con 0x7fdd9c070430 2023-12-25T20:07:15.621 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:15.620+0000 7fdd997fa700 1 --2- 172.21.15.50:0/3713946862 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fdd80033990 0x7fdd80035d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:07:15.621 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:15.620+0000 7fdd997fa700 1 -- 172.21.15.50:0/3713946862 --> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdd9c059100 con 0x7fdd80033990 2023-12-25T20:07:15.621 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:15.621+0000 7fdd9bfff700 1 --2- 172.21.15.50:0/3713946862 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fdd80033990 0x7fdd80035d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:07:15.622 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:15.621+0000 7fdd9bfff700 1 --2- 172.21.15.50:0/3713946862 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fdd80033990 0x7fdd80035d80 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7fdd88006fd0 tx=0x7fdd88006cf0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:07:15.622 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:15.622+0000 7fdd997fa700 1 -- 172.21.15.50:0/3713946862 <== mgr.14162 v2:172.21.15.50:6800/1043424882 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (secure 0 0 0) 0x7fdd9c059100 con 0x7fdd80033990 2023-12-25T20:07:15.628 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:15.628+0000 7fdda31fe700 1 -- 172.21.15.50:0/3713946862 --> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fdd9c045ad0 con 0x7fdd80033990 2023-12-25T20:07:15.630 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:15.629+0000 7fdd997fa700 1 -- 172.21.15.50:0/3713946862 <== mgr.14162 v2:172.21.15.50:6800/1043424882 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7fdd9c045ad0 con 0x7fdd80033990 2023-12-25T20:07:15.630 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: { 2023-12-25T20:07:15.630 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "mgrmap_epoch": 15, 2023-12-25T20:07:15.630 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: "initialized": true 2023-12-25T20:07:15.630 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: } 2023-12-25T20:07:15.630 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:15.629+0000 7fdda31fe700 1 -- 172.21.15.50:0/3713946862 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fdd80033990 msgr2=0x7fdd80035d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:07:15.630 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:15.629+0000 7fdda31fe700 1 --2- 172.21.15.50:0/3713946862 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fdd80033990 0x7fdd80035d80 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7fdd88006fd0 tx=0x7fdd88006cf0).stop 2023-12-25T20:07:15.630 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:15.629+0000 7fdda31fe700 1 -- 172.21.15.50:0/3713946862 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdd9c070430 msgr2=0x7fdd9c10ab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:07:15.631 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:15.629+0000 7fdda31fe700 1 --2- 172.21.15.50:0/3713946862 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdd9c070430 0x7fdd9c10ab00 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7fdd90006800 tx=0x7fdd90006ee0).stop 2023-12-25T20:07:15.631 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:15.630+0000 7fdda31fe700 1 -- 172.21.15.50:0/3713946862 shutdown_connections 2023-12-25T20:07:15.631 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:15.630+0000 7fdda31fe700 1 --2- 172.21.15.50:0/3713946862 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fdd80033990 0x7fdd80035d80 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:07:15.631 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:15.630+0000 7fdda31fe700 1 --2- 172.21.15.50:0/3713946862 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdd9c070430 0x7fdd9c10ab00 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:07:15.631 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:15.630+0000 7fdda31fe700 1 --2- 172.21.15.50:0/3713946862 >> [v2:172.21.15.50:6800/2637522512,v1:172.21.15.50:6801/2637522512] conn(0x7fdd8002feb0 0x7fdd80032370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-12-25T20:07:15.631 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:15.630+0000 7fdda31fe700 1 -- 172.21.15.50:0/3713946862 >> 172.21.15.50:0/3713946862 conn(0x7fdd9c0f63c0 msgr2=0x7fdd9c0f7f80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:07:15.631 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:15.630+0000 7fdda31fe700 1 -- 172.21.15.50:0/3713946862 shutdown_connections 2023-12-25T20:07:15.631 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:15.631+0000 7fdda31fe700 1 -- 172.21.15.50:0/3713946862 wait complete. 2023-12-25T20:07:15.794 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:07:15 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:07:14.567218+0000 mon.smithi050 (mon.0) 92 : cluster [INF] Active manager daemon smithi050.nbzhun restarted 2023-12-25T20:07:15.794 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:07:15 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:07:14.567658+0000 mon.smithi050 (mon.0) 93 : cluster [INF] Activating manager daemon smithi050.nbzhun 2023-12-25T20:07:15.794 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:07:15 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:07:14.569238+0000 mon.smithi050 (mon.0) 94 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2023-12-25T20:07:15.795 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:07:15 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:07:14.620048+0000 mon.smithi050 (mon.0) 95 : cluster [DBG] mgrmap e14: smithi050.nbzhun(active, starting, since 0.0525157s) 2023-12-25T20:07:15.795 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:07:15 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:07:14.621198+0000 mon.smithi050 (mon.0) 96 : audit [DBG] from='mgr.14162 172.21.15.50:0/3178367813' entity='mgr.smithi050.nbzhun' cmd=[{"prefix": "mon metadata", "id": "smithi050"}]: dispatch 2023-12-25T20:07:15.795 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:07:15 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:07:14.621839+0000 mon.smithi050 (mon.0) 97 : audit [DBG] from='mgr.14162 172.21.15.50:0/3178367813' entity='mgr.smithi050.nbzhun' cmd=[{"prefix": "mgr metadata", "who": "smithi050.nbzhun", "id": "smithi050.nbzhun"}]: dispatch 2023-12-25T20:07:15.795 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:07:15 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:07:14.622538+0000 mon.smithi050 (mon.0) 98 : audit [DBG] from='mgr.14162 172.21.15.50:0/3178367813' entity='mgr.smithi050.nbzhun' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-12-25T20:07:15.795 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:07:15 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:07:14.622695+0000 mon.smithi050 (mon.0) 99 : audit [DBG] from='mgr.14162 172.21.15.50:0/3178367813' entity='mgr.smithi050.nbzhun' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-12-25T20:07:15.795 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:07:15 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:07:14.622829+0000 mon.smithi050 (mon.0) 100 : audit [DBG] from='mgr.14162 172.21.15.50:0/3178367813' entity='mgr.smithi050.nbzhun' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-12-25T20:07:15.795 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:07:15 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:07:14.636852+0000 mon.smithi050 (mon.0) 101 : cluster [INF] Manager daemon smithi050.nbzhun is now available 2023-12-25T20:07:15.795 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:07:15 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:07:14.740832+0000 mon.smithi050 (mon.0) 102 : audit [DBG] from='mgr.14162 172.21.15.50:0/3178367813' entity='mgr.smithi050.nbzhun' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-25T20:07:15.796 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:07:15 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:07:14.742863+0000 mon.smithi050 (mon.0) 103 : audit [DBG] from='mgr.14162 172.21.15.50:0/3178367813' entity='mgr.smithi050.nbzhun' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-25T20:07:15.796 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:07:15 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:07:14.759570+0000 mon.smithi050 (mon.0) 104 : audit [INF] from='mgr.14162 172.21.15.50:0/3178367813' entity='mgr.smithi050.nbzhun' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi050.nbzhun/mirror_snapshot_schedule"}]: dispatch 2023-12-25T20:07:15.796 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:07:15 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:07:14.761740+0000 mon.smithi050 (mon.0) 105 : audit [INF] from='mgr.14162 172.21.15.50:0/3178367813' entity='mgr.smithi050.nbzhun' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi050.nbzhun/trash_purge_schedule"}]: dispatch 2023-12-25T20:07:15.987 INFO:teuthology.orchestra.run.smithi050.stderr:mgr epoch 13 is available 2023-12-25T20:07:15.987 INFO:teuthology.orchestra.run.smithi050.stderr:Generating a dashboard self-signed certificate... 2023-12-25T20:07:15.987 INFO:teuthology.orchestra.run.smithi050.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi050 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/02c8f25c-a361-11ee-95a5-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmps7kf4jlu:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpu2g3lwcx:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 dashboard create-self-signed-cert 2023-12-25T20:07:16.357 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:16.356+0000 7fbc822d0700 1 Processor -- start 2023-12-25T20:07:16.357 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:16.356+0000 7fbc822d0700 1 -- start start 2023-12-25T20:07:16.357 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:16.356+0000 7fbc822d0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbc7c070430 0x7fbc7c0fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-25T20:07:16.357 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:16.356+0000 7fbc822d0700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc7c0fedd0 con 0x7fbc7c070430 2023-12-25T20:07:16.357 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:16.356+0000 7fbc7bfff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbc7c070430 0x7fbc7c0fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:07:16.357 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:16.356+0000 7fbc7bfff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbc7c070430 0x7fbc7c0fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:41082/0 (socket says 172.21.15.50:41082) 2023-12-25T20:07:16.357 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:16.356+0000 7fbc7bfff700 1 -- 172.21.15.50:0/1802913135 learned_addr learned my addr 172.21.15.50:0/1802913135 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-25T20:07:16.357 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:16.357+0000 7fbc7bfff700 1 -- 172.21.15.50:0/1802913135 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc7c0fef10 con 0x7fbc7c070430 2023-12-25T20:07:16.357 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:16.357+0000 7fbc7bfff700 1 --2- 172.21.15.50:0/1802913135 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbc7c070430 0x7fbc7c0fe8e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7fbc6c00b5f0 tx=0x7fbc6c006ee0).ready entity=mon.0 client_cookie=d18e833c005d0ac9 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:07:16.357 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:16.357+0000 7fbc7affd700 1 -- 172.21.15.50:0/1802913135 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbc6c00d600 con 0x7fbc7c070430 2023-12-25T20:07:16.358 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:16.357+0000 7fbc7affd700 1 -- 172.21.15.50:0/1802913135 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbc6c00dc10 con 0x7fbc7c070430 2023-12-25T20:07:16.358 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:16.357+0000 7fbc7affd700 1 -- 172.21.15.50:0/1802913135 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbc6c011cd0 con 0x7fbc7c070430 2023-12-25T20:07:16.360 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:16.357+0000 7fbc822d0700 1 -- 172.21.15.50:0/1802913135 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbc7c070430 msgr2=0x7fbc7c0fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:07:16.360 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:16.357+0000 7fbc822d0700 1 --2- 172.21.15.50:0/1802913135 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbc7c070430 0x7fbc7c0fe8e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7fbc6c00b5f0 tx=0x7fbc6c006ee0).stop 2023-12-25T20:07:16.360 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:16.357+0000 7fbc822d0700 1 -- 172.21.15.50:0/1802913135 shutdown_connections 2023-12-25T20:07:16.360 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:16.357+0000 7fbc822d0700 1 --2- 172.21.15.50:0/1802913135 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbc7c070430 0x7fbc7c0fe8e0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:07:16.360 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:16.357+0000 7fbc822d0700 1 -- 172.21.15.50:0/1802913135 >> 172.21.15.50:0/1802913135 conn(0x7fbc7c0f63c0 msgr2=0x7fbc7c0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:07:16.360 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:16.357+0000 7fbc822d0700 1 -- 172.21.15.50:0/1802913135 shutdown_connections 2023-12-25T20:07:16.360 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:16.357+0000 7fbc822d0700 1 -- 172.21.15.50:0/1802913135 wait complete. 2023-12-25T20:07:16.360 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:16.358+0000 7fbc822d0700 1 Processor -- start 2023-12-25T20:07:16.360 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:16.358+0000 7fbc822d0700 1 -- start start 2023-12-25T20:07:16.360 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:16.359+0000 7fbc822d0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbc7c070430 0x7fbc7c108840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:07:16.361 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:16.359+0000 7fbc822d0700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc7c108d30 con 0x7fbc7c070430 2023-12-25T20:07:16.361 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:16.359+0000 7fbc7bfff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbc7c070430 0x7fbc7c108840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:07:16.361 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:16.359+0000 7fbc7bfff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbc7c070430 0x7fbc7c108840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:41096/0 (socket says 172.21.15.50:41096) 2023-12-25T20:07:16.361 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:16.359+0000 7fbc7bfff700 1 -- 172.21.15.50:0/4235042156 learned_addr learned my addr 172.21.15.50:0/4235042156 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-25T20:07:16.361 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:16.359+0000 7fbc7bfff700 1 -- 172.21.15.50:0/4235042156 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc6c00b200 con 0x7fbc7c070430 2023-12-25T20:07:16.361 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:16.359+0000 7fbc7bfff700 1 --2- 172.21.15.50:0/4235042156 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbc7c070430 0x7fbc7c108840 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7fbc6c00b1d0 tx=0x7fbc6c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:07:16.361 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:16.359+0000 7fbc78ff9700 1 -- 172.21.15.50:0/4235042156 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbc6c011df0 con 0x7fbc7c070430 2023-12-25T20:07:16.361 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:16.359+0000 7fbc78ff9700 1 -- 172.21.15.50:0/4235042156 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbc6c01b8c0 con 0x7fbc7c070430 2023-12-25T20:07:16.361 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:16.359+0000 7fbc822d0700 1 -- 172.21.15.50:0/4235042156 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc7c108f30 con 0x7fbc7c070430 2023-12-25T20:07:16.361 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:16.359+0000 7fbc78ff9700 1 -- 172.21.15.50:0/4235042156 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbc6c01a830 con 0x7fbc7c070430 2023-12-25T20:07:16.361 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:16.360+0000 7fbc822d0700 1 -- 172.21.15.50:0/4235042156 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc7c109340 con 0x7fbc7c070430 2023-12-25T20:07:16.361 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:16.360+0000 7fbc78ff9700 1 -- 172.21.15.50:0/4235042156 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 15) v1 ==== 36067+0+0 (secure 0 0 0) 0x7fbc6c00f070 con 0x7fbc7c070430 2023-12-25T20:07:16.362 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:16.361+0000 7fbc822d0700 1 -- 172.21.15.50:0/4235042156 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc7c045ad0 con 0x7fbc7c070430 2023-12-25T20:07:16.362 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:16.361+0000 7fbc78ff9700 1 --2- 172.21.15.50:0/4235042156 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fbc680341e0 0x7fbc680366a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:07:16.362 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:16.361+0000 7fbc78ff9700 1 -- 172.21.15.50:0/4235042156 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbc6c040e40 con 0x7fbc7c070430 2023-12-25T20:07:16.362 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:16.362+0000 7fbc7b7fe700 1 --2- 172.21.15.50:0/4235042156 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fbc680341e0 0x7fbc680366a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:07:16.362 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:16.362+0000 7fbc7b7fe700 1 --2- 172.21.15.50:0/4235042156 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fbc680341e0 0x7fbc680366a0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fbc700096a0 tx=0x7fbc70006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:07:16.366 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:16.366+0000 7fbc78ff9700 1 -- 172.21.15.50:0/4235042156 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbc6c0107b0 con 0x7fbc7c070430 2023-12-25T20:07:16.543 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:16.542+0000 7fbc822d0700 1 -- 172.21.15.50:0/4235042156 --> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7fbc7c10a510 con 0x7fbc680341e0 2023-12-25T20:07:16.603 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:16.603+0000 7fbc78ff9700 1 -- 172.21.15.50:0/4235042156 <== mgr.14162 v2:172.21.15.50:6800/1043424882 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7fbc7c10a510 con 0x7fbc680341e0 2023-12-25T20:07:16.603 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: Self-signed certificate created 2023-12-25T20:07:16.605 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:16.605+0000 7fbc822d0700 1 -- 172.21.15.50:0/4235042156 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fbc680341e0 msgr2=0x7fbc680366a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:07:16.605 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:16.605+0000 7fbc822d0700 1 --2- 172.21.15.50:0/4235042156 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fbc680341e0 0x7fbc680366a0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fbc700096a0 tx=0x7fbc70006b40).stop 2023-12-25T20:07:16.605 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:16.605+0000 7fbc822d0700 1 -- 172.21.15.50:0/4235042156 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbc7c070430 msgr2=0x7fbc7c108840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:07:16.605 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:16.605+0000 7fbc822d0700 1 --2- 172.21.15.50:0/4235042156 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbc7c070430 0x7fbc7c108840 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7fbc6c00b1d0 tx=0x7fbc6c006ee0).stop 2023-12-25T20:07:16.605 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:16.605+0000 7fbc822d0700 1 -- 172.21.15.50:0/4235042156 shutdown_connections 2023-12-25T20:07:16.605 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:16.605+0000 7fbc822d0700 1 --2- 172.21.15.50:0/4235042156 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fbc680341e0 0x7fbc680366a0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:07:16.605 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:16.605+0000 7fbc822d0700 1 --2- 172.21.15.50:0/4235042156 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbc7c070430 0x7fbc7c108840 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:07:16.605 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:16.605+0000 7fbc822d0700 1 -- 172.21.15.50:0/4235042156 >> 172.21.15.50:0/4235042156 conn(0x7fbc7c0f63c0 msgr2=0x7fbc7c0fc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:07:16.606 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:16.605+0000 7fbc822d0700 1 -- 172.21.15.50:0/4235042156 shutdown_connections 2023-12-25T20:07:16.606 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:16.605+0000 7fbc822d0700 1 -- 172.21.15.50:0/4235042156 wait complete. 2023-12-25T20:07:16.630 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:07:16 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:07:15.622120+0000 mgr.smithi050.nbzhun (mgr.14162) 1 : audit [DBG] from='client.14166 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2023-12-25T20:07:16.630 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:07:16 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:07:15.622432+0000 mon.smithi050 (mon.0) 106 : cluster [DBG] mgrmap e15: smithi050.nbzhun(active, since 1.05489s) 2023-12-25T20:07:16.630 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:07:16 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:07:15.629228+0000 mgr.smithi050.nbzhun (mgr.14162) 2 : audit [DBG] from='client.14166 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2023-12-25T20:07:16.630 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:07:16 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:07:15.668555+0000 mon.smithi050 (mon.0) 107 : audit [INF] from='mgr.14162 172.21.15.50:0/3178367813' entity='mgr.smithi050.nbzhun' 2023-12-25T20:07:16.630 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:07:16 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:07:16.601519+0000 mon.smithi050 (mon.0) 108 : audit [INF] from='mgr.14162 172.21.15.50:0/3178367813' entity='mgr.smithi050.nbzhun' 2023-12-25T20:07:16.630 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:07:16 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:07:16.603349+0000 mon.smithi050 (mon.0) 109 : audit [INF] from='mgr.14162 172.21.15.50:0/3178367813' entity='mgr.smithi050.nbzhun' 2023-12-25T20:07:17.078 INFO:teuthology.orchestra.run.smithi050.stderr:Creating initial admin user... 2023-12-25T20:07:17.079 INFO:teuthology.orchestra.run.smithi050.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi050 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/02c8f25c-a361-11ee-95a5-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmps7kf4jlu:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpu2g3lwcx:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpr8ie8y16:/tmp/dashboard.pw:z docker.io/ceph/ceph:v16.2.4 dashboard ac-user-create admin -i /tmp/dashboard.pw administrator --force-password --pwd-update-required 2023-12-25T20:07:17.473 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:17.472+0000 7f99be824700 1 Processor -- start 2023-12-25T20:07:17.473 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:17.473+0000 7f99be824700 1 -- start start 2023-12-25T20:07:17.473 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:17.473+0000 7f99be824700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f99b8070430 0x7f99b80fe730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-25T20:07:17.473 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:17.473+0000 7f99be824700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99b80fec20 con 0x7f99b8070430 2023-12-25T20:07:17.473 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:17.473+0000 7f99bd822700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f99b8070430 0x7f99b80fe730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:07:17.474 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:17.473+0000 7f99bd822700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f99b8070430 0x7f99b80fe730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:42838/0 (socket says 172.21.15.50:42838) 2023-12-25T20:07:17.474 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:17.473+0000 7f99bd822700 1 -- 172.21.15.50:0/3680808126 learned_addr learned my addr 172.21.15.50:0/3680808126 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-25T20:07:17.475 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:17.473+0000 7f99bd822700 1 -- 172.21.15.50:0/3680808126 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99b80fed60 con 0x7f99b8070430 2023-12-25T20:07:17.475 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:17.474+0000 7f99bd822700 1 --2- 172.21.15.50:0/3680808126 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f99b8070430 0x7f99b80fe730 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f99a800b5f0 tx=0x7f99a8006ee0).ready entity=mon.0 client_cookie=48a68df4baca774b server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:07:17.475 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:17.474+0000 7f99bc820700 1 -- 172.21.15.50:0/3680808126 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f99a800d600 con 0x7f99b8070430 2023-12-25T20:07:17.475 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:17.474+0000 7f99bc820700 1 -- 172.21.15.50:0/3680808126 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f99a800dc10 con 0x7f99b8070430 2023-12-25T20:07:17.476 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:17.474+0000 7f99bc820700 1 -- 172.21.15.50:0/3680808126 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f99a8011cd0 con 0x7f99b8070430 2023-12-25T20:07:17.476 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:17.474+0000 7f99be824700 1 -- 172.21.15.50:0/3680808126 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f99b8070430 msgr2=0x7f99b80fe730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:07:17.476 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:17.474+0000 7f99be824700 1 --2- 172.21.15.50:0/3680808126 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f99b8070430 0x7f99b80fe730 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f99a800b5f0 tx=0x7f99a8006ee0).stop 2023-12-25T20:07:17.476 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:17.474+0000 7f99be824700 1 -- 172.21.15.50:0/3680808126 shutdown_connections 2023-12-25T20:07:17.476 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:17.474+0000 7f99be824700 1 --2- 172.21.15.50:0/3680808126 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f99b8070430 0x7f99b80fe730 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:07:17.476 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:17.474+0000 7f99be824700 1 -- 172.21.15.50:0/3680808126 >> 172.21.15.50:0/3680808126 conn(0x7f99b80f6210 msgr2=0x7f99b80f8630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:07:17.476 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:17.475+0000 7f99be824700 1 -- 172.21.15.50:0/3680808126 shutdown_connections 2023-12-25T20:07:17.476 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:17.475+0000 7f99be824700 1 -- 172.21.15.50:0/3680808126 wait complete. 2023-12-25T20:07:17.476 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:17.475+0000 7f99be824700 1 Processor -- start 2023-12-25T20:07:17.476 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:17.475+0000 7f99be824700 1 -- start start 2023-12-25T20:07:17.476 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:17.475+0000 7f99be824700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f99b8070430 0x7f99b810a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:07:17.476 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:17.475+0000 7f99be824700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99b810ad80 con 0x7f99b8070430 2023-12-25T20:07:17.477 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:17.475+0000 7f99bd822700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f99b8070430 0x7f99b810a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:07:17.477 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:17.475+0000 7f99bd822700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f99b8070430 0x7f99b810a890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:42848/0 (socket says 172.21.15.50:42848) 2023-12-25T20:07:17.477 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:17.475+0000 7f99bd822700 1 -- 172.21.15.50:0/2155886796 learned_addr learned my addr 172.21.15.50:0/2155886796 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-25T20:07:17.477 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:17.476+0000 7f99bd822700 1 -- 172.21.15.50:0/2155886796 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99a800b200 con 0x7f99b8070430 2023-12-25T20:07:17.477 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:17.476+0000 7f99bd822700 1 --2- 172.21.15.50:0/2155886796 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f99b8070430 0x7f99b810a890 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f99a8006a40 tx=0x7f99a8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:07:17.477 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:17.476+0000 7f99a67fc700 1 -- 172.21.15.50:0/2155886796 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f99a8011c90 con 0x7f99b8070430 2023-12-25T20:07:17.478 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:17.476+0000 7f99a67fc700 1 -- 172.21.15.50:0/2155886796 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f99a801b8c0 con 0x7f99b8070430 2023-12-25T20:07:17.478 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:17.476+0000 7f99be824700 1 -- 172.21.15.50:0/2155886796 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99b810af80 con 0x7f99b8070430 2023-12-25T20:07:17.478 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:17.476+0000 7f99a67fc700 1 -- 172.21.15.50:0/2155886796 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f99a801a8c0 con 0x7f99b8070430 2023-12-25T20:07:17.478 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:17.476+0000 7f99be824700 1 -- 172.21.15.50:0/2155886796 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99b810b390 con 0x7f99b8070430 2023-12-25T20:07:17.478 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:17.477+0000 7f99a67fc700 1 -- 172.21.15.50:0/2155886796 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 16) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f99a800f070 con 0x7f99b8070430 2023-12-25T20:07:17.478 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:17.477+0000 7f99a67fc700 1 --2- 172.21.15.50:0/2155886796 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f99a002feb0 0x7f99a0032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:07:17.478 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:17.477+0000 7f99a67fc700 1 -- 172.21.15.50:0/2155886796 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f99a8040af0 con 0x7f99b8070430 2023-12-25T20:07:17.478 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:17.477+0000 7f99bd021700 1 --2- 172.21.15.50:0/2155886796 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f99a002feb0 0x7f99a0032370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:07:17.478 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:17.477+0000 7f99be824700 1 -- 172.21.15.50:0/2155886796 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99b8059100 con 0x7f99b8070430 2023-12-25T20:07:17.479 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:17.477+0000 7f99bd021700 1 --2- 172.21.15.50:0/2155886796 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f99a002feb0 0x7f99a0032370 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f99b40096a0 tx=0x7f99b4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:07:17.481 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:17.481+0000 7f99a67fc700 1 -- 172.21.15.50:0/2155886796 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f99a8016070 con 0x7f99b8070430 2023-12-25T20:07:17.657 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:17.657+0000 7f99be824700 1 -- 172.21.15.50:0/2155886796 --> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f99b8045ad0 con 0x7f99a002feb0 2023-12-25T20:07:17.902 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:17.902+0000 7f99a67fc700 1 -- 172.21.15.50:0/2155886796 <== mgr.14162 v2:172.21.15.50:6800/1043424882 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f99b8045ad0 con 0x7f99a002feb0 2023-12-25T20:07:17.902 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$sgp0Q6P9mmvqlr/i9Dod2.RRo4w4fKrviSaQmGUBndHvLu3wKkbBu", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1703534837, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2023-12-25T20:07:17.904 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:17.903+0000 7f99be824700 1 -- 172.21.15.50:0/2155886796 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f99a002feb0 msgr2=0x7f99a0032370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:07:17.904 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:17.903+0000 7f99be824700 1 --2- 172.21.15.50:0/2155886796 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f99a002feb0 0x7f99a0032370 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f99b40096a0 tx=0x7f99b4006b40).stop 2023-12-25T20:07:17.904 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:17.904+0000 7f99be824700 1 -- 172.21.15.50:0/2155886796 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f99b8070430 msgr2=0x7f99b810a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:07:17.904 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:17.904+0000 7f99be824700 1 --2- 172.21.15.50:0/2155886796 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f99b8070430 0x7f99b810a890 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f99a8006a40 tx=0x7f99a8006ee0).stop 2023-12-25T20:07:17.904 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:17.904+0000 7f99be824700 1 -- 172.21.15.50:0/2155886796 shutdown_connections 2023-12-25T20:07:17.904 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:17.904+0000 7f99be824700 1 --2- 172.21.15.50:0/2155886796 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f99a002feb0 0x7f99a0032370 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:07:17.904 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:17.904+0000 7f99be824700 1 --2- 172.21.15.50:0/2155886796 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f99b8070430 0x7f99b810a890 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:07:17.904 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:17.904+0000 7f99be824700 1 -- 172.21.15.50:0/2155886796 >> 172.21.15.50:0/2155886796 conn(0x7f99b80f6210 msgr2=0x7f99b80f7d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:07:17.904 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:17.904+0000 7f99be824700 1 -- 172.21.15.50:0/2155886796 shutdown_connections 2023-12-25T20:07:17.905 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:17.904+0000 7f99be824700 1 -- 172.21.15.50:0/2155886796 wait complete. 2023-12-25T20:07:18.068 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:07:17 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:07:16.543645+0000 mgr.smithi050.nbzhun (mgr.14162) 3 : audit [DBG] from='client.14172 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2023-12-25T20:07:18.068 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:07:17 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:07:16.745997+0000 mon.smithi050 (mon.0) 110 : cluster [DBG] mgrmap e16: smithi050.nbzhun(active, since 2s) 2023-12-25T20:07:18.253 INFO:teuthology.orchestra.run.smithi050.stderr:Fetching dashboard port number... 2023-12-25T20:07:18.253 INFO:teuthology.orchestra.run.smithi050.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi050 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/02c8f25c-a361-11ee-95a5-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmps7kf4jlu:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpu2g3lwcx:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config get mgr mgr/dashboard/ssl_server_port 2023-12-25T20:07:18.646 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:18.645+0000 7f2c7d752700 1 Processor -- start 2023-12-25T20:07:18.646 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:18.645+0000 7f2c7d752700 1 -- start start 2023-12-25T20:07:18.646 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:18.645+0000 7f2c7d752700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2c78070430 0x7f2c780fe730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-25T20:07:18.646 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:18.645+0000 7f2c7d752700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c780fec20 con 0x7f2c78070430 2023-12-25T20:07:18.646 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:18.645+0000 7f2c77fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2c78070430 0x7f2c780fe730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:07:18.646 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:18.645+0000 7f2c77fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2c78070430 0x7f2c780fe730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:42852/0 (socket says 172.21.15.50:42852) 2023-12-25T20:07:18.646 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:18.645+0000 7f2c77fff700 1 -- 172.21.15.50:0/503999962 learned_addr learned my addr 172.21.15.50:0/503999962 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-25T20:07:18.647 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:18.645+0000 7f2c77fff700 1 -- 172.21.15.50:0/503999962 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c780fed60 con 0x7f2c78070430 2023-12-25T20:07:18.647 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:18.646+0000 7f2c77fff700 1 --2- 172.21.15.50:0/503999962 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2c78070430 0x7f2c780fe730 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f2c6800b5f0 tx=0x7f2c68006ee0).ready entity=mon.0 client_cookie=de3c64e1a51843c7 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:07:18.647 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:18.646+0000 7f2c76ffd700 1 -- 172.21.15.50:0/503999962 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2c6800d600 con 0x7f2c78070430 2023-12-25T20:07:18.647 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:18.646+0000 7f2c76ffd700 1 -- 172.21.15.50:0/503999962 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2c6800dc10 con 0x7f2c78070430 2023-12-25T20:07:18.650 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:18.646+0000 7f2c76ffd700 1 -- 172.21.15.50:0/503999962 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2c68011cd0 con 0x7f2c78070430 2023-12-25T20:07:18.650 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:18.647+0000 7f2c7d752700 1 -- 172.21.15.50:0/503999962 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2c78070430 msgr2=0x7f2c780fe730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:07:18.650 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:18.647+0000 7f2c7d752700 1 --2- 172.21.15.50:0/503999962 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2c78070430 0x7f2c780fe730 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f2c6800b5f0 tx=0x7f2c68006ee0).stop 2023-12-25T20:07:18.650 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:18.647+0000 7f2c7d752700 1 -- 172.21.15.50:0/503999962 shutdown_connections 2023-12-25T20:07:18.650 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:18.647+0000 7f2c7d752700 1 --2- 172.21.15.50:0/503999962 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2c78070430 0x7f2c780fe730 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:07:18.650 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:18.647+0000 7f2c7d752700 1 -- 172.21.15.50:0/503999962 >> 172.21.15.50:0/503999962 conn(0x7f2c780f6210 msgr2=0x7f2c780f8630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:07:18.650 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:18.647+0000 7f2c7d752700 1 -- 172.21.15.50:0/503999962 shutdown_connections 2023-12-25T20:07:18.650 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:18.647+0000 7f2c7d752700 1 -- 172.21.15.50:0/503999962 wait complete. 2023-12-25T20:07:18.650 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:18.647+0000 7f2c7d752700 1 Processor -- start 2023-12-25T20:07:18.650 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:18.648+0000 7f2c7d752700 1 -- start start 2023-12-25T20:07:18.650 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:18.648+0000 7f2c7d752700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2c78070430 0x7f2c7810a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:07:18.650 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:18.648+0000 7f2c7d752700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c7810adb0 con 0x7f2c78070430 2023-12-25T20:07:18.651 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:18.648+0000 7f2c77fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2c78070430 0x7f2c7810a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:07:18.651 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:18.648+0000 7f2c77fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2c78070430 0x7f2c7810a8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:42854/0 (socket says 172.21.15.50:42854) 2023-12-25T20:07:18.651 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:18.648+0000 7f2c77fff700 1 -- 172.21.15.50:0/1874423809 learned_addr learned my addr 172.21.15.50:0/1874423809 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-25T20:07:18.651 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:18.648+0000 7f2c77fff700 1 -- 172.21.15.50:0/1874423809 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c6800b200 con 0x7f2c78070430 2023-12-25T20:07:18.651 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:18.648+0000 7f2c77fff700 1 --2- 172.21.15.50:0/1874423809 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2c78070430 0x7f2c7810a8c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f2c68006a40 tx=0x7f2c68006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:07:18.651 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:18.649+0000 7f2c74ff9700 1 -- 172.21.15.50:0/1874423809 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2c68011c90 con 0x7f2c78070430 2023-12-25T20:07:18.651 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:18.649+0000 7f2c74ff9700 1 -- 172.21.15.50:0/1874423809 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2c6801b8c0 con 0x7f2c78070430 2023-12-25T20:07:18.651 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:18.649+0000 7f2c7d752700 1 -- 172.21.15.50:0/1874423809 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c7810afb0 con 0x7f2c78070430 2023-12-25T20:07:18.651 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:18.649+0000 7f2c74ff9700 1 -- 172.21.15.50:0/1874423809 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2c6801a8c0 con 0x7f2c78070430 2023-12-25T20:07:18.651 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:18.649+0000 7f2c7d752700 1 -- 172.21.15.50:0/1874423809 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c7810b3c0 con 0x7f2c78070430 2023-12-25T20:07:18.651 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:18.649+0000 7f2c74ff9700 1 -- 172.21.15.50:0/1874423809 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 16) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f2c6800f070 con 0x7f2c78070430 2023-12-25T20:07:18.651 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:18.650+0000 7f2c74ff9700 1 --2- 172.21.15.50:0/1874423809 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f2c6402ff00 0x7f2c640323c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:07:18.651 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:18.650+0000 7f2c74ff9700 1 -- 172.21.15.50:0/1874423809 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2c68040ba0 con 0x7f2c78070430 2023-12-25T20:07:18.651 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:18.650+0000 7f2c777fe700 1 --2- 172.21.15.50:0/1874423809 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f2c6402ff00 0x7f2c640323c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:07:18.652 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:18.650+0000 7f2c7d752700 1 -- 172.21.15.50:0/1874423809 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c78059100 con 0x7f2c78070430 2023-12-25T20:07:18.652 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:18.650+0000 7f2c777fe700 1 --2- 172.21.15.50:0/1874423809 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f2c6402ff00 0x7f2c640323c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f2c6c0096a0 tx=0x7f2c6c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:07:18.653 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:18.654+0000 7f2c74ff9700 1 -- 172.21.15.50:0/1874423809 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2c68016070 con 0x7f2c78070430 2023-12-25T20:07:18.814 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:18.814+0000 7f2c7d752700 1 -- 172.21.15.50:0/1874423809 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f2c780ffa50 con 0x7f2c78070430 2023-12-25T20:07:18.814 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:18.814+0000 7f2c74ff9700 1 -- 172.21.15.50:0/1874423809 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v10) v1 ==== 112+0+5 (secure 0 0 0) 0x7f2c6802f0b0 con 0x7f2c78070430 2023-12-25T20:07:18.814 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 8443 2023-12-25T20:07:18.816 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:18.816+0000 7f2c7d752700 1 -- 172.21.15.50:0/1874423809 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f2c6402ff00 msgr2=0x7f2c640323c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:07:18.817 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:18.816+0000 7f2c7d752700 1 --2- 172.21.15.50:0/1874423809 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f2c6402ff00 0x7f2c640323c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f2c6c0096a0 tx=0x7f2c6c006b40).stop 2023-12-25T20:07:18.817 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:18.816+0000 7f2c7d752700 1 -- 172.21.15.50:0/1874423809 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2c78070430 msgr2=0x7f2c7810a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:07:18.817 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:18.816+0000 7f2c7d752700 1 --2- 172.21.15.50:0/1874423809 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2c78070430 0x7f2c7810a8c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f2c68006a40 tx=0x7f2c68006ee0).stop 2023-12-25T20:07:18.817 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:18.816+0000 7f2c7d752700 1 -- 172.21.15.50:0/1874423809 shutdown_connections 2023-12-25T20:07:18.817 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:18.816+0000 7f2c7d752700 1 --2- 172.21.15.50:0/1874423809 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f2c6402ff00 0x7f2c640323c0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:07:18.817 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:18.816+0000 7f2c7d752700 1 --2- 172.21.15.50:0/1874423809 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2c78070430 0x7f2c7810a8c0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:07:18.817 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:18.816+0000 7f2c7d752700 1 -- 172.21.15.50:0/1874423809 >> 172.21.15.50:0/1874423809 conn(0x7f2c780f6210 msgr2=0x7f2c780f7d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:07:18.817 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:18.816+0000 7f2c7d752700 1 -- 172.21.15.50:0/1874423809 shutdown_connections 2023-12-25T20:07:18.817 INFO:teuthology.orchestra.run.smithi050.stderr:/usr/bin/ceph: 2023-12-25T20:07:18.816+0000 7f2c7d752700 1 -- 172.21.15.50:0/1874423809 wait complete. 2023-12-25T20:07:19.147 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:07:18 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:07:17.658113+0000 mgr.smithi050.nbzhun (mgr.14162) 4 : audit [DBG] from='client.14174 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2023-12-25T20:07:19.147 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:07:18 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:07:17.902280+0000 mon.smithi050 (mon.0) 111 : audit [INF] from='mgr.14162 172.21.15.50:0/3178367813' entity='mgr.smithi050.nbzhun' 2023-12-25T20:07:19.147 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:07:18 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:07:18.814977+0000 mon.smithi050 (mon.0) 112 : audit [DBG] from='client.? 172.21.15.50:0/1874423809' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2023-12-25T20:07:19.162 INFO:teuthology.orchestra.run.smithi050.stderr:Running command: systemctl is-enabled firewalld.service 2023-12-25T20:07:19.170 INFO:teuthology.orchestra.run.smithi050.stderr:systemctl: disabled 2023-12-25T20:07:19.171 INFO:teuthology.orchestra.run.smithi050.stderr:Running command: systemctl is-active firewalld.service 2023-12-25T20:07:19.177 INFO:teuthology.orchestra.run.smithi050.stderr:systemctl: inactive 2023-12-25T20:07:19.178 INFO:teuthology.orchestra.run.smithi050.stderr:firewalld.service is not enabled 2023-12-25T20:07:19.178 INFO:teuthology.orchestra.run.smithi050.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2023-12-25T20:07:19.181 INFO:teuthology.orchestra.run.smithi050.stderr:Ceph Dashboard is now available at: 2023-12-25T20:07:19.182 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-12-25T20:07:19.182 INFO:teuthology.orchestra.run.smithi050.stderr: URL: https://smithi050.front.sepia.ceph.com:8443/ 2023-12-25T20:07:19.182 INFO:teuthology.orchestra.run.smithi050.stderr: User: admin 2023-12-25T20:07:19.182 INFO:teuthology.orchestra.run.smithi050.stderr: Password: apflwijhs1 2023-12-25T20:07:19.182 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-12-25T20:07:19.182 INFO:teuthology.orchestra.run.smithi050.stderr:You can access the Ceph CLI with: 2023-12-25T20:07:19.182 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-12-25T20:07:19.182 INFO:teuthology.orchestra.run.smithi050.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2023-12-25T20:07:19.182 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-12-25T20:07:19.182 INFO:teuthology.orchestra.run.smithi050.stderr:Please consider enabling telemetry to help improve Ceph: 2023-12-25T20:07:19.182 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-12-25T20:07:19.183 INFO:teuthology.orchestra.run.smithi050.stderr: ceph telemetry on 2023-12-25T20:07:19.183 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-12-25T20:07:19.183 INFO:teuthology.orchestra.run.smithi050.stderr:For more information see: 2023-12-25T20:07:19.183 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-12-25T20:07:19.183 INFO:teuthology.orchestra.run.smithi050.stderr: https://docs.ceph.com/docs/pacific/mgr/telemetry/ 2023-12-25T20:07:19.183 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-12-25T20:07:19.183 INFO:teuthology.orchestra.run.smithi050.stderr:Bootstrap complete. 2023-12-25T20:07:19.183 INFO:teuthology.orchestra.run.smithi050.stderr:Releasing lock 140098145065448 on /run/cephadm/02c8f25c-a361-11ee-95a5-87774f69a715.lock 2023-12-25T20:07:19.183 INFO:teuthology.orchestra.run.smithi050.stderr:Lock 140098145065448 released on /run/cephadm/02c8f25c-a361-11ee-95a5-87774f69a715.lock 2023-12-25T20:07:19.219 INFO:tasks.cephadm:Fetching config... 2023-12-25T20:07:19.219 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2023-12-25T20:07:19.219 DEBUG:teuthology.orchestra.run.smithi050:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2023-12-25T20:07:19.277 INFO:tasks.cephadm:Fetching client.admin keyring... 2023-12-25T20:07:19.278 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2023-12-25T20:07:19.278 DEBUG:teuthology.orchestra.run.smithi050:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2023-12-25T20:07:19.337 INFO:tasks.cephadm:Fetching mon keyring... 2023-12-25T20:07:19.338 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2023-12-25T20:07:19.338 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/var/lib/ceph/02c8f25c-a361-11ee-95a5-87774f69a715/mon.smithi050/keyring of=/dev/stdout 2023-12-25T20:07:19.412 INFO:tasks.cephadm:Fetching pub ssh key... 2023-12-25T20:07:19.413 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2023-12-25T20:07:19.413 DEBUG:teuthology.orchestra.run.smithi050:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2023-12-25T20:07:19.471 INFO:tasks.cephadm:Installing pub ssh key for root users... 2023-12-25T20:07:19.471 DEBUG:teuthology.orchestra.run.smithi050:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDBcLPR/YSyydiXtXm47+k2ciQm4QF77G2lYmwHiS+H61PQs1KnHmbDj55yK1iabJYJLy0AfqaAEgXGKJXCuA4Ma5pjRu9hD3k7zotzb3matENxJJ+k0ubYvVij7DWjUTJborwswJy3/hySM0S6mPxul05CI7dThlnxIRonWJCvriSyiQm4o+OMUWB3M09lyEbVIw3oM2PtYCNGcUw8CWPPtawiSZKaUvE0K/nEr1jBLOOYCHEx9grHI9GVJxyMJc6rVzmNm4Ls9QdQWTN73CJlVFDs37pZF6j+i9w0cbvtDFf+pCiFQOkPyWb0tvbXqbKssU7ajwOXa3dgOtDX/2ZafblngS1XnrduJtJXDE0y7mvyya8JPjO/Sb934kK73wtuwUZ8JrtrHXzy81Gt/9Eb0jBMs72VPlPCsKhK3Jzobw1G11iftwoS9kIMwO9xEBG4lCEYqCpDuPaiHoVAUxpRGygXOYSAPWZF4rJ1yf/PaGTi2UrApG5mzenjTXAiifk= ceph-02c8f25c-a361-11ee-95a5-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-12-25T20:07:19.548 INFO:teuthology.orchestra.run.smithi050.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDBcLPR/YSyydiXtXm47+k2ciQm4QF77G2lYmwHiS+H61PQs1KnHmbDj55yK1iabJYJLy0AfqaAEgXGKJXCuA4Ma5pjRu9hD3k7zotzb3matENxJJ+k0ubYvVij7DWjUTJborwswJy3/hySM0S6mPxul05CI7dThlnxIRonWJCvriSyiQm4o+OMUWB3M09lyEbVIw3oM2PtYCNGcUw8CWPPtawiSZKaUvE0K/nEr1jBLOOYCHEx9grHI9GVJxyMJc6rVzmNm4Ls9QdQWTN73CJlVFDs37pZF6j+i9w0cbvtDFf+pCiFQOkPyWb0tvbXqbKssU7ajwOXa3dgOtDX/2ZafblngS1XnrduJtJXDE0y7mvyya8JPjO/Sb934kK73wtuwUZ8JrtrHXzy81Gt/9Eb0jBMs72VPlPCsKhK3Jzobw1G11iftwoS9kIMwO9xEBG4lCEYqCpDuPaiHoVAUxpRGygXOYSAPWZF4rJ1yf/PaGTi2UrApG5mzenjTXAiifk= ceph-02c8f25c-a361-11ee-95a5-87774f69a715 2023-12-25T20:07:19.560 DEBUG:teuthology.orchestra.run.smithi160:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDBcLPR/YSyydiXtXm47+k2ciQm4QF77G2lYmwHiS+H61PQs1KnHmbDj55yK1iabJYJLy0AfqaAEgXGKJXCuA4Ma5pjRu9hD3k7zotzb3matENxJJ+k0ubYvVij7DWjUTJborwswJy3/hySM0S6mPxul05CI7dThlnxIRonWJCvriSyiQm4o+OMUWB3M09lyEbVIw3oM2PtYCNGcUw8CWPPtawiSZKaUvE0K/nEr1jBLOOYCHEx9grHI9GVJxyMJc6rVzmNm4Ls9QdQWTN73CJlVFDs37pZF6j+i9w0cbvtDFf+pCiFQOkPyWb0tvbXqbKssU7ajwOXa3dgOtDX/2ZafblngS1XnrduJtJXDE0y7mvyya8JPjO/Sb934kK73wtuwUZ8JrtrHXzy81Gt/9Eb0jBMs72VPlPCsKhK3Jzobw1G11iftwoS9kIMwO9xEBG4lCEYqCpDuPaiHoVAUxpRGygXOYSAPWZF4rJ1yf/PaGTi2UrApG5mzenjTXAiifk= ceph-02c8f25c-a361-11ee-95a5-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-12-25T20:07:19.605 INFO:teuthology.orchestra.run.smithi160.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDBcLPR/YSyydiXtXm47+k2ciQm4QF77G2lYmwHiS+H61PQs1KnHmbDj55yK1iabJYJLy0AfqaAEgXGKJXCuA4Ma5pjRu9hD3k7zotzb3matENxJJ+k0ubYvVij7DWjUTJborwswJy3/hySM0S6mPxul05CI7dThlnxIRonWJCvriSyiQm4o+OMUWB3M09lyEbVIw3oM2PtYCNGcUw8CWPPtawiSZKaUvE0K/nEr1jBLOOYCHEx9grHI9GVJxyMJc6rVzmNm4Ls9QdQWTN73CJlVFDs37pZF6j+i9w0cbvtDFf+pCiFQOkPyWb0tvbXqbKssU7ajwOXa3dgOtDX/2ZafblngS1XnrduJtJXDE0y7mvyya8JPjO/Sb934kK73wtuwUZ8JrtrHXzy81Gt/9Eb0jBMs72VPlPCsKhK3Jzobw1G11iftwoS9kIMwO9xEBG4lCEYqCpDuPaiHoVAUxpRGygXOYSAPWZF4rJ1yf/PaGTi2UrApG5mzenjTXAiifk= ceph-02c8f25c-a361-11ee-95a5-87774f69a715 2023-12-25T20:07:19.618 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2023-12-25T20:07:21.968 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:07:21 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:07:20.656762+0000 mon.smithi050 (mon.0) 113 : cluster [DBG] mgrmap e17: smithi050.nbzhun(active, since 6s) 2023-12-25T20:07:22.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-25T20:07:22.164+0000 7f466759e700 1 -- 172.21.15.50:0/3367887934 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f466806b920 msgr2=0x7f4668069ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:07:22.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-25T20:07:22.164+0000 7f466759e700 1 --2- 172.21.15.50:0/3367887934 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f466806b920 0x7f4668069ec0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f465c009fb0 tx=0x7f465c009b30).stop 2023-12-25T20:07:22.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-25T20:07:22.165+0000 7f466759e700 1 -- 172.21.15.50:0/3367887934 shutdown_connections 2023-12-25T20:07:22.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-25T20:07:22.165+0000 7f466759e700 1 --2- 172.21.15.50:0/3367887934 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f466806b920 0x7f4668069ec0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:07:22.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-25T20:07:22.165+0000 7f466759e700 1 -- 172.21.15.50:0/3367887934 >> 172.21.15.50:0/3367887934 conn(0x7f46680f1aa0 msgr2=0x7f46680f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:07:22.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-25T20:07:22.165+0000 7f466759e700 1 -- 172.21.15.50:0/3367887934 shutdown_connections 2023-12-25T20:07:22.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-25T20:07:22.165+0000 7f466759e700 1 -- 172.21.15.50:0/3367887934 wait complete. 2023-12-25T20:07:22.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-25T20:07:22.165+0000 7f466759e700 1 Processor -- start 2023-12-25T20:07:22.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-25T20:07:22.165+0000 7f466759e700 1 -- start start 2023-12-25T20:07:22.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-25T20:07:22.166+0000 7f466759e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f466806b920 0x7f466810cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:07:22.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-25T20:07:22.166+0000 7f466759e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f466810cfd0 con 0x7f466806b920 2023-12-25T20:07:22.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-25T20:07:22.166+0000 7f466659c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f466806b920 0x7f466810cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:07:22.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-25T20:07:22.166+0000 7f466659c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f466806b920 0x7f466810cae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:42878/0 (socket says 172.21.15.50:42878) 2023-12-25T20:07:22.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-25T20:07:22.166+0000 7f466659c700 1 -- 172.21.15.50:0/2619970510 learned_addr learned my addr 172.21.15.50:0/2619970510 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-25T20:07:22.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-25T20:07:22.166+0000 7f466659c700 1 -- 172.21.15.50:0/2619970510 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f465c005040 con 0x7f466806b920 2023-12-25T20:07:22.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-25T20:07:22.166+0000 7f466659c700 1 --2- 172.21.15.50:0/2619970510 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f466806b920 0x7f466810cae0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f465c013510 tx=0x7f465c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:07:22.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-25T20:07:22.166+0000 7f4656ffd700 1 -- 172.21.15.50:0/2619970510 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f465c009320 con 0x7f466806b920 2023-12-25T20:07:22.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-25T20:07:22.166+0000 7f4656ffd700 1 -- 172.21.15.50:0/2619970510 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f465c013f00 con 0x7f466806b920 2023-12-25T20:07:22.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-25T20:07:22.166+0000 7f466759e700 1 -- 172.21.15.50:0/2619970510 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f466810d1d0 con 0x7f466806b920 2023-12-25T20:07:22.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-25T20:07:22.166+0000 7f4656ffd700 1 -- 172.21.15.50:0/2619970510 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f465c014d90 con 0x7f466806b920 2023-12-25T20:07:22.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-25T20:07:22.166+0000 7f466759e700 1 -- 172.21.15.50:0/2619970510 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f466810d660 con 0x7f466806b920 2023-12-25T20:07:22.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-25T20:07:22.167+0000 7f4656ffd700 1 -- 172.21.15.50:0/2619970510 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f465c0263a0 con 0x7f466806b920 2023-12-25T20:07:22.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-25T20:07:22.167+0000 7f4656ffd700 1 --2- 172.21.15.50:0/2619970510 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f465002ff60 0x7f4650032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:07:22.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-25T20:07:22.167+0000 7f466759e700 1 -- 172.21.15.50:0/2619970510 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4668045ad0 con 0x7f466806b920 2023-12-25T20:07:22.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-25T20:07:22.167+0000 7f4656ffd700 1 -- 172.21.15.50:0/2619970510 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f465c040ff0 con 0x7f466806b920 2023-12-25T20:07:22.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-25T20:07:22.168+0000 7f4665d9b700 1 --2- 172.21.15.50:0/2619970510 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f465002ff60 0x7f4650032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:07:22.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-25T20:07:22.168+0000 7f4665d9b700 1 --2- 172.21.15.50:0/2619970510 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f465002ff60 0x7f4650032420 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f46580096a0 tx=0x7f4658006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:07:22.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-25T20:07:22.173+0000 7f4656ffd700 1 -- 172.21.15.50:0/2619970510 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f465c06d2b0 con 0x7f466806b920 2023-12-25T20:07:22.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-25T20:07:22.336+0000 7f466759e700 1 -- 172.21.15.50:0/2619970510 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f46680f9ac0 con 0x7f466806b920 2023-12-25T20:07:22.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-25T20:07:22.339+0000 7f4656ffd700 1 -- 172.21.15.50:0/2619970510 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v11)=0 v11) v1 ==== 125+0+0 (secure 0 0 0) 0x7f465c012510 con 0x7f466806b920 2023-12-25T20:07:22.341 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-25T20:07:22.340+0000 7f466759e700 1 -- 172.21.15.50:0/2619970510 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f465002ff60 msgr2=0x7f4650032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:07:22.341 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-25T20:07:22.340+0000 7f466759e700 1 --2- 172.21.15.50:0/2619970510 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f465002ff60 0x7f4650032420 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f46580096a0 tx=0x7f4658006b40).stop 2023-12-25T20:07:22.341 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-25T20:07:22.340+0000 7f466759e700 1 -- 172.21.15.50:0/2619970510 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f466806b920 msgr2=0x7f466810cae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:07:22.341 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-25T20:07:22.340+0000 7f466759e700 1 --2- 172.21.15.50:0/2619970510 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f466806b920 0x7f466810cae0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f465c013510 tx=0x7f465c00b040).stop 2023-12-25T20:07:22.341 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-25T20:07:22.340+0000 7f466759e700 1 -- 172.21.15.50:0/2619970510 shutdown_connections 2023-12-25T20:07:22.341 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-25T20:07:22.340+0000 7f466759e700 1 --2- 172.21.15.50:0/2619970510 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f465002ff60 0x7f4650032420 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:07:22.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-25T20:07:22.340+0000 7f466759e700 1 --2- 172.21.15.50:0/2619970510 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f466806b920 0x7f466810cae0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:07:22.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-25T20:07:22.340+0000 7f466759e700 1 -- 172.21.15.50:0/2619970510 >> 172.21.15.50:0/2619970510 conn(0x7f46680f1aa0 msgr2=0x7f46680f2560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:07:22.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-25T20:07:22.341+0000 7f466759e700 1 -- 172.21.15.50:0/2619970510 shutdown_connections 2023-12-25T20:07:22.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-25T20:07:22.341+0000 7f466759e700 1 -- 172.21.15.50:0/2619970510 wait complete. 2023-12-25T20:07:22.814 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi160 2023-12-25T20:07:22.814 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2023-12-25T20:07:22.814 DEBUG:teuthology.orchestra.run.smithi160:> dd of=/etc/ceph/ceph.conf 2023-12-25T20:07:22.836 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2023-12-25T20:07:22.836 DEBUG:teuthology.orchestra.run.smithi160:> dd of=/etc/ceph/ceph.client.admin.keyring 2023-12-25T20:07:22.898 INFO:tasks.cephadm:Adding host smithi160 to orchestrator... 2023-12-25T20:07:22.898 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph orch host add smithi160 2023-12-25T20:07:23.469 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:07:23 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:07:22.340208+0000 mon.smithi050 (mon.0) 114 : audit [INF] from='client.? 172.21.15.50:0/2619970510' entity='client.admin' 2023-12-25T20:07:25.969 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:07:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:07:24.598526+0000 mon.smithi050 (mon.0) 115 : audit [INF] from='mgr.14162 172.21.15.50:0/3178367813' entity='mgr.smithi050.nbzhun' 2023-12-25T20:07:25.969 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:07:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:07:24.869858+0000 mon.smithi050 (mon.0) 116 : audit [INF] from='mgr.14162 172.21.15.50:0/3178367813' entity='mgr.smithi050.nbzhun' 2023-12-25T20:07:25.969 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:07:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cephadm 2023-12-25T20:07:24.876381+0000 mgr.smithi050.nbzhun (mgr.14162) 5 : cephadm [INF] Deploying daemon alertmanager.smithi050 on smithi050 2023-12-25T20:07:35.969 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:07:35 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:07:34.623863+0000 mgr.smithi050.nbzhun (mgr.14162) 6 : cluster [DBG] pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:07:38.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:07:37 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:07:36.624159+0000 mgr.smithi050.nbzhun (mgr.14162) 7 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:07:40.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:07:39 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:07:38.624481+0000 mgr.smithi050.nbzhun (mgr.14162) 8 : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:07:42.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:07:41 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:07:40.624816+0000 mgr.smithi050.nbzhun (mgr.14162) 9 : cluster [DBG] pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:07:44.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:07:43 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:07:42.625071+0000 mgr.smithi050.nbzhun (mgr.14162) 10 : cluster [DBG] pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:07:45.240 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:45.236+0000 7fcf96236700 1 -- 172.21.15.160:0/1644687681 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcf900f6510 msgr2=0x7fcf900f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:07:45.240 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:45.236+0000 7fcf96236700 1 --2- 172.21.15.160:0/1644687681 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcf900f6510 0x7fcf900f6930 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7fcf84009fb0 tx=0x7fcf84009b30).stop 2023-12-25T20:07:45.240 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:45.236+0000 7fcf96236700 1 -- 172.21.15.160:0/1644687681 shutdown_connections 2023-12-25T20:07:45.240 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:45.236+0000 7fcf96236700 1 --2- 172.21.15.160:0/1644687681 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcf900f6510 0x7fcf900f6930 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:07:45.240 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:45.236+0000 7fcf96236700 1 -- 172.21.15.160:0/1644687681 >> 172.21.15.160:0/1644687681 conn(0x7fcf900f1c70 msgr2=0x7fcf900f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:07:45.240 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:45.236+0000 7fcf96236700 1 -- 172.21.15.160:0/1644687681 shutdown_connections 2023-12-25T20:07:45.241 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:45.236+0000 7fcf96236700 1 -- 172.21.15.160:0/1644687681 wait complete. 2023-12-25T20:07:45.241 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:45.237+0000 7fcf96236700 1 Processor -- start 2023-12-25T20:07:45.241 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:45.237+0000 7fcf96236700 1 -- start start 2023-12-25T20:07:45.241 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:45.237+0000 7fcf96236700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcf900f6510 0x7fcf901088f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:07:45.241 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:45.237+0000 7fcf96236700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf90108de0 con 0x7fcf900f6510 2023-12-25T20:07:45.241 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:45.238+0000 7fcf8f7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcf900f6510 0x7fcf901088f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:07:45.241 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:45.238+0000 7fcf8f7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcf900f6510 0x7fcf901088f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:59878/0 (socket says 172.21.15.160:59878) 2023-12-25T20:07:45.241 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:45.238+0000 7fcf8f7fe700 1 -- 172.21.15.160:0/3818499076 learned_addr learned my addr 172.21.15.160:0/3818499076 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:07:45.242 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:45.238+0000 7fcf8f7fe700 1 -- 172.21.15.160:0/3818499076 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf84005040 con 0x7fcf900f6510 2023-12-25T20:07:45.242 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:45.238+0000 7fcf8f7fe700 1 --2- 172.21.15.160:0/3818499076 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcf900f6510 0x7fcf901088f0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7fcf84013510 tx=0x7fcf8400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:07:45.242 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:45.238+0000 7fcf77fff700 1 -- 172.21.15.160:0/3818499076 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcf84009320 con 0x7fcf900f6510 2023-12-25T20:07:45.242 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:45.238+0000 7fcf96236700 1 -- 172.21.15.160:0/3818499076 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf90108fe0 con 0x7fcf900f6510 2023-12-25T20:07:45.242 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:45.238+0000 7fcf77fff700 1 -- 172.21.15.160:0/3818499076 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcf84013f00 con 0x7fcf900f6510 2023-12-25T20:07:45.242 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:45.238+0000 7fcf77fff700 1 -- 172.21.15.160:0/3818499076 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcf84014d90 con 0x7fcf900f6510 2023-12-25T20:07:45.242 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:45.238+0000 7fcf96236700 1 -- 172.21.15.160:0/3818499076 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf90109470 con 0x7fcf900f6510 2023-12-25T20:07:45.242 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:45.239+0000 7fcf77fff700 1 -- 172.21.15.160:0/3818499076 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcf84014420 con 0x7fcf900f6510 2023-12-25T20:07:45.243 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:45.239+0000 7fcf96236700 1 -- 172.21.15.160:0/3818499076 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf90045ad0 con 0x7fcf900f6510 2023-12-25T20:07:45.243 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:45.239+0000 7fcf77fff700 1 --2- 172.21.15.160:0/3818499076 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fcf7802ffb0 0x7fcf78032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:07:45.243 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:45.239+0000 7fcf77fff700 1 -- 172.21.15.160:0/3818499076 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcf840410c0 con 0x7fcf900f6510 2023-12-25T20:07:45.243 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:45.240+0000 7fcf8effd700 1 --2- 172.21.15.160:0/3818499076 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fcf7802ffb0 0x7fcf78032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:07:45.243 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:45.240+0000 7fcf8effd700 1 --2- 172.21.15.160:0/3818499076 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fcf7802ffb0 0x7fcf78032470 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fcf800096a0 tx=0x7fcf80006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:07:45.244 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:45.244+0000 7fcf77fff700 1 -- 172.21.15.160:0/3818499076 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcf840095f0 con 0x7fcf900f6510 2023-12-25T20:07:45.413 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:45.413+0000 7fcf96236700 1 -- 172.21.15.160:0/3818499076 --> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi160", "target": ["mon-mgr", ""]}) v1 -- 0x7fcf900f2de0 con 0x7fcf7802ffb0 2023-12-25T20:07:46.022 INFO:teuthology.orchestra.run.smithi160.stdout:Added host 'smithi160' 2023-12-25T20:07:46.022 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:46.020+0000 7fcf77fff700 1 -- 172.21.15.160:0/3818499076 <== mgr.14162 v2:172.21.15.50:6800/1043424882 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+23 (secure 0 0 0) 0x7fcf900f2de0 con 0x7fcf7802ffb0 2023-12-25T20:07:46.023 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:46.023+0000 7fcf96236700 1 -- 172.21.15.160:0/3818499076 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fcf7802ffb0 msgr2=0x7fcf78032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:07:46.023 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:46.023+0000 7fcf96236700 1 --2- 172.21.15.160:0/3818499076 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fcf7802ffb0 0x7fcf78032470 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fcf800096a0 tx=0x7fcf80006b40).stop 2023-12-25T20:07:46.023 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:46.023+0000 7fcf96236700 1 -- 172.21.15.160:0/3818499076 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcf900f6510 msgr2=0x7fcf901088f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:07:46.023 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:46.023+0000 7fcf96236700 1 --2- 172.21.15.160:0/3818499076 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcf900f6510 0x7fcf901088f0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7fcf84013510 tx=0x7fcf8400b040).stop 2023-12-25T20:07:46.023 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:46.023+0000 7fcf96236700 1 -- 172.21.15.160:0/3818499076 shutdown_connections 2023-12-25T20:07:46.023 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:46.023+0000 7fcf96236700 1 --2- 172.21.15.160:0/3818499076 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fcf7802ffb0 0x7fcf78032470 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:07:46.023 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:46.023+0000 7fcf96236700 1 --2- 172.21.15.160:0/3818499076 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcf900f6510 0x7fcf901088f0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:07:46.023 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:46.023+0000 7fcf96236700 1 -- 172.21.15.160:0/3818499076 >> 172.21.15.160:0/3818499076 conn(0x7fcf900f1c70 msgr2=0x7fcf900f26d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:07:46.023 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:46.023+0000 7fcf96236700 1 -- 172.21.15.160:0/3818499076 shutdown_connections 2023-12-25T20:07:46.023 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:46.023+0000 7fcf96236700 1 -- 172.21.15.160:0/3818499076 wait complete. 2023-12-25T20:07:46.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:07:45 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:07:44.625311+0000 mgr.smithi050.nbzhun (mgr.14162) 11 : cluster [DBG] pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:07:46.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:07:45 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:07:45.415107+0000 mgr.smithi050.nbzhun (mgr.14162) 12 : audit [DBG] from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi160", "target": ["mon-mgr", ""]}]: dispatch 2023-12-25T20:07:46.380 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph orch host ls --format=json 2023-12-25T20:07:47.469 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:07:47 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:07:46.020360+0000 mon.smithi050 (mon.0) 117 : audit [INF] from='mgr.14162 172.21.15.50:0/3178367813' entity='mgr.smithi050.nbzhun' 2023-12-25T20:07:47.469 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:07:47 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cephadm 2023-12-25T20:07:46.020821+0000 mgr.smithi050.nbzhun (mgr.14162) 13 : cephadm [INF] Added host smithi160 2023-12-25T20:07:47.564 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:47.562+0000 7ff160e68700 1 -- 172.21.15.160:0/3009487806 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff15c0f4740 msgr2=0x7ff15c0f4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:07:47.564 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:47.562+0000 7ff160e68700 1 --2- 172.21.15.160:0/3009487806 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff15c0f4740 0x7ff15c0f4b60 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7ff144009fb0 tx=0x7ff144009b30).stop 2023-12-25T20:07:47.564 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:47.562+0000 7ff160e68700 1 -- 172.21.15.160:0/3009487806 shutdown_connections 2023-12-25T20:07:47.564 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:47.562+0000 7ff160e68700 1 --2- 172.21.15.160:0/3009487806 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff15c0f4740 0x7ff15c0f4b60 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:07:47.564 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:47.562+0000 7ff160e68700 1 -- 172.21.15.160:0/3009487806 >> 172.21.15.160:0/3009487806 conn(0x7ff15c0f0260 msgr2=0x7ff15c0f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:07:47.564 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:47.562+0000 7ff160e68700 1 -- 172.21.15.160:0/3009487806 shutdown_connections 2023-12-25T20:07:47.564 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:47.562+0000 7ff160e68700 1 -- 172.21.15.160:0/3009487806 wait complete. 2023-12-25T20:07:47.564 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:47.563+0000 7ff160e68700 1 Processor -- start 2023-12-25T20:07:47.564 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:47.563+0000 7ff160e68700 1 -- start start 2023-12-25T20:07:47.565 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:47.563+0000 7ff160e68700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff15c0f4740 0x7ff15c1066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:07:47.565 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:47.563+0000 7ff160e68700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff15c106bc0 con 0x7ff15c0f4740 2023-12-25T20:07:47.565 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:47.563+0000 7ff15a59c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff15c0f4740 0x7ff15c1066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:07:47.565 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:47.563+0000 7ff15a59c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff15c0f4740 0x7ff15c1066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:56766/0 (socket says 172.21.15.160:56766) 2023-12-25T20:07:47.565 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:47.563+0000 7ff15a59c700 1 -- 172.21.15.160:0/2062816581 learned_addr learned my addr 172.21.15.160:0/2062816581 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:07:47.565 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:47.563+0000 7ff15a59c700 1 -- 172.21.15.160:0/2062816581 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff144005040 con 0x7ff15c0f4740 2023-12-25T20:07:47.565 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:47.564+0000 7ff15a59c700 1 --2- 172.21.15.160:0/2062816581 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff15c0f4740 0x7ff15c1066d0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7ff144013510 tx=0x7ff14400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:07:47.565 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:47.564+0000 7ff152ffd700 1 -- 172.21.15.160:0/2062816581 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff144009320 con 0x7ff15c0f4740 2023-12-25T20:07:47.565 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:47.564+0000 7ff152ffd700 1 -- 172.21.15.160:0/2062816581 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff144013f00 con 0x7ff15c0f4740 2023-12-25T20:07:47.565 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:47.564+0000 7ff160e68700 1 -- 172.21.15.160:0/2062816581 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff15c106dc0 con 0x7ff15c0f4740 2023-12-25T20:07:47.566 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:47.564+0000 7ff152ffd700 1 -- 172.21.15.160:0/2062816581 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff144014d90 con 0x7ff15c0f4740 2023-12-25T20:07:47.566 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:47.564+0000 7ff160e68700 1 -- 172.21.15.160:0/2062816581 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff15c107250 con 0x7ff15c0f4740 2023-12-25T20:07:47.567 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:47.565+0000 7ff152ffd700 1 -- 172.21.15.160:0/2062816581 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff1440095f0 con 0x7ff15c0f4740 2023-12-25T20:07:47.567 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:47.565+0000 7ff160e68700 1 -- 172.21.15.160:0/2062816581 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff15c045ad0 con 0x7ff15c0f4740 2023-12-25T20:07:47.567 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:47.565+0000 7ff152ffd700 1 --2- 172.21.15.160:0/2062816581 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7ff14802ff60 0x7ff148032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:07:47.567 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:47.565+0000 7ff152ffd700 1 -- 172.21.15.160:0/2062816581 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff144040fc0 con 0x7ff15c0f4740 2023-12-25T20:07:47.567 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:47.565+0000 7ff159d9b700 1 --2- 172.21.15.160:0/2062816581 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7ff14802ff60 0x7ff148032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:07:47.567 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:47.565+0000 7ff159d9b700 1 --2- 172.21.15.160:0/2062816581 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7ff14802ff60 0x7ff148032420 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7ff14c0096a0 tx=0x7ff14c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:07:47.569 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:47.570+0000 7ff152ffd700 1 -- 172.21.15.160:0/2062816581 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff144014420 con 0x7ff15c0f4740 2023-12-25T20:07:47.740 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:47.740+0000 7ff160e68700 1 -- 172.21.15.160:0/2062816581 --> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff15c0f1490 con 0x7ff14802ff60 2023-12-25T20:07:47.742 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:47.742+0000 7ff152ffd700 1 -- 172.21.15.160:0/2062816581 <== mgr.14162 v2:172.21.15.50:6800/1043424882 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+156 (secure 0 0 0) 0x7ff15c0f1490 con 0x7ff14802ff60 2023-12-25T20:07:47.742 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:07:47.742 INFO:teuthology.orchestra.run.smithi160.stdout:[{"addr": "172.21.15.50", "hostname": "smithi050", "labels": [], "status": ""}, {"addr": "smithi160", "hostname": "smithi160", "labels": [], "status": ""}] 2023-12-25T20:07:47.743 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:47.743+0000 7ff160e68700 1 -- 172.21.15.160:0/2062816581 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7ff14802ff60 msgr2=0x7ff148032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:07:47.744 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:47.743+0000 7ff160e68700 1 --2- 172.21.15.160:0/2062816581 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7ff14802ff60 0x7ff148032420 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7ff14c0096a0 tx=0x7ff14c006b40).stop 2023-12-25T20:07:47.744 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:47.744+0000 7ff160e68700 1 -- 172.21.15.160:0/2062816581 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff15c0f4740 msgr2=0x7ff15c1066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:07:47.744 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:47.744+0000 7ff160e68700 1 --2- 172.21.15.160:0/2062816581 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff15c0f4740 0x7ff15c1066d0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7ff144013510 tx=0x7ff14400b040).stop 2023-12-25T20:07:47.744 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:47.744+0000 7ff160e68700 1 -- 172.21.15.160:0/2062816581 shutdown_connections 2023-12-25T20:07:47.744 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:47.744+0000 7ff160e68700 1 --2- 172.21.15.160:0/2062816581 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7ff14802ff60 0x7ff148032420 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:07:47.744 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:47.744+0000 7ff160e68700 1 --2- 172.21.15.160:0/2062816581 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff15c0f4740 0x7ff15c1066d0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:07:47.744 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:47.744+0000 7ff160e68700 1 -- 172.21.15.160:0/2062816581 >> 172.21.15.160:0/2062816581 conn(0x7ff15c0f0260 msgr2=0x7ff15c0f0d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:07:47.744 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:47.744+0000 7ff160e68700 1 -- 172.21.15.160:0/2062816581 shutdown_connections 2023-12-25T20:07:47.744 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:47.744+0000 7ff160e68700 1 -- 172.21.15.160:0/2062816581 wait complete. 2023-12-25T20:07:48.131 INFO:tasks.cephadm:Setting crush tunables to default 2023-12-25T20:07:48.131 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph osd crush tunables default 2023-12-25T20:07:48.456 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:07:48 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:07:46.625748+0000 mgr.smithi050.nbzhun (mgr.14162) 14 : cluster [DBG] pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:07:49.116 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:07:49 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:07:47.742006+0000 mgr.smithi050.nbzhun (mgr.14162) 15 : audit [DBG] from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-12-25T20:07:49.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-25T20:07:49.311+0000 7fbc14536700 1 -- 172.21.15.50:0/4272269968 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbc0c0f6510 msgr2=0x7fbc0c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:07:49.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-25T20:07:49.311+0000 7fbc14536700 1 --2- 172.21.15.50:0/4272269968 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbc0c0f6510 0x7fbc0c0f6930 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7fbc08009fb0 tx=0x7fbc08009b30).stop 2023-12-25T20:07:49.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-25T20:07:49.311+0000 7fbc14536700 1 -- 172.21.15.50:0/4272269968 shutdown_connections 2023-12-25T20:07:49.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-25T20:07:49.311+0000 7fbc14536700 1 --2- 172.21.15.50:0/4272269968 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbc0c0f6510 0x7fbc0c0f6930 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:07:49.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-25T20:07:49.311+0000 7fbc14536700 1 -- 172.21.15.50:0/4272269968 >> 172.21.15.50:0/4272269968 conn(0x7fbc0c0f1c70 msgr2=0x7fbc0c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:07:49.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-25T20:07:49.311+0000 7fbc14536700 1 -- 172.21.15.50:0/4272269968 shutdown_connections 2023-12-25T20:07:49.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-25T20:07:49.311+0000 7fbc14536700 1 -- 172.21.15.50:0/4272269968 wait complete. 2023-12-25T20:07:49.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-25T20:07:49.311+0000 7fbc14536700 1 Processor -- start 2023-12-25T20:07:49.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-25T20:07:49.311+0000 7fbc14536700 1 -- start start 2023-12-25T20:07:49.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-25T20:07:49.311+0000 7fbc14536700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbc0c0f6510 0x7fbc0c06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:07:49.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-25T20:07:49.311+0000 7fbc14536700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc0c069e20 con 0x7fbc0c0f6510 2023-12-25T20:07:49.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-25T20:07:49.312+0000 7fbc122d2700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbc0c0f6510 0x7fbc0c06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:07:49.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-25T20:07:49.312+0000 7fbc122d2700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbc0c0f6510 0x7fbc0c06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:35972/0 (socket says 172.21.15.50:35972) 2023-12-25T20:07:49.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-25T20:07:49.312+0000 7fbc122d2700 1 -- 172.21.15.50:0/1304683141 learned_addr learned my addr 172.21.15.50:0/1304683141 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-25T20:07:49.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-25T20:07:49.312+0000 7fbc122d2700 1 -- 172.21.15.50:0/1304683141 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc08005040 con 0x7fbc0c0f6510 2023-12-25T20:07:49.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-25T20:07:49.312+0000 7fbc122d2700 1 --2- 172.21.15.50:0/1304683141 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbc0c0f6510 0x7fbc0c06b9a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7fbc08013510 tx=0x7fbc0800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:07:49.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-25T20:07:49.312+0000 7fbc02ffd700 1 -- 172.21.15.50:0/1304683141 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbc08009320 con 0x7fbc0c0f6510 2023-12-25T20:07:49.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-25T20:07:49.312+0000 7fbc14536700 1 -- 172.21.15.50:0/1304683141 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc0c069fc0 con 0x7fbc0c0f6510 2023-12-25T20:07:49.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-25T20:07:49.312+0000 7fbc02ffd700 1 -- 172.21.15.50:0/1304683141 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbc08013f00 con 0x7fbc0c0f6510 2023-12-25T20:07:49.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-25T20:07:49.312+0000 7fbc02ffd700 1 -- 172.21.15.50:0/1304683141 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbc08014d90 con 0x7fbc0c0f6510 2023-12-25T20:07:49.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-25T20:07:49.312+0000 7fbc14536700 1 -- 172.21.15.50:0/1304683141 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc0c06a450 con 0x7fbc0c0f6510 2023-12-25T20:07:49.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-25T20:07:49.313+0000 7fbc02ffd700 1 -- 172.21.15.50:0/1304683141 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbc080263a0 con 0x7fbc0c0f6510 2023-12-25T20:07:49.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-25T20:07:49.313+0000 7fbc14536700 1 -- 172.21.15.50:0/1304683141 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc0c059100 con 0x7fbc0c0f6510 2023-12-25T20:07:49.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-25T20:07:49.313+0000 7fbc02ffd700 1 --2- 172.21.15.50:0/1304683141 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fbbf802ff60 0x7fbbf8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:07:49.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-25T20:07:49.313+0000 7fbc02ffd700 1 -- 172.21.15.50:0/1304683141 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbc080412d0 con 0x7fbc0c0f6510 2023-12-25T20:07:49.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-25T20:07:49.313+0000 7fbc11ad1700 1 --2- 172.21.15.50:0/1304683141 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fbbf802ff60 0x7fbbf8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:07:49.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-25T20:07:49.314+0000 7fbc11ad1700 1 --2- 172.21.15.50:0/1304683141 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fbbf802ff60 0x7fbbf8032420 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7fbbfc0096a0 tx=0x7fbbfc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:07:49.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-25T20:07:49.317+0000 7fbc02ffd700 1 -- 172.21.15.50:0/1304683141 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbc08012510 con 0x7fbc0c0f6510 2023-12-25T20:07:49.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-25T20:07:49.484+0000 7fbc14536700 1 -- 172.21.15.50:0/1304683141 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fbc0c0fbf60 con 0x7fbc0c0f6510 2023-12-25T20:07:50.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-25T20:07:50.025+0000 7fbc02ffd700 1 -- 172.21.15.50:0/1304683141 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v5) v1 ==== 124+0+0 (secure 0 0 0) 0x7fbc08012330 con 0x7fbc0c0f6510 2023-12-25T20:07:50.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-25T20:07:50.027+0000 7fbc14536700 1 -- 172.21.15.50:0/1304683141 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fbbf802ff60 msgr2=0x7fbbf8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:07:50.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-25T20:07:50.027+0000 7fbc14536700 1 --2- 172.21.15.50:0/1304683141 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fbbf802ff60 0x7fbbf8032420 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7fbbfc0096a0 tx=0x7fbbfc006b40).stop 2023-12-25T20:07:50.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-25T20:07:50.027+0000 7fbc14536700 1 -- 172.21.15.50:0/1304683141 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbc0c0f6510 msgr2=0x7fbc0c06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:07:50.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-25T20:07:50.027+0000 7fbc14536700 1 --2- 172.21.15.50:0/1304683141 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbc0c0f6510 0x7fbc0c06b9a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7fbc08013510 tx=0x7fbc0800b040).stop 2023-12-25T20:07:50.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-25T20:07:50.027+0000 7fbc14536700 1 -- 172.21.15.50:0/1304683141 shutdown_connections 2023-12-25T20:07:50.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-25T20:07:50.027+0000 7fbc14536700 1 --2- 172.21.15.50:0/1304683141 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fbbf802ff60 0x7fbbf8032420 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:07:50.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-25T20:07:50.027+0000 7fbc14536700 1 --2- 172.21.15.50:0/1304683141 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbc0c0f6510 0x7fbc0c06b9a0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:07:50.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-25T20:07:50.027+0000 7fbc14536700 1 -- 172.21.15.50:0/1304683141 >> 172.21.15.50:0/1304683141 conn(0x7fbc0c0f1c70 msgr2=0x7fbc0c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:07:50.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-25T20:07:50.028+0000 7fbc14536700 1 -- 172.21.15.50:0/1304683141 shutdown_connections 2023-12-25T20:07:50.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-25T20:07:50.028+0000 7fbc14536700 1 -- 172.21.15.50:0/1304683141 wait complete. 2023-12-25T20:07:50.029 INFO:teuthology.orchestra.run.smithi050.stderr:adjusted tunables profile to default 2023-12-25T20:07:50.360 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:07:50 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:07:48.626126+0000 mgr.smithi050.nbzhun (mgr.14162) 16 : cluster [DBG] pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:07:50.360 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:07:50 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:07:49.485467+0000 mon.smithi050 (mon.0) 118 : audit [INF] from='client.? 172.21.15.50:0/1304683141' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2023-12-25T20:07:50.400 INFO:tasks.cephadm:Adding mon.smithi050 on smithi050 2023-12-25T20:07:50.400 INFO:tasks.cephadm:Adding mon.smithi160 on smithi160 2023-12-25T20:07:50.400 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph orch apply mon '2;smithi050:172.21.15.50=smithi050;smithi160:172.21.15.160=smithi160' 2023-12-25T20:07:51.469 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:07:51 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:07:50.026805+0000 mon.smithi050 (mon.0) 119 : audit [INF] from='client.? 172.21.15.50:0/1304683141' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2023-12-25T20:07:51.469 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:07:51 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:07:50.026862+0000 mon.smithi050 (mon.0) 120 : cluster [DBG] osdmap e5: 0 total, 0 up, 0 in 2023-12-25T20:07:51.562 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:51.562+0000 7fecd759e700 1 -- 172.21.15.160:0/3436902770 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fecd80f51a0 msgr2=0x7fecd80f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:07:51.562 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:51.562+0000 7fecd759e700 1 --2- 172.21.15.160:0/3436902770 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fecd80f51a0 0x7fecd80f55c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7fecc8009fb0 tx=0x7fecc8009b30).stop 2023-12-25T20:07:51.562 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:51.562+0000 7fecd759e700 1 -- 172.21.15.160:0/3436902770 shutdown_connections 2023-12-25T20:07:51.563 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:51.562+0000 7fecd759e700 1 --2- 172.21.15.160:0/3436902770 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fecd80f51a0 0x7fecd80f55c0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:07:51.563 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:51.562+0000 7fecd759e700 1 -- 172.21.15.160:0/3436902770 >> 172.21.15.160:0/3436902770 conn(0x7fecd80f09a0 msgr2=0x7fecd80f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:07:51.563 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:51.562+0000 7fecd759e700 1 -- 172.21.15.160:0/3436902770 shutdown_connections 2023-12-25T20:07:51.563 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:51.562+0000 7fecd759e700 1 -- 172.21.15.160:0/3436902770 wait complete. 2023-12-25T20:07:51.563 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:51.562+0000 7fecd759e700 1 Processor -- start 2023-12-25T20:07:51.563 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:51.562+0000 7fecd759e700 1 -- start start 2023-12-25T20:07:51.565 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:51.563+0000 7fecd759e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fecd80f51a0 0x7fecd81091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:07:51.565 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:51.563+0000 7fecd759e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecd8060240 con 0x7fecd80f51a0 2023-12-25T20:07:51.565 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:51.563+0000 7fecd659c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fecd80f51a0 0x7fecd81091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:07:51.565 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:51.563+0000 7fecd659c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fecd80f51a0 0x7fecd81091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:56788/0 (socket says 172.21.15.160:56788) 2023-12-25T20:07:51.565 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:51.563+0000 7fecd659c700 1 -- 172.21.15.160:0/1314337768 learned_addr learned my addr 172.21.15.160:0/1314337768 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:07:51.565 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:51.563+0000 7fecd659c700 1 -- 172.21.15.160:0/1314337768 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecc8005040 con 0x7fecd80f51a0 2023-12-25T20:07:51.565 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:51.563+0000 7fecd659c700 1 --2- 172.21.15.160:0/1314337768 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fecd80f51a0 0x7fecd81091b0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7fecc8013510 tx=0x7fecc800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:07:51.566 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:51.564+0000 7fecceffd700 1 -- 172.21.15.160:0/1314337768 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fecc8009320 con 0x7fecd80f51a0 2023-12-25T20:07:51.566 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:51.564+0000 7fecd759e700 1 -- 172.21.15.160:0/1314337768 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecd8060440 con 0x7fecd80f51a0 2023-12-25T20:07:51.566 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:51.564+0000 7fecceffd700 1 -- 172.21.15.160:0/1314337768 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fecc8013f00 con 0x7fecd80f51a0 2023-12-25T20:07:51.566 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:51.564+0000 7fecceffd700 1 -- 172.21.15.160:0/1314337768 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fecc8014dc0 con 0x7fecd80f51a0 2023-12-25T20:07:51.566 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:51.564+0000 7fecd759e700 1 -- 172.21.15.160:0/1314337768 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecd8109860 con 0x7fecd80f51a0 2023-12-25T20:07:51.566 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:51.564+0000 7fecceffd700 1 -- 172.21.15.160:0/1314337768 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fecc8014420 con 0x7fecd80f51a0 2023-12-25T20:07:51.566 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:51.564+0000 7fecd759e700 1 -- 172.21.15.160:0/1314337768 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fecd8045ad0 con 0x7fecd80f51a0 2023-12-25T20:07:51.566 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:51.565+0000 7fecceffd700 1 --2- 172.21.15.160:0/1314337768 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fecc4030000 0x7fecc40324c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:07:51.566 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:51.565+0000 7fecceffd700 1 -- 172.21.15.160:0/1314337768 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fecc8041070 con 0x7fecd80f51a0 2023-12-25T20:07:51.566 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:51.565+0000 7fecd5d9b700 1 --2- 172.21.15.160:0/1314337768 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fecc4030000 0x7fecc40324c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:07:51.566 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:51.565+0000 7fecd5d9b700 1 --2- 172.21.15.160:0/1314337768 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fecc4030000 0x7fecc40324c0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7fecc00096a0 tx=0x7fecc0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:07:51.569 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:51.569+0000 7fecceffd700 1 -- 172.21.15.160:0/1314337768 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fecc806d2b0 con 0x7fecd80f51a0 2023-12-25T20:07:51.745 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:51.745+0000 7fecd759e700 1 -- 172.21.15.160:0/1314337768 --> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi050:172.21.15.50=smithi050;smithi160:172.21.15.160=smithi160", "target": ["mon-mgr", ""]}) v1 -- 0x7fecd80f1c20 con 0x7fecc4030000 2023-12-25T20:07:51.751 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:51.751+0000 7fecceffd700 1 -- 172.21.15.160:0/1314337768 <== mgr.14162 v2:172.21.15.50:6800/1043424882 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fecd80f1c20 con 0x7fecc4030000 2023-12-25T20:07:51.751 INFO:teuthology.orchestra.run.smithi160.stdout:Scheduled mon update... 2023-12-25T20:07:51.753 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:51.753+0000 7fecd759e700 1 -- 172.21.15.160:0/1314337768 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fecc4030000 msgr2=0x7fecc40324c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:07:51.753 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:51.753+0000 7fecd759e700 1 --2- 172.21.15.160:0/1314337768 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fecc4030000 0x7fecc40324c0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7fecc00096a0 tx=0x7fecc0006b40).stop 2023-12-25T20:07:51.753 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:51.753+0000 7fecd759e700 1 -- 172.21.15.160:0/1314337768 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fecd80f51a0 msgr2=0x7fecd81091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:07:51.753 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:51.753+0000 7fecd759e700 1 --2- 172.21.15.160:0/1314337768 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fecd80f51a0 0x7fecd81091b0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7fecc8013510 tx=0x7fecc800b040).stop 2023-12-25T20:07:51.753 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:51.753+0000 7fecd759e700 1 -- 172.21.15.160:0/1314337768 shutdown_connections 2023-12-25T20:07:51.753 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:51.753+0000 7fecd759e700 1 --2- 172.21.15.160:0/1314337768 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fecc4030000 0x7fecc40324c0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:07:51.753 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:51.753+0000 7fecd759e700 1 --2- 172.21.15.160:0/1314337768 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fecd80f51a0 0x7fecd81091b0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:07:51.753 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:51.753+0000 7fecd759e700 1 -- 172.21.15.160:0/1314337768 >> 172.21.15.160:0/1314337768 conn(0x7fecd80f09a0 msgr2=0x7fecd80f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:07:51.753 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:51.753+0000 7fecd759e700 1 -- 172.21.15.160:0/1314337768 shutdown_connections 2023-12-25T20:07:51.753 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:51.753+0000 7fecd759e700 1 -- 172.21.15.160:0/1314337768 wait complete. 2023-12-25T20:07:52.123 DEBUG:teuthology.orchestra.run.smithi160:mon.smithi160> sudo journalctl -f -n 0 -u ceph-02c8f25c-a361-11ee-95a5-87774f69a715@mon.smithi160.service 2023-12-25T20:07:52.126 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:07:52.126 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:07:52.154 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:-- Logs begin at Mon 2023-12-25 19:53:38 UTC. -- 2023-12-25T20:07:52.469 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:07:52 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:07:50.626490+0000 mgr.smithi050.nbzhun (mgr.14162) 17 : cluster [DBG] pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:07:52.469 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:07:52 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:07:51.751567+0000 mon.smithi050 (mon.0) 121 : audit [INF] from='mgr.14162 172.21.15.50:0/3178367813' entity='mgr.smithi050.nbzhun' 2023-12-25T20:07:53.269 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:53.269+0000 7fa321e0c700 1 -- 172.21.15.160:0/4207481259 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa31c0f6510 msgr2=0x7fa31c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:07:53.270 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:53.269+0000 7fa321e0c700 1 --2- 172.21.15.160:0/4207481259 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa31c0f6510 0x7fa31c0f6930 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7fa310009fb0 tx=0x7fa310009b30).stop 2023-12-25T20:07:53.270 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:53.269+0000 7fa321e0c700 1 -- 172.21.15.160:0/4207481259 shutdown_connections 2023-12-25T20:07:53.270 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:53.269+0000 7fa321e0c700 1 --2- 172.21.15.160:0/4207481259 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa31c0f6510 0x7fa31c0f6930 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:07:53.270 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:53.269+0000 7fa321e0c700 1 -- 172.21.15.160:0/4207481259 >> 172.21.15.160:0/4207481259 conn(0x7fa31c0f1c70 msgr2=0x7fa31c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:07:53.270 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:53.269+0000 7fa321e0c700 1 -- 172.21.15.160:0/4207481259 shutdown_connections 2023-12-25T20:07:53.270 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:53.269+0000 7fa321e0c700 1 -- 172.21.15.160:0/4207481259 wait complete. 2023-12-25T20:07:53.270 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:53.269+0000 7fa321e0c700 1 Processor -- start 2023-12-25T20:07:53.270 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:53.270+0000 7fa321e0c700 1 -- start start 2023-12-25T20:07:53.270 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:53.270+0000 7fa321e0c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa31c0f6510 0x7fa31c06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:07:53.270 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:53.270+0000 7fa321e0c700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa31c069e20 con 0x7fa31c0f6510 2023-12-25T20:07:53.271 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:53.270+0000 7fa31b7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa31c0f6510 0x7fa31c06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:07:53.271 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:53.270+0000 7fa31b7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa31c0f6510 0x7fa31c06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:56806/0 (socket says 172.21.15.160:56806) 2023-12-25T20:07:53.272 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:53.270+0000 7fa31b7fe700 1 -- 172.21.15.160:0/2676997049 learned_addr learned my addr 172.21.15.160:0/2676997049 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:07:53.272 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:53.270+0000 7fa31b7fe700 1 -- 172.21.15.160:0/2676997049 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa310005040 con 0x7fa31c0f6510 2023-12-25T20:07:53.272 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:53.271+0000 7fa31b7fe700 1 --2- 172.21.15.160:0/2676997049 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa31c0f6510 0x7fa31c06b9a0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7fa310013510 tx=0x7fa31000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:07:53.272 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:53.271+0000 7fa303fff700 1 -- 172.21.15.160:0/2676997049 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa310009320 con 0x7fa31c0f6510 2023-12-25T20:07:53.272 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:53.271+0000 7fa303fff700 1 -- 172.21.15.160:0/2676997049 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa310013f00 con 0x7fa31c0f6510 2023-12-25T20:07:53.272 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:53.271+0000 7fa321e0c700 1 -- 172.21.15.160:0/2676997049 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa31c069fc0 con 0x7fa31c0f6510 2023-12-25T20:07:53.272 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:53.271+0000 7fa303fff700 1 -- 172.21.15.160:0/2676997049 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa310014d90 con 0x7fa31c0f6510 2023-12-25T20:07:53.272 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:53.271+0000 7fa321e0c700 1 -- 172.21.15.160:0/2676997049 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa31c06a450 con 0x7fa31c0f6510 2023-12-25T20:07:53.272 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:53.272+0000 7fa303fff700 1 -- 172.21.15.160:0/2676997049 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa310014420 con 0x7fa31c0f6510 2023-12-25T20:07:53.272 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:53.272+0000 7fa321e0c700 1 -- 172.21.15.160:0/2676997049 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa31c059100 con 0x7fa31c0f6510 2023-12-25T20:07:53.272 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:53.272+0000 7fa303fff700 1 --2- 172.21.15.160:0/2676997049 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fa30402ff60 0x7fa304032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:07:53.272 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:53.272+0000 7fa303fff700 1 -- 172.21.15.160:0/2676997049 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa3100410c0 con 0x7fa31c0f6510 2023-12-25T20:07:53.272 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:53.272+0000 7fa31affd700 1 --2- 172.21.15.160:0/2676997049 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fa30402ff60 0x7fa304032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:07:53.273 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:53.272+0000 7fa31affd700 1 --2- 172.21.15.160:0/2676997049 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fa30402ff60 0x7fa304032420 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7fa30c0096a0 tx=0x7fa30c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:07:53.276 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:53.276+0000 7fa303fff700 1 -- 172.21.15.160:0/2676997049 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa3100095f0 con 0x7fa31c0f6510 2023-12-25T20:07:53.469 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:07:53 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:07:51.747465+0000 mgr.smithi050.nbzhun (mgr.14162) 18 : audit [DBG] from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi050:172.21.15.50=smithi050;smithi160:172.21.15.160=smithi160", "target": ["mon-mgr", ""]}]: dispatch 2023-12-25T20:07:53.469 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:07:53 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cephadm 2023-12-25T20:07:51.748935+0000 mgr.smithi050.nbzhun (mgr.14162) 19 : cephadm [INF] Saving service mon spec with placement smithi050:172.21.15.50=smithi050;smithi160:172.21.15.160=smithi160;count:2 2023-12-25T20:07:53.481 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:53.482+0000 7fa321e0c700 1 -- 172.21.15.160:0/2676997049 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa31c0fbf60 con 0x7fa31c0f6510 2023-12-25T20:07:53.482 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:53.482+0000 7fa303fff700 1 -- 172.21.15.160:0/2676997049 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa31001db70 con 0x7fa31c0f6510 2023-12-25T20:07:53.483 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:07:53.483 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:07:53.484 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:53.484+0000 7fa321e0c700 1 -- 172.21.15.160:0/2676997049 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fa30402ff60 msgr2=0x7fa304032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:07:53.484 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:53.484+0000 7fa321e0c700 1 --2- 172.21.15.160:0/2676997049 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fa30402ff60 0x7fa304032420 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7fa30c0096a0 tx=0x7fa30c006b40).stop 2023-12-25T20:07:53.484 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:53.484+0000 7fa321e0c700 1 -- 172.21.15.160:0/2676997049 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa31c0f6510 msgr2=0x7fa31c06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:07:53.484 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:53.484+0000 7fa321e0c700 1 --2- 172.21.15.160:0/2676997049 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa31c0f6510 0x7fa31c06b9a0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7fa310013510 tx=0x7fa31000b040).stop 2023-12-25T20:07:53.484 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:53.484+0000 7fa321e0c700 1 -- 172.21.15.160:0/2676997049 shutdown_connections 2023-12-25T20:07:53.484 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:53.484+0000 7fa321e0c700 1 --2- 172.21.15.160:0/2676997049 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fa30402ff60 0x7fa304032420 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:07:53.484 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:53.484+0000 7fa321e0c700 1 --2- 172.21.15.160:0/2676997049 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa31c0f6510 0x7fa31c06b9a0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:07:53.485 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:53.484+0000 7fa321e0c700 1 -- 172.21.15.160:0/2676997049 >> 172.21.15.160:0/2676997049 conn(0x7fa31c0f1c70 msgr2=0x7fa31c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:07:53.485 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:53.484+0000 7fa321e0c700 1 -- 172.21.15.160:0/2676997049 shutdown_connections 2023-12-25T20:07:53.485 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:53.484+0000 7fa321e0c700 1 -- 172.21.15.160:0/2676997049 wait complete. 2023-12-25T20:07:53.485 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:07:54.468 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:07:54 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:07:52.626830+0000 mgr.smithi050.nbzhun (mgr.14162) 20 : cluster [DBG] pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:07:54.469 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:07:54 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:07:53.483039+0000 mon.smithi050 (mon.0) 122 : audit [DBG] from='client.? 172.21.15.160:0/2676997049' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:07:54.871 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:07:54.871 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:07:56.063 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:56.062+0000 7f8841633700 1 -- 172.21.15.160:0/128949434 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f883c0f6a00 msgr2=0x7f883c0f6e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:07:56.063 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:56.062+0000 7f8841633700 1 --2- 172.21.15.160:0/128949434 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f883c0f6a00 0x7f883c0f6e20 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f882c009fb0 tx=0x7f882c009b30).stop 2023-12-25T20:07:56.063 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:56.062+0000 7f8841633700 1 -- 172.21.15.160:0/128949434 shutdown_connections 2023-12-25T20:07:56.063 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:56.062+0000 7f8841633700 1 --2- 172.21.15.160:0/128949434 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f883c0f6a00 0x7f883c0f6e20 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:07:56.063 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:56.062+0000 7f8841633700 1 -- 172.21.15.160:0/128949434 >> 172.21.15.160:0/128949434 conn(0x7f883c0f0260 msgr2=0x7f883c0f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:07:56.063 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:56.062+0000 7f8841633700 1 -- 172.21.15.160:0/128949434 shutdown_connections 2023-12-25T20:07:56.064 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:56.062+0000 7f8841633700 1 -- 172.21.15.160:0/128949434 wait complete. 2023-12-25T20:07:56.064 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:56.063+0000 7f8841633700 1 Processor -- start 2023-12-25T20:07:56.064 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:56.063+0000 7f8841633700 1 -- start start 2023-12-25T20:07:56.064 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:56.063+0000 7f8841633700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f883c0f6a00 0x7f883c10b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:07:56.064 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:56.063+0000 7f8841633700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f883c1071c0 con 0x7f883c0f6a00 2023-12-25T20:07:56.066 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:56.064+0000 7f883affd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f883c0f6a00 0x7f883c10b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:07:56.066 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:56.064+0000 7f883affd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f883c0f6a00 0x7f883c10b7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:56838/0 (socket says 172.21.15.160:56838) 2023-12-25T20:07:56.066 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:56.064+0000 7f883affd700 1 -- 172.21.15.160:0/146853425 learned_addr learned my addr 172.21.15.160:0/146853425 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:07:56.066 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:56.064+0000 7f883affd700 1 -- 172.21.15.160:0/146853425 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f882c005040 con 0x7f883c0f6a00 2023-12-25T20:07:56.066 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:56.064+0000 7f883affd700 1 --2- 172.21.15.160:0/146853425 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f883c0f6a00 0x7f883c10b7d0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f882c013510 tx=0x7f882c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:07:56.066 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:56.064+0000 7f88237fe700 1 -- 172.21.15.160:0/146853425 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f882c009320 con 0x7f883c0f6a00 2023-12-25T20:07:56.066 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:56.064+0000 7f8841633700 1 -- 172.21.15.160:0/146853425 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f883c0f5290 con 0x7f883c0f6a00 2023-12-25T20:07:56.066 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:56.064+0000 7f88237fe700 1 -- 172.21.15.160:0/146853425 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f882c013f00 con 0x7f883c0f6a00 2023-12-25T20:07:56.066 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:56.064+0000 7f88237fe700 1 -- 172.21.15.160:0/146853425 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f882c014dc0 con 0x7f883c0f6a00 2023-12-25T20:07:56.066 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:56.065+0000 7f8841633700 1 -- 172.21.15.160:0/146853425 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f883c10bcc0 con 0x7f883c0f6a00 2023-12-25T20:07:56.066 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:56.065+0000 7f88237fe700 1 -- 172.21.15.160:0/146853425 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f882c01d3d0 con 0x7f883c0f6a00 2023-12-25T20:07:56.067 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:56.065+0000 7f88237fe700 1 --2- 172.21.15.160:0/146853425 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f882402ff60 0x7f8824032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:07:56.067 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:56.065+0000 7f88237fe700 1 -- 172.21.15.160:0/146853425 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f882c041fd0 con 0x7f883c0f6a00 2023-12-25T20:07:56.067 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:56.066+0000 7f883a7fc700 1 --2- 172.21.15.160:0/146853425 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f882402ff60 0x7f8824032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:07:56.067 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:56.066+0000 7f8841633700 1 -- 172.21.15.160:0/146853425 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f883c059100 con 0x7f883c0f6a00 2023-12-25T20:07:56.067 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:56.066+0000 7f883a7fc700 1 --2- 172.21.15.160:0/146853425 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f882402ff60 0x7f8824032420 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f88300096a0 tx=0x7f8830006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:07:56.070 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:56.070+0000 7f88237fe700 1 -- 172.21.15.160:0/146853425 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f882c012920 con 0x7f883c0f6a00 2023-12-25T20:07:56.277 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:56.277+0000 7f8841633700 1 -- 172.21.15.160:0/146853425 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f883c045ad0 con 0x7f883c0f6a00 2023-12-25T20:07:56.277 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:56.278+0000 7f88237fe700 1 -- 172.21.15.160:0/146853425 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f883c045ad0 con 0x7f883c0f6a00 2023-12-25T20:07:56.278 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:07:56.278 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:07:56.279 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:56.279+0000 7f8841633700 1 -- 172.21.15.160:0/146853425 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f882402ff60 msgr2=0x7f8824032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:07:56.279 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:56.279+0000 7f8841633700 1 --2- 172.21.15.160:0/146853425 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f882402ff60 0x7f8824032420 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f88300096a0 tx=0x7f8830006b40).stop 2023-12-25T20:07:56.279 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:56.279+0000 7f8841633700 1 -- 172.21.15.160:0/146853425 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f883c0f6a00 msgr2=0x7f883c10b7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:07:56.280 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:56.279+0000 7f8841633700 1 --2- 172.21.15.160:0/146853425 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f883c0f6a00 0x7f883c10b7d0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f882c013510 tx=0x7f882c00b040).stop 2023-12-25T20:07:56.280 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:56.279+0000 7f8841633700 1 -- 172.21.15.160:0/146853425 shutdown_connections 2023-12-25T20:07:56.280 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:56.279+0000 7f8841633700 1 --2- 172.21.15.160:0/146853425 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f882402ff60 0x7f8824032420 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:07:56.280 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:56.279+0000 7f8841633700 1 --2- 172.21.15.160:0/146853425 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f883c0f6a00 0x7f883c10b7d0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:07:56.280 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:56.279+0000 7f8841633700 1 -- 172.21.15.160:0/146853425 >> 172.21.15.160:0/146853425 conn(0x7f883c0f0260 msgr2=0x7f883c0f0d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:07:56.280 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:56.279+0000 7f8841633700 1 -- 172.21.15.160:0/146853425 shutdown_connections 2023-12-25T20:07:56.280 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:56.279+0000 7f8841633700 1 -- 172.21.15.160:0/146853425 wait complete. 2023-12-25T20:07:56.280 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:07:56.469 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:07:56 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:07:54.627163+0000 mgr.smithi050.nbzhun (mgr.14162) 21 : cluster [DBG] pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:07:57.469 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:07:57 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:07:56.278264+0000 mon.smithi050 (mon.0) 123 : audit [DBG] from='client.? 172.21.15.160:0/146853425' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:07:57.678 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:07:57.679 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:07:58.469 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:07:58 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:07:56.627493+0000 mgr.smithi050.nbzhun (mgr.14162) 22 : cluster [DBG] pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:07:58.929 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:58.927+0000 7f672447e700 1 -- 172.21.15.160:0/2935921085 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f671c0f6510 msgr2=0x7f671c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:07:58.929 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:58.927+0000 7f672447e700 1 --2- 172.21.15.160:0/2935921085 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f671c0f6510 0x7f671c0f6930 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f670c009fb0 tx=0x7f670c009b30).stop 2023-12-25T20:07:58.929 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:58.927+0000 7f672447e700 1 -- 172.21.15.160:0/2935921085 shutdown_connections 2023-12-25T20:07:58.929 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:58.927+0000 7f672447e700 1 --2- 172.21.15.160:0/2935921085 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f671c0f6510 0x7f671c0f6930 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:07:58.929 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:58.927+0000 7f672447e700 1 -- 172.21.15.160:0/2935921085 >> 172.21.15.160:0/2935921085 conn(0x7f671c0f1c70 msgr2=0x7f671c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:07:58.929 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:58.927+0000 7f672447e700 1 -- 172.21.15.160:0/2935921085 shutdown_connections 2023-12-25T20:07:58.930 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:58.927+0000 7f672447e700 1 -- 172.21.15.160:0/2935921085 wait complete. 2023-12-25T20:07:58.930 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:58.928+0000 7f672447e700 1 Processor -- start 2023-12-25T20:07:58.930 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:58.928+0000 7f672447e700 1 -- start start 2023-12-25T20:07:58.930 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:58.928+0000 7f672447e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f671c0f6510 0x7f671c106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:07:58.930 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:58.928+0000 7f672447e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f671c106c20 con 0x7f671c0f6510 2023-12-25T20:07:58.930 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:58.928+0000 7f672221a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f671c0f6510 0x7f671c106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:07:58.930 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:58.928+0000 7f672221a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f671c0f6510 0x7f671c106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:37820/0 (socket says 172.21.15.160:37820) 2023-12-25T20:07:58.930 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:58.928+0000 7f672221a700 1 -- 172.21.15.160:0/3702433383 learned_addr learned my addr 172.21.15.160:0/3702433383 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:07:58.930 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:58.928+0000 7f672221a700 1 -- 172.21.15.160:0/3702433383 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f670c005040 con 0x7f671c0f6510 2023-12-25T20:07:58.930 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:58.928+0000 7f672221a700 1 --2- 172.21.15.160:0/3702433383 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f671c0f6510 0x7f671c106730 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f670c013510 tx=0x7f670c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:07:58.931 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:58.929+0000 7f6712ffd700 1 -- 172.21.15.160:0/3702433383 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f670c009320 con 0x7f671c0f6510 2023-12-25T20:07:58.932 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:58.929+0000 7f672447e700 1 -- 172.21.15.160:0/3702433383 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f671c106e20 con 0x7f671c0f6510 2023-12-25T20:07:58.932 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:58.929+0000 7f6712ffd700 1 -- 172.21.15.160:0/3702433383 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f670c013f00 con 0x7f671c0f6510 2023-12-25T20:07:58.932 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:58.929+0000 7f6712ffd700 1 -- 172.21.15.160:0/3702433383 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f670c014d90 con 0x7f671c0f6510 2023-12-25T20:07:58.932 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:58.929+0000 7f672447e700 1 -- 172.21.15.160:0/3702433383 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f671c1072b0 con 0x7f671c0f6510 2023-12-25T20:07:58.932 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:58.929+0000 7f6712ffd700 1 -- 172.21.15.160:0/3702433383 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f670c0095f0 con 0x7f671c0f6510 2023-12-25T20:07:58.932 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:58.929+0000 7f672447e700 1 -- 172.21.15.160:0/3702433383 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f671c045ad0 con 0x7f671c0f6510 2023-12-25T20:07:58.932 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:58.930+0000 7f6712ffd700 1 --2- 172.21.15.160:0/3702433383 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f670802ff60 0x7f6708032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:07:58.932 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:58.930+0000 7f6712ffd700 1 -- 172.21.15.160:0/3702433383 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f670c041000 con 0x7f671c0f6510 2023-12-25T20:07:58.932 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:58.930+0000 7f6721a19700 1 --2- 172.21.15.160:0/3702433383 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f670802ff60 0x7f6708032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:07:58.932 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:58.930+0000 7f6721a19700 1 --2- 172.21.15.160:0/3702433383 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f670802ff60 0x7f6708032420 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f67180096a0 tx=0x7f6718006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:07:58.935 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:58.934+0000 7f6712ffd700 1 -- 172.21.15.160:0/3702433383 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f670c014420 con 0x7f671c0f6510 2023-12-25T20:07:59.138 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:59.137+0000 7f672447e700 1 -- 172.21.15.160:0/3702433383 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f671c0fbf60 con 0x7f671c0f6510 2023-12-25T20:07:59.139 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:59.138+0000 7f6712ffd700 1 -- 172.21.15.160:0/3702433383 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f670c010230 con 0x7f671c0f6510 2023-12-25T20:07:59.139 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:07:59.139 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:07:59.141 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:59.140+0000 7f672447e700 1 -- 172.21.15.160:0/3702433383 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f670802ff60 msgr2=0x7f6708032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:07:59.141 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:59.140+0000 7f672447e700 1 --2- 172.21.15.160:0/3702433383 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f670802ff60 0x7f6708032420 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f67180096a0 tx=0x7f6718006b40).stop 2023-12-25T20:07:59.141 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:59.140+0000 7f672447e700 1 -- 172.21.15.160:0/3702433383 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f671c0f6510 msgr2=0x7f671c106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:07:59.141 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:59.140+0000 7f672447e700 1 --2- 172.21.15.160:0/3702433383 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f671c0f6510 0x7f671c106730 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f670c013510 tx=0x7f670c00b040).stop 2023-12-25T20:07:59.141 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:59.140+0000 7f672447e700 1 -- 172.21.15.160:0/3702433383 shutdown_connections 2023-12-25T20:07:59.141 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:59.140+0000 7f672447e700 1 --2- 172.21.15.160:0/3702433383 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f670802ff60 0x7f6708032420 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:07:59.141 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:59.140+0000 7f672447e700 1 --2- 172.21.15.160:0/3702433383 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f671c0f6510 0x7f671c106730 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:07:59.141 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:59.140+0000 7f672447e700 1 -- 172.21.15.160:0/3702433383 >> 172.21.15.160:0/3702433383 conn(0x7f671c0f1c70 msgr2=0x7f671c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:07:59.141 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:59.140+0000 7f672447e700 1 -- 172.21.15.160:0/3702433383 shutdown_connections 2023-12-25T20:07:59.142 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:07:59.140+0000 7f672447e700 1 -- 172.21.15.160:0/3702433383 wait complete. 2023-12-25T20:07:59.142 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:08:00.469 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:08:00 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:07:58.627850+0000 mgr.smithi050.nbzhun (mgr.14162) 23 : cluster [DBG] pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:08:00.469 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:08:00 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:07:59.139810+0000 mon.smithi050 (mon.0) 124 : audit [DBG] from='client.? 172.21.15.160:0/3702433383' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:08:00.534 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:08:00.534 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:08:01.814 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:01.812+0000 7f311adb7700 1 -- 172.21.15.160:0/3136258443 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f31140f51a0 msgr2=0x7f31140f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:08:01.814 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:01.812+0000 7f311adb7700 1 --2- 172.21.15.160:0/3136258443 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f31140f51a0 0x7f31140f55c0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f3104009fb0 tx=0x7f3104009b30).stop 2023-12-25T20:08:01.814 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:01.812+0000 7f311adb7700 1 -- 172.21.15.160:0/3136258443 shutdown_connections 2023-12-25T20:08:01.814 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:01.812+0000 7f311adb7700 1 --2- 172.21.15.160:0/3136258443 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f31140f51a0 0x7f31140f55c0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:08:01.814 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:01.812+0000 7f311adb7700 1 -- 172.21.15.160:0/3136258443 >> 172.21.15.160:0/3136258443 conn(0x7f31140f09a0 msgr2=0x7f31140f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:08:01.814 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:01.812+0000 7f311adb7700 1 -- 172.21.15.160:0/3136258443 shutdown_connections 2023-12-25T20:08:01.814 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:01.812+0000 7f311adb7700 1 -- 172.21.15.160:0/3136258443 wait complete. 2023-12-25T20:08:01.815 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:01.812+0000 7f311adb7700 1 Processor -- start 2023-12-25T20:08:01.815 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:01.812+0000 7f311adb7700 1 -- start start 2023-12-25T20:08:01.815 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:01.812+0000 7f311adb7700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f31140f51a0 0x7f31141091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:08:01.815 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:01.812+0000 7f311adb7700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3114060240 con 0x7f31140f51a0 2023-12-25T20:08:01.815 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:01.813+0000 7f3119db5700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f31140f51a0 0x7f31141091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:08:01.815 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:01.813+0000 7f3119db5700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f31140f51a0 0x7f31141091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:37840/0 (socket says 172.21.15.160:37840) 2023-12-25T20:08:01.815 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:01.813+0000 7f3119db5700 1 -- 172.21.15.160:0/2772611338 learned_addr learned my addr 172.21.15.160:0/2772611338 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:08:01.815 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:01.813+0000 7f3119db5700 1 -- 172.21.15.160:0/2772611338 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3104005040 con 0x7f31140f51a0 2023-12-25T20:08:01.815 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:01.813+0000 7f3119db5700 1 --2- 172.21.15.160:0/2772611338 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f31140f51a0 0x7f31141091b0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f3104013510 tx=0x7f310400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:08:01.815 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:01.813+0000 7f310a7fc700 1 -- 172.21.15.160:0/2772611338 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3104009320 con 0x7f31140f51a0 2023-12-25T20:08:01.816 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:01.813+0000 7f311adb7700 1 -- 172.21.15.160:0/2772611338 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3114060440 con 0x7f31140f51a0 2023-12-25T20:08:01.816 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:01.813+0000 7f310a7fc700 1 -- 172.21.15.160:0/2772611338 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3104013f00 con 0x7f31140f51a0 2023-12-25T20:08:01.816 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:01.813+0000 7f310a7fc700 1 -- 172.21.15.160:0/2772611338 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3104014dc0 con 0x7f31140f51a0 2023-12-25T20:08:01.816 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:01.813+0000 7f311adb7700 1 -- 172.21.15.160:0/2772611338 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3114109860 con 0x7f31140f51a0 2023-12-25T20:08:01.817 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:01.814+0000 7f310a7fc700 1 -- 172.21.15.160:0/2772611338 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f31040095f0 con 0x7f31140f51a0 2023-12-25T20:08:01.817 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:01.814+0000 7f311adb7700 1 -- 172.21.15.160:0/2772611338 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3114045ad0 con 0x7f31140f51a0 2023-12-25T20:08:01.817 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:01.814+0000 7f310a7fc700 1 --2- 172.21.15.160:0/2772611338 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f310002ff60 0x7f3100032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:08:01.817 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:01.814+0000 7f310a7fc700 1 -- 172.21.15.160:0/2772611338 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3104041000 con 0x7f31140f51a0 2023-12-25T20:08:01.817 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:01.815+0000 7f31195b4700 1 --2- 172.21.15.160:0/2772611338 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f310002ff60 0x7f3100032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:08:01.818 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:01.815+0000 7f31195b4700 1 --2- 172.21.15.160:0/2772611338 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f310002ff60 0x7f3100032420 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f31100096a0 tx=0x7f3110006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:08:01.820 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:01.819+0000 7f310a7fc700 1 -- 172.21.15.160:0/2772611338 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3104014420 con 0x7f31140f51a0 2023-12-25T20:08:02.023 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:02.022+0000 7f311adb7700 1 -- 172.21.15.160:0/2772611338 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f311410a080 con 0x7f31140f51a0 2023-12-25T20:08:02.024 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:02.023+0000 7f310a7fc700 1 -- 172.21.15.160:0/2772611338 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3104010230 con 0x7f31140f51a0 2023-12-25T20:08:02.024 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:08:02.024 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:08:02.026 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:02.025+0000 7f311adb7700 1 -- 172.21.15.160:0/2772611338 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f310002ff60 msgr2=0x7f3100032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:08:02.026 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:02.025+0000 7f311adb7700 1 --2- 172.21.15.160:0/2772611338 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f310002ff60 0x7f3100032420 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f31100096a0 tx=0x7f3110006b40).stop 2023-12-25T20:08:02.026 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:02.025+0000 7f311adb7700 1 -- 172.21.15.160:0/2772611338 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f31140f51a0 msgr2=0x7f31141091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:08:02.026 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:02.025+0000 7f311adb7700 1 --2- 172.21.15.160:0/2772611338 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f31140f51a0 0x7f31141091b0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f3104013510 tx=0x7f310400b040).stop 2023-12-25T20:08:02.026 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:02.025+0000 7f311adb7700 1 -- 172.21.15.160:0/2772611338 shutdown_connections 2023-12-25T20:08:02.026 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:02.025+0000 7f311adb7700 1 --2- 172.21.15.160:0/2772611338 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f310002ff60 0x7f3100032420 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:08:02.026 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:02.025+0000 7f311adb7700 1 --2- 172.21.15.160:0/2772611338 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f31140f51a0 0x7f31141091b0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:08:02.026 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:02.025+0000 7f311adb7700 1 -- 172.21.15.160:0/2772611338 >> 172.21.15.160:0/2772611338 conn(0x7f31140f09a0 msgr2=0x7f31140f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:08:02.026 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:02.025+0000 7f311adb7700 1 -- 172.21.15.160:0/2772611338 shutdown_connections 2023-12-25T20:08:02.026 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:02.025+0000 7f311adb7700 1 -- 172.21.15.160:0/2772611338 wait complete. 2023-12-25T20:08:02.027 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:08:02.469 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:08:02 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:08:00.628250+0000 mgr.smithi050.nbzhun (mgr.14162) 24 : cluster [DBG] pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:08:02.469 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:08:02 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:08:02.024903+0000 mon.smithi050 (mon.0) 125 : audit [DBG] from='client.? 172.21.15.160:0/2772611338' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:08:03.411 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:08:03.411 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:08:04.468 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:08:04 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:08:02.628608+0000 mgr.smithi050.nbzhun (mgr.14162) 25 : cluster [DBG] pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:08:04.657 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:04.654+0000 7f4b0d609700 1 -- 172.21.15.160:0/1553392395 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4b080f6510 msgr2=0x7f4b080f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:08:04.657 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:04.654+0000 7f4b0d609700 1 --2- 172.21.15.160:0/1553392395 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4b080f6510 0x7f4b080f6930 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f4af8009fb0 tx=0x7f4af8009b30).stop 2023-12-25T20:08:04.658 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:04.654+0000 7f4b0d609700 1 -- 172.21.15.160:0/1553392395 shutdown_connections 2023-12-25T20:08:04.658 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:04.654+0000 7f4b0d609700 1 --2- 172.21.15.160:0/1553392395 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4b080f6510 0x7f4b080f6930 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:08:04.658 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:04.654+0000 7f4b0d609700 1 -- 172.21.15.160:0/1553392395 >> 172.21.15.160:0/1553392395 conn(0x7f4b080f1c70 msgr2=0x7f4b080f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:08:04.658 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:04.654+0000 7f4b0d609700 1 -- 172.21.15.160:0/1553392395 shutdown_connections 2023-12-25T20:08:04.658 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:04.654+0000 7f4b0d609700 1 -- 172.21.15.160:0/1553392395 wait complete. 2023-12-25T20:08:04.658 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:04.655+0000 7f4b0d609700 1 Processor -- start 2023-12-25T20:08:04.658 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:04.655+0000 7f4b0d609700 1 -- start start 2023-12-25T20:08:04.658 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:04.655+0000 7f4b0d609700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4b080f6510 0x7f4b08106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:08:04.658 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:04.655+0000 7f4b0d609700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b08106c20 con 0x7f4b080f6510 2023-12-25T20:08:04.658 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:04.655+0000 7f4b06ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4b080f6510 0x7f4b08106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:08:04.658 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:04.655+0000 7f4b06ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4b080f6510 0x7f4b08106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:37862/0 (socket says 172.21.15.160:37862) 2023-12-25T20:08:04.658 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:04.655+0000 7f4b06ffd700 1 -- 172.21.15.160:0/3978156253 learned_addr learned my addr 172.21.15.160:0/3978156253 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:08:04.659 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:04.656+0000 7f4b06ffd700 1 -- 172.21.15.160:0/3978156253 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4af8005040 con 0x7f4b080f6510 2023-12-25T20:08:04.659 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:04.656+0000 7f4b06ffd700 1 --2- 172.21.15.160:0/3978156253 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4b080f6510 0x7f4b08106730 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f4af8013510 tx=0x7f4af800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:08:04.659 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:04.656+0000 7f4aff7fe700 1 -- 172.21.15.160:0/3978156253 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4af8009320 con 0x7f4b080f6510 2023-12-25T20:08:04.659 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:04.656+0000 7f4b0d609700 1 -- 172.21.15.160:0/3978156253 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b08106e20 con 0x7f4b080f6510 2023-12-25T20:08:04.659 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:04.656+0000 7f4aff7fe700 1 -- 172.21.15.160:0/3978156253 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4af8013f00 con 0x7f4b080f6510 2023-12-25T20:08:04.659 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:04.656+0000 7f4aff7fe700 1 -- 172.21.15.160:0/3978156253 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4af8014d90 con 0x7f4b080f6510 2023-12-25T20:08:04.659 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:04.656+0000 7f4b0d609700 1 -- 172.21.15.160:0/3978156253 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b081072b0 con 0x7f4b080f6510 2023-12-25T20:08:04.660 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:04.657+0000 7f4aff7fe700 1 -- 172.21.15.160:0/3978156253 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4af80263a0 con 0x7f4b080f6510 2023-12-25T20:08:04.660 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:04.657+0000 7f4aff7fe700 1 --2- 172.21.15.160:0/3978156253 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f4af402ff60 0x7f4af4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:08:04.660 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:04.657+0000 7f4aff7fe700 1 -- 172.21.15.160:0/3978156253 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4af80412c0 con 0x7f4b080f6510 2023-12-25T20:08:04.660 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:04.657+0000 7f4b0d609700 1 -- 172.21.15.160:0/3978156253 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b08045ad0 con 0x7f4b080f6510 2023-12-25T20:08:04.660 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:04.657+0000 7f4b067fc700 1 --2- 172.21.15.160:0/3978156253 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f4af402ff60 0x7f4af4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:08:04.661 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:04.658+0000 7f4b067fc700 1 --2- 172.21.15.160:0/3978156253 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f4af402ff60 0x7f4af4032420 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f4af00096a0 tx=0x7f4af0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:08:04.662 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:04.662+0000 7f4aff7fe700 1 -- 172.21.15.160:0/3978156253 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4af8012920 con 0x7f4b080f6510 2023-12-25T20:08:04.866 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:04.865+0000 7f4b0d609700 1 -- 172.21.15.160:0/3978156253 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4b080fc050 con 0x7f4b080f6510 2023-12-25T20:08:04.867 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:04.866+0000 7f4aff7fe700 1 -- 172.21.15.160:0/3978156253 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4af800f340 con 0x7f4b080f6510 2023-12-25T20:08:04.867 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:08:04.867 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:08:04.869 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:04.868+0000 7f4b0d609700 1 -- 172.21.15.160:0/3978156253 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f4af402ff60 msgr2=0x7f4af4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:08:04.869 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:04.868+0000 7f4b0d609700 1 --2- 172.21.15.160:0/3978156253 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f4af402ff60 0x7f4af4032420 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f4af00096a0 tx=0x7f4af0006b40).stop 2023-12-25T20:08:04.869 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:04.868+0000 7f4b0d609700 1 -- 172.21.15.160:0/3978156253 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4b080f6510 msgr2=0x7f4b08106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:08:04.869 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:04.868+0000 7f4b0d609700 1 --2- 172.21.15.160:0/3978156253 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4b080f6510 0x7f4b08106730 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f4af8013510 tx=0x7f4af800b040).stop 2023-12-25T20:08:04.870 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:04.868+0000 7f4b0d609700 1 -- 172.21.15.160:0/3978156253 shutdown_connections 2023-12-25T20:08:04.870 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:04.868+0000 7f4b0d609700 1 --2- 172.21.15.160:0/3978156253 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f4af402ff60 0x7f4af4032420 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:08:04.870 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:04.868+0000 7f4b0d609700 1 --2- 172.21.15.160:0/3978156253 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4b080f6510 0x7f4b08106730 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:08:04.870 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:04.868+0000 7f4b0d609700 1 -- 172.21.15.160:0/3978156253 >> 172.21.15.160:0/3978156253 conn(0x7f4b080f1c70 msgr2=0x7f4b080f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:08:04.870 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:04.868+0000 7f4b0d609700 1 -- 172.21.15.160:0/3978156253 shutdown_connections 2023-12-25T20:08:04.870 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:04.868+0000 7f4b0d609700 1 -- 172.21.15.160:0/3978156253 wait complete. 2023-12-25T20:08:04.870 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:08:05.468 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:08:05 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:08:04.867763+0000 mon.smithi050 (mon.0) 126 : audit [DBG] from='client.? 172.21.15.160:0/3978156253' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:08:06.277 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:08:06.278 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:08:06.469 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:08:06 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:08:04.628903+0000 mgr.smithi050.nbzhun (mgr.14162) 26 : cluster [DBG] pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:08:07.516 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:07.514+0000 7f7825c21700 1 -- 172.21.15.160:0/3323934518 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f78200f5410 msgr2=0x7f78200f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:08:07.516 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:07.514+0000 7f7825c21700 1 --2- 172.21.15.160:0/3323934518 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f78200f5410 0x7f78200f5830 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f7814009fb0 tx=0x7f7814009b30).stop 2023-12-25T20:08:07.516 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:07.515+0000 7f7825c21700 1 -- 172.21.15.160:0/3323934518 shutdown_connections 2023-12-25T20:08:07.516 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:07.515+0000 7f7825c21700 1 --2- 172.21.15.160:0/3323934518 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f78200f5410 0x7f78200f5830 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:08:07.516 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:07.515+0000 7f7825c21700 1 -- 172.21.15.160:0/3323934518 >> 172.21.15.160:0/3323934518 conn(0x7f78200f0b70 msgr2=0x7f78200f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:08:07.516 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:07.515+0000 7f7825c21700 1 -- 172.21.15.160:0/3323934518 shutdown_connections 2023-12-25T20:08:07.517 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:07.515+0000 7f7825c21700 1 -- 172.21.15.160:0/3323934518 wait complete. 2023-12-25T20:08:07.517 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:07.515+0000 7f7825c21700 1 Processor -- start 2023-12-25T20:08:07.517 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:07.515+0000 7f7825c21700 1 -- start start 2023-12-25T20:08:07.517 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:07.515+0000 7f7825c21700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f78200f5410 0x7f7820105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:08:07.517 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:07.515+0000 7f7825c21700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7820105b20 con 0x7f78200f5410 2023-12-25T20:08:07.518 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:07.516+0000 7f781f7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f78200f5410 0x7f7820105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:08:07.518 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:07.516+0000 7f781f7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f78200f5410 0x7f7820105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:48056/0 (socket says 172.21.15.160:48056) 2023-12-25T20:08:07.518 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:07.516+0000 7f781f7fe700 1 -- 172.21.15.160:0/347390939 learned_addr learned my addr 172.21.15.160:0/347390939 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:08:07.518 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:07.516+0000 7f781f7fe700 1 -- 172.21.15.160:0/347390939 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7814005040 con 0x7f78200f5410 2023-12-25T20:08:07.518 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:07.516+0000 7f781f7fe700 1 --2- 172.21.15.160:0/347390939 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f78200f5410 0x7f7820105630 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f7814013510 tx=0x7f781400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:08:07.519 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:07.516+0000 7f7807fff700 1 -- 172.21.15.160:0/347390939 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7814009320 con 0x7f78200f5410 2023-12-25T20:08:07.519 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:07.516+0000 7f7825c21700 1 -- 172.21.15.160:0/347390939 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7820105d20 con 0x7f78200f5410 2023-12-25T20:08:07.519 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:07.516+0000 7f7807fff700 1 -- 172.21.15.160:0/347390939 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7814013f00 con 0x7f78200f5410 2023-12-25T20:08:07.519 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:07.516+0000 7f7807fff700 1 -- 172.21.15.160:0/347390939 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7814014d90 con 0x7f78200f5410 2023-12-25T20:08:07.519 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:07.516+0000 7f7825c21700 1 -- 172.21.15.160:0/347390939 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78201061b0 con 0x7f78200f5410 2023-12-25T20:08:07.519 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:07.517+0000 7f7807fff700 1 -- 172.21.15.160:0/347390939 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7814014420 con 0x7f78200f5410 2023-12-25T20:08:07.519 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:07.517+0000 7f7825c21700 1 -- 172.21.15.160:0/347390939 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7820045ad0 con 0x7f78200f5410 2023-12-25T20:08:07.519 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:07.517+0000 7f7807fff700 1 --2- 172.21.15.160:0/347390939 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f780802ff60 0x7f7808032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:08:07.519 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:07.518+0000 7f7807fff700 1 -- 172.21.15.160:0/347390939 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f78140410e0 con 0x7f78200f5410 2023-12-25T20:08:07.519 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:07.518+0000 7f781effd700 1 --2- 172.21.15.160:0/347390939 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f780802ff60 0x7f7808032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:08:07.519 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:07.518+0000 7f781effd700 1 --2- 172.21.15.160:0/347390939 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f780802ff60 0x7f7808032420 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f78100096a0 tx=0x7f7810006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:08:07.523 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:07.522+0000 7f7807fff700 1 -- 172.21.15.160:0/347390939 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f78140095f0 con 0x7f78200f5410 2023-12-25T20:08:07.729 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:07.729+0000 7f7825c21700 1 -- 172.21.15.160:0/347390939 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f78200fae60 con 0x7f78200f5410 2023-12-25T20:08:07.730 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:07.729+0000 7f7807fff700 1 -- 172.21.15.160:0/347390939 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f781401db70 con 0x7f78200f5410 2023-12-25T20:08:07.730 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:08:07.730 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:08:07.732 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:07.731+0000 7f7825c21700 1 -- 172.21.15.160:0/347390939 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f780802ff60 msgr2=0x7f7808032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:08:07.732 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:07.731+0000 7f7825c21700 1 --2- 172.21.15.160:0/347390939 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f780802ff60 0x7f7808032420 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f78100096a0 tx=0x7f7810006b40).stop 2023-12-25T20:08:07.732 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:07.731+0000 7f7825c21700 1 -- 172.21.15.160:0/347390939 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f78200f5410 msgr2=0x7f7820105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:08:07.732 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:07.731+0000 7f7825c21700 1 --2- 172.21.15.160:0/347390939 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f78200f5410 0x7f7820105630 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f7814013510 tx=0x7f781400b040).stop 2023-12-25T20:08:07.732 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:07.731+0000 7f7825c21700 1 -- 172.21.15.160:0/347390939 shutdown_connections 2023-12-25T20:08:07.732 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:07.731+0000 7f7825c21700 1 --2- 172.21.15.160:0/347390939 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f780802ff60 0x7f7808032420 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:08:07.732 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:07.731+0000 7f7825c21700 1 --2- 172.21.15.160:0/347390939 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f78200f5410 0x7f7820105630 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:08:07.732 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:07.731+0000 7f7825c21700 1 -- 172.21.15.160:0/347390939 >> 172.21.15.160:0/347390939 conn(0x7f78200f0b70 msgr2=0x7f78200f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:08:07.732 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:07.731+0000 7f7825c21700 1 -- 172.21.15.160:0/347390939 shutdown_connections 2023-12-25T20:08:07.733 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:07.731+0000 7f7825c21700 1 -- 172.21.15.160:0/347390939 wait complete. 2023-12-25T20:08:07.733 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:08:08.469 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:08:08 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:08:06.629206+0000 mgr.smithi050.nbzhun (mgr.14162) 27 : cluster [DBG] pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:08:08.469 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:08:08 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:08:07.730980+0000 mon.smithi050 (mon.0) 127 : audit [DBG] from='client.? 172.21.15.160:0/347390939' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:08:09.177 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:08:09.177 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:08:10.469 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:08:10 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:08:08.629487+0000 mgr.smithi050.nbzhun (mgr.14162) 28 : cluster [DBG] pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:08:11.545 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:11.542+0000 7f310c4a5700 1 -- 172.21.15.160:0/1860770866 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f31040f6510 msgr2=0x7f31040f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:08:11.545 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:11.542+0000 7f310c4a5700 1 --2- 172.21.15.160:0/1860770866 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f31040f6510 0x7f31040f6930 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f3100009fb0 tx=0x7f3100009b30).stop 2023-12-25T20:08:11.545 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:11.542+0000 7f310c4a5700 1 -- 172.21.15.160:0/1860770866 shutdown_connections 2023-12-25T20:08:11.545 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:11.542+0000 7f310c4a5700 1 --2- 172.21.15.160:0/1860770866 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f31040f6510 0x7f31040f6930 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:08:11.545 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:11.542+0000 7f310c4a5700 1 -- 172.21.15.160:0/1860770866 >> 172.21.15.160:0/1860770866 conn(0x7f31040f1c70 msgr2=0x7f31040f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:08:11.546 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:11.542+0000 7f310c4a5700 1 -- 172.21.15.160:0/1860770866 shutdown_connections 2023-12-25T20:08:11.546 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:11.542+0000 7f310c4a5700 1 -- 172.21.15.160:0/1860770866 wait complete. 2023-12-25T20:08:11.546 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:11.543+0000 7f310c4a5700 1 Processor -- start 2023-12-25T20:08:11.546 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:11.543+0000 7f310c4a5700 1 -- start start 2023-12-25T20:08:11.546 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:11.543+0000 7f310c4a5700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f31040f6510 0x7f310406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:08:11.546 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:11.543+0000 7f310c4a5700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3104069e20 con 0x7f31040f6510 2023-12-25T20:08:11.546 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:11.543+0000 7f310a241700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f31040f6510 0x7f310406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:08:11.546 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:11.543+0000 7f310a241700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f31040f6510 0x7f310406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:48074/0 (socket says 172.21.15.160:48074) 2023-12-25T20:08:11.546 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:11.543+0000 7f310a241700 1 -- 172.21.15.160:0/909249858 learned_addr learned my addr 172.21.15.160:0/909249858 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:08:11.546 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:11.544+0000 7f310a241700 1 -- 172.21.15.160:0/909249858 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3100005040 con 0x7f31040f6510 2023-12-25T20:08:11.546 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:11.544+0000 7f310a241700 1 --2- 172.21.15.160:0/909249858 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f31040f6510 0x7f310406b9a0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f3100013510 tx=0x7f310000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:08:11.546 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:11.544+0000 7f30faffd700 1 -- 172.21.15.160:0/909249858 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3100009320 con 0x7f31040f6510 2023-12-25T20:08:11.546 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:11.544+0000 7f310c4a5700 1 -- 172.21.15.160:0/909249858 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3104069fc0 con 0x7f31040f6510 2023-12-25T20:08:11.546 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:11.544+0000 7f30faffd700 1 -- 172.21.15.160:0/909249858 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3100013f00 con 0x7f31040f6510 2023-12-25T20:08:11.547 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:11.544+0000 7f30faffd700 1 -- 172.21.15.160:0/909249858 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3100014d90 con 0x7f31040f6510 2023-12-25T20:08:11.547 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:11.544+0000 7f310c4a5700 1 -- 172.21.15.160:0/909249858 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f310406a450 con 0x7f31040f6510 2023-12-25T20:08:11.547 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:11.545+0000 7f30faffd700 1 -- 172.21.15.160:0/909249858 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f31000095f0 con 0x7f31040f6510 2023-12-25T20:08:11.547 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:11.545+0000 7f310c4a5700 1 -- 172.21.15.160:0/909249858 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3104059100 con 0x7f31040f6510 2023-12-25T20:08:11.548 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:11.545+0000 7f30faffd700 1 --2- 172.21.15.160:0/909249858 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f30f002ff60 0x7f30f0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:08:11.548 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:11.545+0000 7f30faffd700 1 -- 172.21.15.160:0/909249858 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3100041000 con 0x7f31040f6510 2023-12-25T20:08:11.548 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:11.545+0000 7f3109a40700 1 --2- 172.21.15.160:0/909249858 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f30f002ff60 0x7f30f0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:08:11.548 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:11.546+0000 7f3109a40700 1 --2- 172.21.15.160:0/909249858 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f30f002ff60 0x7f30f0032420 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f30f40096a0 tx=0x7f30f4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:08:11.550 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:11.550+0000 7f30faffd700 1 -- 172.21.15.160:0/909249858 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3100014420 con 0x7f31040f6510 2023-12-25T20:08:11.754 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:11.754+0000 7f310c4a5700 1 -- 172.21.15.160:0/909249858 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f31040fbf60 con 0x7f31040f6510 2023-12-25T20:08:11.755 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:11.754+0000 7f30faffd700 1 -- 172.21.15.160:0/909249858 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3100010230 con 0x7f31040f6510 2023-12-25T20:08:11.756 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:08:11.756 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:08:11.757 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:11.756+0000 7f310c4a5700 1 -- 172.21.15.160:0/909249858 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f30f002ff60 msgr2=0x7f30f0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:08:11.758 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:11.756+0000 7f310c4a5700 1 --2- 172.21.15.160:0/909249858 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f30f002ff60 0x7f30f0032420 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f30f40096a0 tx=0x7f30f4006b40).stop 2023-12-25T20:08:11.758 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:11.756+0000 7f310c4a5700 1 -- 172.21.15.160:0/909249858 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f31040f6510 msgr2=0x7f310406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:08:11.758 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:11.756+0000 7f310c4a5700 1 --2- 172.21.15.160:0/909249858 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f31040f6510 0x7f310406b9a0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f3100013510 tx=0x7f310000b040).stop 2023-12-25T20:08:11.758 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:11.756+0000 7f310c4a5700 1 -- 172.21.15.160:0/909249858 shutdown_connections 2023-12-25T20:08:11.758 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:11.756+0000 7f310c4a5700 1 --2- 172.21.15.160:0/909249858 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f30f002ff60 0x7f30f0032420 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:08:11.758 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:11.756+0000 7f310c4a5700 1 --2- 172.21.15.160:0/909249858 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f31040f6510 0x7f310406b9a0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:08:11.758 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:11.756+0000 7f310c4a5700 1 -- 172.21.15.160:0/909249858 >> 172.21.15.160:0/909249858 conn(0x7f31040f1c70 msgr2=0x7f31040f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:08:11.758 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:11.756+0000 7f310c4a5700 1 -- 172.21.15.160:0/909249858 shutdown_connections 2023-12-25T20:08:11.758 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:11.756+0000 7f310c4a5700 1 -- 172.21.15.160:0/909249858 wait complete. 2023-12-25T20:08:11.758 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:08:12.468 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:08:12 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:08:10.629743+0000 mgr.smithi050.nbzhun (mgr.14162) 29 : cluster [DBG] pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:08:12.469 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:08:12 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:08:11.756106+0000 mon.smithi050 (mon.0) 128 : audit [DBG] from='client.? 172.21.15.160:0/909249858' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:08:13.174 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:08:13.174 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:08:14.386 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:14.384+0000 7f31bad12700 1 -- 172.21.15.160:0/3662917695 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f31b40f62a0 msgr2=0x7f31b40f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:08:14.387 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:14.384+0000 7f31bad12700 1 --2- 172.21.15.160:0/3662917695 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f31b40f62a0 0x7f31b40f66c0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f31a4009fb0 tx=0x7f31a4009b30).stop 2023-12-25T20:08:14.387 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:14.384+0000 7f31bad12700 1 -- 172.21.15.160:0/3662917695 shutdown_connections 2023-12-25T20:08:14.387 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:14.384+0000 7f31bad12700 1 --2- 172.21.15.160:0/3662917695 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f31b40f62a0 0x7f31b40f66c0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:08:14.387 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:14.384+0000 7f31bad12700 1 -- 172.21.15.160:0/3662917695 >> 172.21.15.160:0/3662917695 conn(0x7f31b40f1aa0 msgr2=0x7f31b40f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:08:14.387 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:14.384+0000 7f31bad12700 1 -- 172.21.15.160:0/3662917695 shutdown_connections 2023-12-25T20:08:14.387 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:14.384+0000 7f31bad12700 1 -- 172.21.15.160:0/3662917695 wait complete. 2023-12-25T20:08:14.387 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:14.385+0000 7f31bad12700 1 Processor -- start 2023-12-25T20:08:14.387 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:14.385+0000 7f31bad12700 1 -- start start 2023-12-25T20:08:14.387 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:14.385+0000 7f31bad12700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f31b40f62a0 0x7f31b41064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:08:14.387 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:14.385+0000 7f31bad12700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31b41069a0 con 0x7f31b40f62a0 2023-12-25T20:08:14.387 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:14.385+0000 7f31b9d10700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f31b40f62a0 0x7f31b41064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:08:14.387 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:14.385+0000 7f31b9d10700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f31b40f62a0 0x7f31b41064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:48100/0 (socket says 172.21.15.160:48100) 2023-12-25T20:08:14.388 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:14.385+0000 7f31b9d10700 1 -- 172.21.15.160:0/1702320822 learned_addr learned my addr 172.21.15.160:0/1702320822 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:08:14.388 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:14.385+0000 7f31b9d10700 1 -- 172.21.15.160:0/1702320822 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31a4005040 con 0x7f31b40f62a0 2023-12-25T20:08:14.388 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:14.386+0000 7f31b9d10700 1 --2- 172.21.15.160:0/1702320822 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f31b40f62a0 0x7f31b41064b0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f31a4013510 tx=0x7f31a400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:08:14.388 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:14.386+0000 7f31aa7fc700 1 -- 172.21.15.160:0/1702320822 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f31a4009320 con 0x7f31b40f62a0 2023-12-25T20:08:14.388 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:14.386+0000 7f31bad12700 1 -- 172.21.15.160:0/1702320822 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31b4106ba0 con 0x7f31b40f62a0 2023-12-25T20:08:14.388 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:14.386+0000 7f31aa7fc700 1 -- 172.21.15.160:0/1702320822 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f31a4013f00 con 0x7f31b40f62a0 2023-12-25T20:08:14.388 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:14.386+0000 7f31aa7fc700 1 -- 172.21.15.160:0/1702320822 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f31a4014dc0 con 0x7f31b40f62a0 2023-12-25T20:08:14.388 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:14.386+0000 7f31bad12700 1 -- 172.21.15.160:0/1702320822 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31b4107030 con 0x7f31b40f62a0 2023-12-25T20:08:14.389 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:14.387+0000 7f31aa7fc700 1 -- 172.21.15.160:0/1702320822 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f31a40095f0 con 0x7f31b40f62a0 2023-12-25T20:08:14.389 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:14.387+0000 7f31bad12700 1 -- 172.21.15.160:0/1702320822 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31b4045ad0 con 0x7f31b40f62a0 2023-12-25T20:08:14.389 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:14.387+0000 7f31aa7fc700 1 --2- 172.21.15.160:0/1702320822 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f31a002ff60 0x7f31a0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:08:14.389 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:14.387+0000 7f31aa7fc700 1 -- 172.21.15.160:0/1702320822 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f31a4041000 con 0x7f31b40f62a0 2023-12-25T20:08:14.389 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:14.387+0000 7f31b950f700 1 --2- 172.21.15.160:0/1702320822 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f31a002ff60 0x7f31a0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:08:14.389 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:14.387+0000 7f31b950f700 1 --2- 172.21.15.160:0/1702320822 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f31a002ff60 0x7f31a0032420 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f31b00096a0 tx=0x7f31b0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:08:14.392 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:14.391+0000 7f31aa7fc700 1 -- 172.21.15.160:0/1702320822 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f31a4014420 con 0x7f31b40f62a0 2023-12-25T20:08:14.469 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:08:14 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:08:12.630143+0000 mgr.smithi050.nbzhun (mgr.14162) 30 : cluster [DBG] pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:08:14.597 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:14.596+0000 7f31bad12700 1 -- 172.21.15.160:0/1702320822 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f31b4107ee0 con 0x7f31b40f62a0 2023-12-25T20:08:14.598 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:14.597+0000 7f31aa7fc700 1 -- 172.21.15.160:0/1702320822 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f31a4010230 con 0x7f31b40f62a0 2023-12-25T20:08:14.598 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:08:14.598 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:08:14.599 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:14.598+0000 7f31bad12700 1 -- 172.21.15.160:0/1702320822 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f31a002ff60 msgr2=0x7f31a0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:08:14.599 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:14.598+0000 7f31bad12700 1 --2- 172.21.15.160:0/1702320822 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f31a002ff60 0x7f31a0032420 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f31b00096a0 tx=0x7f31b0006b40).stop 2023-12-25T20:08:14.600 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:14.598+0000 7f31bad12700 1 -- 172.21.15.160:0/1702320822 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f31b40f62a0 msgr2=0x7f31b41064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:08:14.600 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:14.598+0000 7f31bad12700 1 --2- 172.21.15.160:0/1702320822 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f31b40f62a0 0x7f31b41064b0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f31a4013510 tx=0x7f31a400b040).stop 2023-12-25T20:08:14.600 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:14.598+0000 7f31bad12700 1 -- 172.21.15.160:0/1702320822 shutdown_connections 2023-12-25T20:08:14.600 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:14.598+0000 7f31bad12700 1 --2- 172.21.15.160:0/1702320822 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f31a002ff60 0x7f31a0032420 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:08:14.600 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:14.598+0000 7f31bad12700 1 --2- 172.21.15.160:0/1702320822 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f31b40f62a0 0x7f31b41064b0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:08:14.600 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:14.598+0000 7f31bad12700 1 -- 172.21.15.160:0/1702320822 >> 172.21.15.160:0/1702320822 conn(0x7f31b40f1aa0 msgr2=0x7f31b40f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:08:14.600 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:14.598+0000 7f31bad12700 1 -- 172.21.15.160:0/1702320822 shutdown_connections 2023-12-25T20:08:14.600 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:14.598+0000 7f31bad12700 1 -- 172.21.15.160:0/1702320822 wait complete. 2023-12-25T20:08:14.600 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:08:15.469 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:08:15 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:08:14.598509+0000 mon.smithi050 (mon.0) 129 : audit [DBG] from='client.? 172.21.15.160:0/1702320822' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:08:15.469 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:08:15 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:08:14.761500+0000 mon.smithi050 (mon.0) 130 : audit [INF] from='mgr.14162 172.21.15.50:0/3178367813' entity='mgr.smithi050.nbzhun' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi050.nbzhun/mirror_snapshot_schedule"}]: dispatch 2023-12-25T20:08:15.469 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:08:15 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:08:14.763558+0000 mon.smithi050 (mon.0) 131 : audit [INF] from='mgr.14162 172.21.15.50:0/3178367813' entity='mgr.smithi050.nbzhun' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi050.nbzhun/trash_purge_schedule"}]: dispatch 2023-12-25T20:08:15.998 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:08:15.998 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:08:16.468 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:08:16 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:08:14.630496+0000 mgr.smithi050.nbzhun (mgr.14162) 31 : cluster [DBG] pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:08:18.469 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:08:18 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:08:16.630780+0000 mgr.smithi050.nbzhun (mgr.14162) 32 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:08:20.132 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:20.130+0000 7f9b8c918700 1 -- 172.21.15.160:0/2725976684 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9b880f57b0 msgr2=0x7f9b880f5bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:08:20.132 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:20.130+0000 7f9b8c918700 1 --2- 172.21.15.160:0/2725976684 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9b880f57b0 0x7f9b880f5bd0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f9b78009fb0 tx=0x7f9b78009b30).stop 2023-12-25T20:08:20.132 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:20.130+0000 7f9b8c918700 1 -- 172.21.15.160:0/2725976684 shutdown_connections 2023-12-25T20:08:20.132 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:20.130+0000 7f9b8c918700 1 --2- 172.21.15.160:0/2725976684 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9b880f57b0 0x7f9b880f5bd0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:08:20.132 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:20.130+0000 7f9b8c918700 1 -- 172.21.15.160:0/2725976684 >> 172.21.15.160:0/2725976684 conn(0x7f9b880f0120 msgr2=0x7f9b880f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:08:20.132 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:20.131+0000 7f9b8c918700 1 -- 172.21.15.160:0/2725976684 shutdown_connections 2023-12-25T20:08:20.133 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:20.131+0000 7f9b8c918700 1 -- 172.21.15.160:0/2725976684 wait complete. 2023-12-25T20:08:20.133 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:20.131+0000 7f9b8c918700 1 Processor -- start 2023-12-25T20:08:20.133 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:20.131+0000 7f9b8c918700 1 -- start start 2023-12-25T20:08:20.133 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:20.131+0000 7f9b8c918700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9b880f57b0 0x7f9b881091e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:08:20.133 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:20.131+0000 7f9b8c918700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b880f5170 con 0x7f9b880f57b0 2023-12-25T20:08:20.134 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:20.132+0000 7f9b86d9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9b880f57b0 0x7f9b881091e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:08:20.135 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:20.132+0000 7f9b86d9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9b880f57b0 0x7f9b881091e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:39386/0 (socket says 172.21.15.160:39386) 2023-12-25T20:08:20.135 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:20.132+0000 7f9b86d9d700 1 -- 172.21.15.160:0/736913986 learned_addr learned my addr 172.21.15.160:0/736913986 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:08:20.135 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:20.132+0000 7f9b86d9d700 1 -- 172.21.15.160:0/736913986 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b78005040 con 0x7f9b880f57b0 2023-12-25T20:08:20.135 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:20.132+0000 7f9b86d9d700 1 --2- 172.21.15.160:0/736913986 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9b880f57b0 0x7f9b881091e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f9b78013510 tx=0x7f9b7800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:08:20.135 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:20.132+0000 7f9b7f7fe700 1 -- 172.21.15.160:0/736913986 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9b78009320 con 0x7f9b880f57b0 2023-12-25T20:08:20.135 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:20.132+0000 7f9b8c918700 1 -- 172.21.15.160:0/736913986 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b880f5370 con 0x7f9b880f57b0 2023-12-25T20:08:20.135 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:20.132+0000 7f9b7f7fe700 1 -- 172.21.15.160:0/736913986 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9b78013f00 con 0x7f9b880f57b0 2023-12-25T20:08:20.135 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:20.132+0000 7f9b7f7fe700 1 -- 172.21.15.160:0/736913986 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9b78014dc0 con 0x7f9b880f57b0 2023-12-25T20:08:20.135 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:20.132+0000 7f9b8c918700 1 -- 172.21.15.160:0/736913986 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b88109890 con 0x7f9b880f57b0 2023-12-25T20:08:20.135 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:20.133+0000 7f9b7f7fe700 1 -- 172.21.15.160:0/736913986 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9b780095f0 con 0x7f9b880f57b0 2023-12-25T20:08:20.135 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:20.133+0000 7f9b8c918700 1 -- 172.21.15.160:0/736913986 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b88059100 con 0x7f9b880f57b0 2023-12-25T20:08:20.135 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:20.133+0000 7f9b7f7fe700 1 --2- 172.21.15.160:0/736913986 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f9b7402ff60 0x7f9b74032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:08:20.135 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:20.133+0000 7f9b7f7fe700 1 -- 172.21.15.160:0/736913986 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9b78041000 con 0x7f9b880f57b0 2023-12-25T20:08:20.136 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:20.133+0000 7f9b8659c700 1 --2- 172.21.15.160:0/736913986 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f9b7402ff60 0x7f9b74032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:08:20.136 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:20.134+0000 7f9b8659c700 1 --2- 172.21.15.160:0/736913986 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f9b7402ff60 0x7f9b74032420 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f9b700096a0 tx=0x7f9b70006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:08:20.138 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:20.138+0000 7f9b7f7fe700 1 -- 172.21.15.160:0/736913986 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9b7801c790 con 0x7f9b880f57b0 2023-12-25T20:08:20.343 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:20.342+0000 7f9b8c918700 1 -- 172.21.15.160:0/736913986 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9b88045ad0 con 0x7f9b880f57b0 2023-12-25T20:08:20.344 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:20.343+0000 7f9b7f7fe700 1 -- 172.21.15.160:0/736913986 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9b78010230 con 0x7f9b880f57b0 2023-12-25T20:08:20.344 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:08:20.344 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:08:20.346 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:20.345+0000 7f9b8c918700 1 -- 172.21.15.160:0/736913986 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f9b7402ff60 msgr2=0x7f9b74032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:08:20.346 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:20.345+0000 7f9b8c918700 1 --2- 172.21.15.160:0/736913986 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f9b7402ff60 0x7f9b74032420 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f9b700096a0 tx=0x7f9b70006b40).stop 2023-12-25T20:08:20.346 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:20.345+0000 7f9b8c918700 1 -- 172.21.15.160:0/736913986 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9b880f57b0 msgr2=0x7f9b881091e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:08:20.346 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:20.345+0000 7f9b8c918700 1 --2- 172.21.15.160:0/736913986 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9b880f57b0 0x7f9b881091e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f9b78013510 tx=0x7f9b7800b040).stop 2023-12-25T20:08:20.346 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:20.345+0000 7f9b8c918700 1 -- 172.21.15.160:0/736913986 shutdown_connections 2023-12-25T20:08:20.346 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:20.345+0000 7f9b8c918700 1 --2- 172.21.15.160:0/736913986 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f9b7402ff60 0x7f9b74032420 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:08:20.346 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:20.345+0000 7f9b8c918700 1 --2- 172.21.15.160:0/736913986 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9b880f57b0 0x7f9b881091e0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:08:20.346 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:20.345+0000 7f9b8c918700 1 -- 172.21.15.160:0/736913986 >> 172.21.15.160:0/736913986 conn(0x7f9b880f0120 msgr2=0x7f9b880f0c40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:08:20.346 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:20.345+0000 7f9b8c918700 1 -- 172.21.15.160:0/736913986 shutdown_connections 2023-12-25T20:08:20.346 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:20.345+0000 7f9b8c918700 1 -- 172.21.15.160:0/736913986 wait complete. 2023-12-25T20:08:20.347 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:08:20.468 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:08:20 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:08:18.631110+0000 mgr.smithi050.nbzhun (mgr.14162) 33 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:08:21.469 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:08:21 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:08:20.344742+0000 mon.smithi050 (mon.0) 132 : audit [DBG] from='client.? 172.21.15.160:0/736913986' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:08:21.769 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:08:21.769 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:08:22.468 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:08:22 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:08:20.631474+0000 mgr.smithi050.nbzhun (mgr.14162) 34 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:08:22.921 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:22.920+0000 7ff85aab6700 1 -- 172.21.15.160:0/3396904860 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff8540f51a0 msgr2=0x7ff8540f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:08:22.921 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:22.920+0000 7ff85aab6700 1 --2- 172.21.15.160:0/3396904860 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff8540f51a0 0x7ff8540f55c0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7ff850009fb0 tx=0x7ff850009b30).stop 2023-12-25T20:08:22.921 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:22.920+0000 7ff85aab6700 1 -- 172.21.15.160:0/3396904860 shutdown_connections 2023-12-25T20:08:22.921 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:22.920+0000 7ff85aab6700 1 --2- 172.21.15.160:0/3396904860 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff8540f51a0 0x7ff8540f55c0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:08:22.922 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:22.920+0000 7ff85aab6700 1 -- 172.21.15.160:0/3396904860 >> 172.21.15.160:0/3396904860 conn(0x7ff8540f09a0 msgr2=0x7ff8540f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:08:22.922 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:22.920+0000 7ff85aab6700 1 -- 172.21.15.160:0/3396904860 shutdown_connections 2023-12-25T20:08:22.922 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:22.920+0000 7ff85aab6700 1 -- 172.21.15.160:0/3396904860 wait complete. 2023-12-25T20:08:22.922 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:22.920+0000 7ff85aab6700 1 Processor -- start 2023-12-25T20:08:22.922 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:22.920+0000 7ff85aab6700 1 -- start start 2023-12-25T20:08:22.922 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:22.920+0000 7ff85aab6700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff8540f51a0 0x7ff8541064c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:08:22.922 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:22.920+0000 7ff85aab6700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8541069b0 con 0x7ff8540f51a0 2023-12-25T20:08:22.923 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:22.921+0000 7ff859ab4700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff8540f51a0 0x7ff8541064c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:08:22.923 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:22.921+0000 7ff859ab4700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff8540f51a0 0x7ff8541064c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:39402/0 (socket says 172.21.15.160:39402) 2023-12-25T20:08:22.923 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:22.921+0000 7ff859ab4700 1 -- 172.21.15.160:0/1920957830 learned_addr learned my addr 172.21.15.160:0/1920957830 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:08:22.924 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:22.921+0000 7ff859ab4700 1 -- 172.21.15.160:0/1920957830 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff850005040 con 0x7ff8540f51a0 2023-12-25T20:08:22.924 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:22.921+0000 7ff859ab4700 1 --2- 172.21.15.160:0/1920957830 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff8540f51a0 0x7ff8541064c0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7ff850013510 tx=0x7ff85000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:08:22.924 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:22.921+0000 7ff84a7fc700 1 -- 172.21.15.160:0/1920957830 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff850009320 con 0x7ff8540f51a0 2023-12-25T20:08:22.924 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:22.921+0000 7ff85aab6700 1 -- 172.21.15.160:0/1920957830 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff854106bb0 con 0x7ff8540f51a0 2023-12-25T20:08:22.924 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:22.921+0000 7ff84a7fc700 1 -- 172.21.15.160:0/1920957830 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff850013f00 con 0x7ff8540f51a0 2023-12-25T20:08:22.924 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:22.922+0000 7ff84a7fc700 1 -- 172.21.15.160:0/1920957830 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff850014dc0 con 0x7ff8540f51a0 2023-12-25T20:08:22.924 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:22.922+0000 7ff85aab6700 1 -- 172.21.15.160:0/1920957830 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff854107040 con 0x7ff8540f51a0 2023-12-25T20:08:22.924 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:22.922+0000 7ff84a7fc700 1 -- 172.21.15.160:0/1920957830 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff850014420 con 0x7ff8540f51a0 2023-12-25T20:08:22.924 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:22.922+0000 7ff85aab6700 1 -- 172.21.15.160:0/1920957830 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff854045ad0 con 0x7ff8540f51a0 2023-12-25T20:08:22.924 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:22.922+0000 7ff84a7fc700 1 --2- 172.21.15.160:0/1920957830 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7ff84002ff60 0x7ff840032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:08:22.925 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:22.922+0000 7ff84a7fc700 1 -- 172.21.15.160:0/1920957830 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff850041050 con 0x7ff8540f51a0 2023-12-25T20:08:22.925 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:22.923+0000 7ff8592b3700 1 --2- 172.21.15.160:0/1920957830 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7ff84002ff60 0x7ff840032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:08:22.925 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:22.923+0000 7ff8592b3700 1 --2- 172.21.15.160:0/1920957830 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7ff84002ff60 0x7ff840032420 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7ff8440096a0 tx=0x7ff844006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:08:22.928 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:22.927+0000 7ff84a7fc700 1 -- 172.21.15.160:0/1920957830 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff8500095f0 con 0x7ff8540f51a0 2023-12-25T20:08:23.133 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:23.132+0000 7ff85aab6700 1 -- 172.21.15.160:0/1920957830 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff854107ef0 con 0x7ff8540f51a0 2023-12-25T20:08:23.133 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:23.133+0000 7ff84a7fc700 1 -- 172.21.15.160:0/1920957830 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff85001cb10 con 0x7ff8540f51a0 2023-12-25T20:08:23.134 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:08:23.134 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:08:23.136 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:23.134+0000 7ff85aab6700 1 -- 172.21.15.160:0/1920957830 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7ff84002ff60 msgr2=0x7ff840032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:08:23.136 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:23.134+0000 7ff85aab6700 1 --2- 172.21.15.160:0/1920957830 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7ff84002ff60 0x7ff840032420 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7ff8440096a0 tx=0x7ff844006b40).stop 2023-12-25T20:08:23.136 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:23.134+0000 7ff85aab6700 1 -- 172.21.15.160:0/1920957830 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff8540f51a0 msgr2=0x7ff8541064c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:08:23.136 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:23.134+0000 7ff85aab6700 1 --2- 172.21.15.160:0/1920957830 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff8540f51a0 0x7ff8541064c0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7ff850013510 tx=0x7ff85000b040).stop 2023-12-25T20:08:23.136 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:23.134+0000 7ff85aab6700 1 -- 172.21.15.160:0/1920957830 shutdown_connections 2023-12-25T20:08:23.136 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:23.134+0000 7ff85aab6700 1 --2- 172.21.15.160:0/1920957830 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7ff84002ff60 0x7ff840032420 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:08:23.136 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:23.134+0000 7ff85aab6700 1 --2- 172.21.15.160:0/1920957830 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff8540f51a0 0x7ff8541064c0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:08:23.136 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:23.134+0000 7ff85aab6700 1 -- 172.21.15.160:0/1920957830 >> 172.21.15.160:0/1920957830 conn(0x7ff8540f09a0 msgr2=0x7ff8540f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:08:23.136 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:23.134+0000 7ff85aab6700 1 -- 172.21.15.160:0/1920957830 shutdown_connections 2023-12-25T20:08:23.136 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:23.134+0000 7ff85aab6700 1 -- 172.21.15.160:0/1920957830 wait complete. 2023-12-25T20:08:23.136 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:08:24.469 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:08:24 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:08:22.631853+0000 mgr.smithi050.nbzhun (mgr.14162) 35 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:08:24.469 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:08:24 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:08:23.134463+0000 mon.smithi050 (mon.0) 133 : audit [DBG] from='client.? 172.21.15.160:0/1920957830' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:08:24.517 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:08:24.517 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:08:25.715 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:25.712+0000 7fe384339700 1 -- 172.21.15.160:0/3681556367 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe37c0f5410 msgr2=0x7fe37c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:08:25.715 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:25.712+0000 7fe384339700 1 --2- 172.21.15.160:0/3681556367 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe37c0f5410 0x7fe37c0f5830 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7fe36c009fb0 tx=0x7fe36c009b30).stop 2023-12-25T20:08:25.715 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:25.712+0000 7fe384339700 1 -- 172.21.15.160:0/3681556367 shutdown_connections 2023-12-25T20:08:25.715 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:25.712+0000 7fe384339700 1 --2- 172.21.15.160:0/3681556367 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe37c0f5410 0x7fe37c0f5830 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:08:25.715 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:25.712+0000 7fe384339700 1 -- 172.21.15.160:0/3681556367 >> 172.21.15.160:0/3681556367 conn(0x7fe37c0f0b70 msgr2=0x7fe37c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:08:25.715 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:25.712+0000 7fe384339700 1 -- 172.21.15.160:0/3681556367 shutdown_connections 2023-12-25T20:08:25.715 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:25.712+0000 7fe384339700 1 -- 172.21.15.160:0/3681556367 wait complete. 2023-12-25T20:08:25.715 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:25.713+0000 7fe384339700 1 Processor -- start 2023-12-25T20:08:25.715 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:25.713+0000 7fe384339700 1 -- start start 2023-12-25T20:08:25.716 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:25.713+0000 7fe384339700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe37c0f5410 0x7fe37c105600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:08:25.716 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:25.713+0000 7fe384339700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe37c105af0 con 0x7fe37c0f5410 2023-12-25T20:08:25.716 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:25.713+0000 7fe3820d5700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe37c0f5410 0x7fe37c105600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:08:25.716 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:25.713+0000 7fe3820d5700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe37c0f5410 0x7fe37c105600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:39428/0 (socket says 172.21.15.160:39428) 2023-12-25T20:08:25.716 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:25.713+0000 7fe3820d5700 1 -- 172.21.15.160:0/2861691743 learned_addr learned my addr 172.21.15.160:0/2861691743 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:08:25.716 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:25.714+0000 7fe3820d5700 1 -- 172.21.15.160:0/2861691743 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe36c005040 con 0x7fe37c0f5410 2023-12-25T20:08:25.716 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:25.714+0000 7fe3820d5700 1 --2- 172.21.15.160:0/2861691743 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe37c0f5410 0x7fe37c105600 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7fe36c013510 tx=0x7fe36c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:08:25.716 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:25.714+0000 7fe372ffd700 1 -- 172.21.15.160:0/2861691743 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe36c009320 con 0x7fe37c0f5410 2023-12-25T20:08:25.716 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:25.714+0000 7fe372ffd700 1 -- 172.21.15.160:0/2861691743 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe36c013f00 con 0x7fe37c0f5410 2023-12-25T20:08:25.716 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:25.714+0000 7fe384339700 1 -- 172.21.15.160:0/2861691743 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe37c105cf0 con 0x7fe37c0f5410 2023-12-25T20:08:25.716 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:25.714+0000 7fe372ffd700 1 -- 172.21.15.160:0/2861691743 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe36c014d90 con 0x7fe37c0f5410 2023-12-25T20:08:25.716 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:25.714+0000 7fe384339700 1 -- 172.21.15.160:0/2861691743 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe37c106180 con 0x7fe37c0f5410 2023-12-25T20:08:25.717 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:25.715+0000 7fe384339700 1 -- 172.21.15.160:0/2861691743 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe37c045ad0 con 0x7fe37c0f5410 2023-12-25T20:08:25.717 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:25.715+0000 7fe372ffd700 1 -- 172.21.15.160:0/2861691743 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe36c014420 con 0x7fe37c0f5410 2023-12-25T20:08:25.717 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:25.715+0000 7fe372ffd700 1 --2- 172.21.15.160:0/2861691743 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fe36802ff60 0x7fe368032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:08:25.717 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:25.715+0000 7fe372ffd700 1 -- 172.21.15.160:0/2861691743 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe36c041000 con 0x7fe37c0f5410 2023-12-25T20:08:25.717 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:25.715+0000 7fe3818d4700 1 --2- 172.21.15.160:0/2861691743 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fe36802ff60 0x7fe368032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:08:25.718 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:25.716+0000 7fe3818d4700 1 --2- 172.21.15.160:0/2861691743 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fe36802ff60 0x7fe368032420 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7fe3780096a0 tx=0x7fe378006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:08:25.720 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:25.720+0000 7fe372ffd700 1 -- 172.21.15.160:0/2861691743 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe36c0095f0 con 0x7fe37c0f5410 2023-12-25T20:08:25.925 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:25.924+0000 7fe384339700 1 -- 172.21.15.160:0/2861691743 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe37c0fae60 con 0x7fe37c0f5410 2023-12-25T20:08:25.926 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:25.925+0000 7fe372ffd700 1 -- 172.21.15.160:0/2861691743 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe36c01db70 con 0x7fe37c0f5410 2023-12-25T20:08:25.926 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:08:25.926 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:08:25.928 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:25.927+0000 7fe384339700 1 -- 172.21.15.160:0/2861691743 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fe36802ff60 msgr2=0x7fe368032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:08:25.928 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:25.927+0000 7fe384339700 1 --2- 172.21.15.160:0/2861691743 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fe36802ff60 0x7fe368032420 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7fe3780096a0 tx=0x7fe378006b40).stop 2023-12-25T20:08:25.928 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:25.927+0000 7fe384339700 1 -- 172.21.15.160:0/2861691743 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe37c0f5410 msgr2=0x7fe37c105600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:08:25.928 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:25.927+0000 7fe384339700 1 --2- 172.21.15.160:0/2861691743 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe37c0f5410 0x7fe37c105600 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7fe36c013510 tx=0x7fe36c00b040).stop 2023-12-25T20:08:25.928 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:25.927+0000 7fe384339700 1 -- 172.21.15.160:0/2861691743 shutdown_connections 2023-12-25T20:08:25.928 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:25.927+0000 7fe384339700 1 --2- 172.21.15.160:0/2861691743 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fe36802ff60 0x7fe368032420 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:08:25.929 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:25.927+0000 7fe384339700 1 --2- 172.21.15.160:0/2861691743 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe37c0f5410 0x7fe37c105600 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:08:25.929 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:25.927+0000 7fe384339700 1 -- 172.21.15.160:0/2861691743 >> 172.21.15.160:0/2861691743 conn(0x7fe37c0f0b70 msgr2=0x7fe37c0f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:08:25.929 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:25.927+0000 7fe384339700 1 -- 172.21.15.160:0/2861691743 shutdown_connections 2023-12-25T20:08:25.929 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:25.927+0000 7fe384339700 1 -- 172.21.15.160:0/2861691743 wait complete. 2023-12-25T20:08:25.929 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:08:26.469 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:08:26 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:08:24.632140+0000 mgr.smithi050.nbzhun (mgr.14162) 36 : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:08:26.469 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:08:26 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:08:25.926943+0000 mon.smithi050 (mon.0) 134 : audit [DBG] from='client.? 172.21.15.160:0/2861691743' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:08:27.322 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:08:27.323 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:08:28.468 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:08:28 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:08:26.632488+0000 mgr.smithi050.nbzhun (mgr.14162) 37 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:08:28.521 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:28.519+0000 7fdaacdef700 1 -- 172.21.15.160:0/786033818 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdaa8060280 msgr2=0x7fdaa80f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:08:28.521 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:28.519+0000 7fdaacdef700 1 --2- 172.21.15.160:0/786033818 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdaa8060280 0x7fdaa80f90f0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7fda9c009fb0 tx=0x7fda9c009b30).stop 2023-12-25T20:08:28.521 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:28.519+0000 7fdaacdef700 1 -- 172.21.15.160:0/786033818 shutdown_connections 2023-12-25T20:08:28.521 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:28.519+0000 7fdaacdef700 1 --2- 172.21.15.160:0/786033818 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdaa8060280 0x7fdaa80f90f0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:08:28.521 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:28.519+0000 7fdaacdef700 1 -- 172.21.15.160:0/786033818 >> 172.21.15.160:0/786033818 conn(0x7fdaa80f0b70 msgr2=0x7fdaa80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:08:28.522 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:28.519+0000 7fdaacdef700 1 -- 172.21.15.160:0/786033818 shutdown_connections 2023-12-25T20:08:28.522 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:28.519+0000 7fdaacdef700 1 -- 172.21.15.160:0/786033818 wait complete. 2023-12-25T20:08:28.522 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:28.520+0000 7fdaacdef700 1 Processor -- start 2023-12-25T20:08:28.522 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:28.520+0000 7fdaacdef700 1 -- start start 2023-12-25T20:08:28.522 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:28.520+0000 7fdaacdef700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdaa8060280 0x7fdaa81066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:08:28.522 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:28.520+0000 7fdaacdef700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdaa8106bd0 con 0x7fdaa8060280 2023-12-25T20:08:28.522 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:28.520+0000 7fdaa659c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdaa8060280 0x7fdaa81066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:08:28.522 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:28.520+0000 7fdaa659c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdaa8060280 0x7fdaa81066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:38720/0 (socket says 172.21.15.160:38720) 2023-12-25T20:08:28.522 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:28.520+0000 7fdaa659c700 1 -- 172.21.15.160:0/3806307769 learned_addr learned my addr 172.21.15.160:0/3806307769 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:08:28.522 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:28.520+0000 7fdaa659c700 1 -- 172.21.15.160:0/3806307769 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda9c005040 con 0x7fdaa8060280 2023-12-25T20:08:28.522 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:28.520+0000 7fdaa659c700 1 --2- 172.21.15.160:0/3806307769 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdaa8060280 0x7fdaa81066e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7fda9c013510 tx=0x7fda9c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:08:28.523 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:28.521+0000 7fda96ffd700 1 -- 172.21.15.160:0/3806307769 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda9c009320 con 0x7fdaa8060280 2023-12-25T20:08:28.523 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:28.521+0000 7fdaacdef700 1 -- 172.21.15.160:0/3806307769 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdaa8106dd0 con 0x7fdaa8060280 2023-12-25T20:08:28.523 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:28.521+0000 7fda96ffd700 1 -- 172.21.15.160:0/3806307769 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fda9c013f00 con 0x7fdaa8060280 2023-12-25T20:08:28.524 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:28.521+0000 7fda96ffd700 1 -- 172.21.15.160:0/3806307769 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda9c014dc0 con 0x7fdaa8060280 2023-12-25T20:08:28.524 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:28.521+0000 7fdaacdef700 1 -- 172.21.15.160:0/3806307769 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdaa8107260 con 0x7fdaa8060280 2023-12-25T20:08:28.524 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:28.522+0000 7fda96ffd700 1 -- 172.21.15.160:0/3806307769 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fda9c0095f0 con 0x7fdaa8060280 2023-12-25T20:08:28.524 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:28.522+0000 7fdaacdef700 1 -- 172.21.15.160:0/3806307769 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdaa8045ad0 con 0x7fdaa8060280 2023-12-25T20:08:28.524 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:28.522+0000 7fda96ffd700 1 --2- 172.21.15.160:0/3806307769 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fda9002ff60 0x7fda90032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:08:28.524 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:28.522+0000 7fda96ffd700 1 -- 172.21.15.160:0/3806307769 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fda9c041000 con 0x7fdaa8060280 2023-12-25T20:08:28.524 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:28.522+0000 7fdaa5d9b700 1 --2- 172.21.15.160:0/3806307769 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fda9002ff60 0x7fda90032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:08:28.524 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:28.522+0000 7fdaa5d9b700 1 --2- 172.21.15.160:0/3806307769 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fda9002ff60 0x7fda90032420 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7fda980096a0 tx=0x7fda98006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:08:28.527 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:28.527+0000 7fda96ffd700 1 -- 172.21.15.160:0/3806307769 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fda9c01c790 con 0x7fdaa8060280 2023-12-25T20:08:28.729 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:28.728+0000 7fdaacdef700 1 -- 172.21.15.160:0/3806307769 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdaa8108110 con 0x7fdaa8060280 2023-12-25T20:08:28.730 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:28.729+0000 7fda96ffd700 1 -- 172.21.15.160:0/3806307769 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fda9c010230 con 0x7fdaa8060280 2023-12-25T20:08:28.730 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:08:28.730 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:08:28.732 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:28.731+0000 7fdaacdef700 1 -- 172.21.15.160:0/3806307769 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fda9002ff60 msgr2=0x7fda90032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:08:28.732 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:28.731+0000 7fdaacdef700 1 --2- 172.21.15.160:0/3806307769 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fda9002ff60 0x7fda90032420 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7fda980096a0 tx=0x7fda98006b40).stop 2023-12-25T20:08:28.732 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:28.731+0000 7fdaacdef700 1 -- 172.21.15.160:0/3806307769 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdaa8060280 msgr2=0x7fdaa81066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:08:28.732 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:28.731+0000 7fdaacdef700 1 --2- 172.21.15.160:0/3806307769 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdaa8060280 0x7fdaa81066e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7fda9c013510 tx=0x7fda9c00b040).stop 2023-12-25T20:08:28.732 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:28.731+0000 7fdaacdef700 1 -- 172.21.15.160:0/3806307769 shutdown_connections 2023-12-25T20:08:28.732 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:28.731+0000 7fdaacdef700 1 --2- 172.21.15.160:0/3806307769 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fda9002ff60 0x7fda90032420 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:08:28.732 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:28.731+0000 7fdaacdef700 1 --2- 172.21.15.160:0/3806307769 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdaa8060280 0x7fdaa81066e0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:08:28.732 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:28.731+0000 7fdaacdef700 1 -- 172.21.15.160:0/3806307769 >> 172.21.15.160:0/3806307769 conn(0x7fdaa80f0b70 msgr2=0x7fdaa80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:08:28.732 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:28.731+0000 7fdaacdef700 1 -- 172.21.15.160:0/3806307769 shutdown_connections 2023-12-25T20:08:28.732 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:28.731+0000 7fdaacdef700 1 -- 172.21.15.160:0/3806307769 wait complete. 2023-12-25T20:08:28.733 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:08:29.469 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:08:29 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:08:28.730534+0000 mon.smithi050 (mon.0) 135 : audit [DBG] from='client.? 172.21.15.160:0/3806307769' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:08:30.128 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:08:30.129 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:08:30.469 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:08:30 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:08:28.632813+0000 mgr.smithi050.nbzhun (mgr.14162) 38 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:08:31.318 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:31.316+0000 7f9fbe35a700 1 -- 172.21.15.160:0/1414780680 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9fb80f5410 msgr2=0x7f9fb80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:08:31.318 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:31.316+0000 7f9fbe35a700 1 --2- 172.21.15.160:0/1414780680 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9fb80f5410 0x7f9fb80f5830 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f9fa0009fb0 tx=0x7f9fa0009b30).stop 2023-12-25T20:08:31.318 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:31.316+0000 7f9fbe35a700 1 -- 172.21.15.160:0/1414780680 shutdown_connections 2023-12-25T20:08:31.318 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:31.316+0000 7f9fbe35a700 1 --2- 172.21.15.160:0/1414780680 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9fb80f5410 0x7f9fb80f5830 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:08:31.318 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:31.316+0000 7f9fbe35a700 1 -- 172.21.15.160:0/1414780680 >> 172.21.15.160:0/1414780680 conn(0x7f9fb80f0b70 msgr2=0x7f9fb80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:08:31.318 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:31.316+0000 7f9fbe35a700 1 -- 172.21.15.160:0/1414780680 shutdown_connections 2023-12-25T20:08:31.318 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:31.316+0000 7f9fbe35a700 1 -- 172.21.15.160:0/1414780680 wait complete. 2023-12-25T20:08:31.318 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:31.317+0000 7f9fbe35a700 1 Processor -- start 2023-12-25T20:08:31.318 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:31.317+0000 7f9fbe35a700 1 -- start start 2023-12-25T20:08:31.319 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:31.317+0000 7f9fbe35a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9fb80f5410 0x7f9fb8109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:08:31.319 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:31.317+0000 7f9fbe35a700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fb8060220 con 0x7f9fb80f5410 2023-12-25T20:08:31.320 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:31.317+0000 7f9fb7fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9fb80f5410 0x7f9fb8109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:08:31.320 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:31.317+0000 7f9fb7fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9fb80f5410 0x7f9fb8109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:38738/0 (socket says 172.21.15.160:38738) 2023-12-25T20:08:31.320 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:31.317+0000 7f9fb7fff700 1 -- 172.21.15.160:0/28081078 learned_addr learned my addr 172.21.15.160:0/28081078 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:08:31.320 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:31.318+0000 7f9fb7fff700 1 -- 172.21.15.160:0/28081078 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fa0005040 con 0x7f9fb80f5410 2023-12-25T20:08:31.320 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:31.318+0000 7f9fb7fff700 1 --2- 172.21.15.160:0/28081078 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9fb80f5410 0x7f9fb8109450 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f9fa0014000 tx=0x7f9fa000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:08:31.320 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:31.318+0000 7f9fb4ff9700 1 -- 172.21.15.160:0/28081078 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9fa00095f0 con 0x7f9fb80f5410 2023-12-25T20:08:31.321 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:31.318+0000 7f9fbe35a700 1 -- 172.21.15.160:0/28081078 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fb8060420 con 0x7f9fb80f5410 2023-12-25T20:08:31.321 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:31.318+0000 7f9fb4ff9700 1 -- 172.21.15.160:0/28081078 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9fa0007e70 con 0x7f9fb80f5410 2023-12-25T20:08:31.321 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:31.318+0000 7f9fb4ff9700 1 -- 172.21.15.160:0/28081078 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9fa0014ea0 con 0x7f9fb80f5410 2023-12-25T20:08:31.321 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:31.318+0000 7f9fbe35a700 1 -- 172.21.15.160:0/28081078 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fb8109b00 con 0x7f9fb80f5410 2023-12-25T20:08:31.321 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:31.319+0000 7f9fbe35a700 1 -- 172.21.15.160:0/28081078 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fb8059100 con 0x7f9fb80f5410 2023-12-25T20:08:31.321 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:31.319+0000 7f9fb4ff9700 1 -- 172.21.15.160:0/28081078 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9fa0025070 con 0x7f9fb80f5410 2023-12-25T20:08:31.321 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:31.320+0000 7f9fb4ff9700 1 --2- 172.21.15.160:0/28081078 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f9fa402ff60 0x7f9fa4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:08:31.321 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:31.320+0000 7f9fb4ff9700 1 -- 172.21.15.160:0/28081078 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9fa0040e60 con 0x7f9fb80f5410 2023-12-25T20:08:31.321 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:31.320+0000 7f9fb77fe700 1 --2- 172.21.15.160:0/28081078 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f9fa402ff60 0x7f9fa4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:08:31.322 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:31.320+0000 7f9fb77fe700 1 --2- 172.21.15.160:0/28081078 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f9fa402ff60 0x7f9fa4032420 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7f9fa80096a0 tx=0x7f9fa8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:08:31.324 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:31.324+0000 7f9fb4ff9700 1 -- 172.21.15.160:0/28081078 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9fa000f300 con 0x7f9fb80f5410 2023-12-25T20:08:31.530 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:31.529+0000 7f9fbe35a700 1 -- 172.21.15.160:0/28081078 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9fb80fae60 con 0x7f9fb80f5410 2023-12-25T20:08:31.531 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:31.530+0000 7f9fb4ff9700 1 -- 172.21.15.160:0/28081078 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9fa000f300 con 0x7f9fb80f5410 2023-12-25T20:08:31.531 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:08:31.531 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:08:31.533 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:31.532+0000 7f9fbe35a700 1 -- 172.21.15.160:0/28081078 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f9fa402ff60 msgr2=0x7f9fa4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:08:31.533 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:31.532+0000 7f9fbe35a700 1 --2- 172.21.15.160:0/28081078 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f9fa402ff60 0x7f9fa4032420 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7f9fa80096a0 tx=0x7f9fa8006b40).stop 2023-12-25T20:08:31.533 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:31.532+0000 7f9fbe35a700 1 -- 172.21.15.160:0/28081078 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9fb80f5410 msgr2=0x7f9fb8109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:08:31.533 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:31.532+0000 7f9fbe35a700 1 --2- 172.21.15.160:0/28081078 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9fb80f5410 0x7f9fb8109450 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f9fa0014000 tx=0x7f9fa000b040).stop 2023-12-25T20:08:31.533 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:31.532+0000 7f9fbe35a700 1 -- 172.21.15.160:0/28081078 shutdown_connections 2023-12-25T20:08:31.533 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:31.532+0000 7f9fbe35a700 1 --2- 172.21.15.160:0/28081078 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f9fa402ff60 0x7f9fa4032420 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:08:31.533 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:31.532+0000 7f9fbe35a700 1 --2- 172.21.15.160:0/28081078 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9fb80f5410 0x7f9fb8109450 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:08:31.533 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:31.532+0000 7f9fbe35a700 1 -- 172.21.15.160:0/28081078 >> 172.21.15.160:0/28081078 conn(0x7f9fb80f0b70 msgr2=0x7f9fb80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:08:31.533 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:31.532+0000 7f9fbe35a700 1 -- 172.21.15.160:0/28081078 shutdown_connections 2023-12-25T20:08:31.533 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:31.532+0000 7f9fbe35a700 1 -- 172.21.15.160:0/28081078 wait complete. 2023-12-25T20:08:31.534 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:08:32.468 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:08:32 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:08:30.633061+0000 mgr.smithi050.nbzhun (mgr.14162) 39 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:08:32.469 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:08:32 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:08:31.531733+0000 mon.smithi050 (mon.0) 136 : audit [DBG] from='client.? 172.21.15.160:0/28081078' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:08:32.919 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:08:32.920 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:08:34.121 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:34.119+0000 7f5b2ec60700 1 -- 172.21.15.160:0/3208513760 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5b280f6480 msgr2=0x7f5b280f68a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:08:34.122 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:34.119+0000 7f5b2ec60700 1 --2- 172.21.15.160:0/3208513760 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5b280f6480 0x7f5b280f68a0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f5b1c009fb0 tx=0x7f5b1c009b30).stop 2023-12-25T20:08:34.122 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:34.119+0000 7f5b2ec60700 1 -- 172.21.15.160:0/3208513760 shutdown_connections 2023-12-25T20:08:34.122 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:34.119+0000 7f5b2ec60700 1 --2- 172.21.15.160:0/3208513760 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5b280f6480 0x7f5b280f68a0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:08:34.122 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:34.119+0000 7f5b2ec60700 1 -- 172.21.15.160:0/3208513760 >> 172.21.15.160:0/3208513760 conn(0x7f5b280f1c00 msgr2=0x7f5b280f4020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:08:34.122 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:34.119+0000 7f5b2ec60700 1 -- 172.21.15.160:0/3208513760 shutdown_connections 2023-12-25T20:08:34.122 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:34.119+0000 7f5b2ec60700 1 -- 172.21.15.160:0/3208513760 wait complete. 2023-12-25T20:08:34.122 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:34.120+0000 7f5b2ec60700 1 Processor -- start 2023-12-25T20:08:34.122 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:34.120+0000 7f5b2ec60700 1 -- start start 2023-12-25T20:08:34.122 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:34.120+0000 7f5b2ec60700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5b280f6480 0x7f5b281066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:08:34.122 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:34.120+0000 7f5b2ec60700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b28106bc0 con 0x7f5b280f6480 2023-12-25T20:08:34.122 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:34.120+0000 7f5b2c9fc700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5b280f6480 0x7f5b281066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:08:34.123 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:34.120+0000 7f5b2c9fc700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5b280f6480 0x7f5b281066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:38750/0 (socket says 172.21.15.160:38750) 2023-12-25T20:08:34.123 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:34.120+0000 7f5b2c9fc700 1 -- 172.21.15.160:0/793658903 learned_addr learned my addr 172.21.15.160:0/793658903 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:08:34.123 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:34.120+0000 7f5b2c9fc700 1 -- 172.21.15.160:0/793658903 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b1c005040 con 0x7f5b280f6480 2023-12-25T20:08:34.123 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:34.121+0000 7f5b2c9fc700 1 --2- 172.21.15.160:0/793658903 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5b280f6480 0x7f5b281066d0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f5b1c013510 tx=0x7f5b1c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:08:34.123 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:34.121+0000 7f5b257fa700 1 -- 172.21.15.160:0/793658903 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5b1c009320 con 0x7f5b280f6480 2023-12-25T20:08:34.123 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:34.121+0000 7f5b257fa700 1 -- 172.21.15.160:0/793658903 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5b1c013f00 con 0x7f5b280f6480 2023-12-25T20:08:34.123 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:34.121+0000 7f5b2ec60700 1 -- 172.21.15.160:0/793658903 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b28106dc0 con 0x7f5b280f6480 2023-12-25T20:08:34.124 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:34.121+0000 7f5b257fa700 1 -- 172.21.15.160:0/793658903 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5b1c014d90 con 0x7f5b280f6480 2023-12-25T20:08:34.124 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:34.121+0000 7f5b2ec60700 1 -- 172.21.15.160:0/793658903 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b28107250 con 0x7f5b280f6480 2023-12-25T20:08:34.124 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:34.122+0000 7f5b257fa700 1 -- 172.21.15.160:0/793658903 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5b1c0263a0 con 0x7f5b280f6480 2023-12-25T20:08:34.124 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:34.122+0000 7f5b257fa700 1 --2- 172.21.15.160:0/793658903 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f5b1802ffb0 0x7f5b18032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:08:34.124 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:34.122+0000 7f5b257fa700 1 -- 172.21.15.160:0/793658903 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5b1c0412e0 con 0x7f5b280f6480 2023-12-25T20:08:34.124 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:34.122+0000 7f5b2ec60700 1 -- 172.21.15.160:0/793658903 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b28045ad0 con 0x7f5b280f6480 2023-12-25T20:08:34.124 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:34.122+0000 7f5b27fff700 1 --2- 172.21.15.160:0/793658903 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f5b1802ffb0 0x7f5b18032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:08:34.124 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:34.123+0000 7f5b27fff700 1 --2- 172.21.15.160:0/793658903 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f5b1802ffb0 0x7f5b18032470 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7f5b140096a0 tx=0x7f5b14006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:08:34.127 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:34.127+0000 7f5b257fa700 1 -- 172.21.15.160:0/793658903 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5b1c01cb10 con 0x7f5b280f6480 2023-12-25T20:08:34.332 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:34.331+0000 7f5b2ec60700 1 -- 172.21.15.160:0/793658903 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5b28108100 con 0x7f5b280f6480 2023-12-25T20:08:34.332 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:34.331+0000 7f5b257fa700 1 -- 172.21.15.160:0/793658903 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5b1c02b030 con 0x7f5b280f6480 2023-12-25T20:08:34.332 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:08:34.333 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:08:34.334 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:34.333+0000 7f5b2ec60700 1 -- 172.21.15.160:0/793658903 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f5b1802ffb0 msgr2=0x7f5b18032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:08:34.334 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:34.333+0000 7f5b2ec60700 1 --2- 172.21.15.160:0/793658903 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f5b1802ffb0 0x7f5b18032470 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7f5b140096a0 tx=0x7f5b14006b40).stop 2023-12-25T20:08:34.334 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:34.333+0000 7f5b2ec60700 1 -- 172.21.15.160:0/793658903 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5b280f6480 msgr2=0x7f5b281066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:08:34.334 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:34.333+0000 7f5b2ec60700 1 --2- 172.21.15.160:0/793658903 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5b280f6480 0x7f5b281066d0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f5b1c013510 tx=0x7f5b1c00b040).stop 2023-12-25T20:08:34.334 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:34.333+0000 7f5b2ec60700 1 -- 172.21.15.160:0/793658903 shutdown_connections 2023-12-25T20:08:34.334 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:34.333+0000 7f5b2ec60700 1 --2- 172.21.15.160:0/793658903 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f5b1802ffb0 0x7f5b18032470 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:08:34.334 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:34.333+0000 7f5b2ec60700 1 --2- 172.21.15.160:0/793658903 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5b280f6480 0x7f5b281066d0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:08:34.334 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:34.333+0000 7f5b2ec60700 1 -- 172.21.15.160:0/793658903 >> 172.21.15.160:0/793658903 conn(0x7f5b280f1c00 msgr2=0x7f5b280f2720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:08:34.334 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:34.333+0000 7f5b2ec60700 1 -- 172.21.15.160:0/793658903 shutdown_connections 2023-12-25T20:08:34.334 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:34.333+0000 7f5b2ec60700 1 -- 172.21.15.160:0/793658903 wait complete. 2023-12-25T20:08:34.335 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:08:34.468 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:08:34 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:08:32.633419+0000 mgr.smithi050.nbzhun (mgr.14162) 40 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:08:35.468 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:08:35 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:08:34.333126+0000 mon.smithi050 (mon.0) 137 : audit [DBG] from='client.? 172.21.15.160:0/793658903' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:08:35.740 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:08:35.740 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:08:36.469 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:08:36 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:08:34.633756+0000 mgr.smithi050.nbzhun (mgr.14162) 41 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:08:36.941 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:36.940+0000 7ff6096c6700 1 -- 172.21.15.160:0/3898594480 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff6040f4860 msgr2=0x7ff6040f4c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:08:36.942 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:36.940+0000 7ff6096c6700 1 --2- 172.21.15.160:0/3898594480 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff6040f4860 0x7ff6040f4c80 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7ff5f8009fb0 tx=0x7ff5f8009b30).stop 2023-12-25T20:08:36.942 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:36.940+0000 7ff6096c6700 1 -- 172.21.15.160:0/3898594480 shutdown_connections 2023-12-25T20:08:36.942 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:36.940+0000 7ff6096c6700 1 --2- 172.21.15.160:0/3898594480 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff6040f4860 0x7ff6040f4c80 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:08:36.942 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:36.940+0000 7ff6096c6700 1 -- 172.21.15.160:0/3898594480 >> 172.21.15.160:0/3898594480 conn(0x7ff6040f02e0 msgr2=0x7ff6040f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:08:36.942 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:36.940+0000 7ff6096c6700 1 -- 172.21.15.160:0/3898594480 shutdown_connections 2023-12-25T20:08:36.942 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:36.940+0000 7ff6096c6700 1 -- 172.21.15.160:0/3898594480 wait complete. 2023-12-25T20:08:36.942 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:36.941+0000 7ff6096c6700 1 Processor -- start 2023-12-25T20:08:36.942 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:36.941+0000 7ff6096c6700 1 -- start start 2023-12-25T20:08:36.942 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:36.941+0000 7ff6096c6700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff6040f4860 0x7ff6041068e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:08:36.942 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:36.941+0000 7ff6096c6700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff604106dd0 con 0x7ff6040f4860 2023-12-25T20:08:36.942 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:36.941+0000 7ff602ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff6040f4860 0x7ff6041068e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:08:36.942 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:36.942+0000 7ff602ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff6040f4860 0x7ff6041068e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:38760/0 (socket says 172.21.15.160:38760) 2023-12-25T20:08:36.942 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:36.942+0000 7ff602ffd700 1 -- 172.21.15.160:0/264860862 learned_addr learned my addr 172.21.15.160:0/264860862 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:08:36.943 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:36.942+0000 7ff602ffd700 1 -- 172.21.15.160:0/264860862 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5f8005040 con 0x7ff6040f4860 2023-12-25T20:08:36.943 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:36.942+0000 7ff602ffd700 1 --2- 172.21.15.160:0/264860862 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff6040f4860 0x7ff6041068e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7ff5f8013510 tx=0x7ff5f800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:08:36.944 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:36.942+0000 7ff5eb7fe700 1 -- 172.21.15.160:0/264860862 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff5f8009320 con 0x7ff6040f4860 2023-12-25T20:08:36.944 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:36.942+0000 7ff6096c6700 1 -- 172.21.15.160:0/264860862 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff604106fd0 con 0x7ff6040f4860 2023-12-25T20:08:36.944 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:36.942+0000 7ff5eb7fe700 1 -- 172.21.15.160:0/264860862 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff5f8013f00 con 0x7ff6040f4860 2023-12-25T20:08:36.944 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:36.942+0000 7ff5eb7fe700 1 -- 172.21.15.160:0/264860862 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff5f8014dc0 con 0x7ff6040f4860 2023-12-25T20:08:36.944 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:36.942+0000 7ff6096c6700 1 -- 172.21.15.160:0/264860862 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff604107460 con 0x7ff6040f4860 2023-12-25T20:08:36.944 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:36.943+0000 7ff5eb7fe700 1 -- 172.21.15.160:0/264860862 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff5f8014420 con 0x7ff6040f4860 2023-12-25T20:08:36.944 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:36.943+0000 7ff6096c6700 1 -- 172.21.15.160:0/264860862 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff604045ad0 con 0x7ff6040f4860 2023-12-25T20:08:36.944 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:36.943+0000 7ff5eb7fe700 1 --2- 172.21.15.160:0/264860862 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7ff5ec02ffb0 0x7ff5ec032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:08:36.944 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:36.943+0000 7ff5eb7fe700 1 -- 172.21.15.160:0/264860862 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff5f80410b0 con 0x7ff6040f4860 2023-12-25T20:08:36.944 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:36.943+0000 7ff6027fc700 1 --2- 172.21.15.160:0/264860862 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7ff5ec02ffb0 0x7ff5ec032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:08:36.944 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:36.944+0000 7ff6027fc700 1 --2- 172.21.15.160:0/264860862 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7ff5ec02ffb0 0x7ff5ec032470 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7ff5f40096a0 tx=0x7ff5f4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:08:36.948 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:36.948+0000 7ff5eb7fe700 1 -- 172.21.15.160:0/264860862 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff5f806d2b0 con 0x7ff6040f4860 2023-12-25T20:08:37.155 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:37.155+0000 7ff6096c6700 1 -- 172.21.15.160:0/264860862 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff60405f370 con 0x7ff6040f4860 2023-12-25T20:08:37.156 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:37.156+0000 7ff5eb7fe700 1 -- 172.21.15.160:0/264860862 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff5f80095f0 con 0x7ff6040f4860 2023-12-25T20:08:37.157 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:08:37.157 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:08:37.158 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:37.158+0000 7ff6096c6700 1 -- 172.21.15.160:0/264860862 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7ff5ec02ffb0 msgr2=0x7ff5ec032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:08:37.159 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:37.158+0000 7ff6096c6700 1 --2- 172.21.15.160:0/264860862 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7ff5ec02ffb0 0x7ff5ec032470 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7ff5f40096a0 tx=0x7ff5f4006b40).stop 2023-12-25T20:08:37.159 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:37.158+0000 7ff6096c6700 1 -- 172.21.15.160:0/264860862 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff6040f4860 msgr2=0x7ff6041068e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:08:37.159 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:37.158+0000 7ff6096c6700 1 --2- 172.21.15.160:0/264860862 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff6040f4860 0x7ff6041068e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7ff5f8013510 tx=0x7ff5f800b040).stop 2023-12-25T20:08:37.159 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:37.158+0000 7ff6096c6700 1 -- 172.21.15.160:0/264860862 shutdown_connections 2023-12-25T20:08:37.159 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:37.158+0000 7ff6096c6700 1 --2- 172.21.15.160:0/264860862 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7ff5ec02ffb0 0x7ff5ec032470 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:08:37.159 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:37.158+0000 7ff6096c6700 1 --2- 172.21.15.160:0/264860862 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff6040f4860 0x7ff6041068e0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:08:37.160 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:37.158+0000 7ff6096c6700 1 -- 172.21.15.160:0/264860862 >> 172.21.15.160:0/264860862 conn(0x7ff6040f02e0 msgr2=0x7ff6040f0e40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:08:37.160 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:37.158+0000 7ff6096c6700 1 -- 172.21.15.160:0/264860862 shutdown_connections 2023-12-25T20:08:37.160 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:37.158+0000 7ff6096c6700 1 -- 172.21.15.160:0/264860862 wait complete. 2023-12-25T20:08:37.160 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:08:38.469 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:08:38 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:08:36.634141+0000 mgr.smithi050.nbzhun (mgr.14162) 42 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:08:38.469 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:08:38 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:08:37.156491+0000 mon.smithi050 (mon.0) 138 : audit [DBG] from='client.? 172.21.15.160:0/264860862' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:08:38.561 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:08:38.561 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:08:39.713 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:39.711+0000 7f94de15b700 1 -- 172.21.15.160:0/524924538 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f94d8060280 msgr2=0x7f94d80fb4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:08:39.713 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:39.711+0000 7f94de15b700 1 --2- 172.21.15.160:0/524924538 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f94d8060280 0x7f94d80fb4f0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f94cc009fb0 tx=0x7f94cc009b30).stop 2023-12-25T20:08:39.713 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:39.711+0000 7f94de15b700 1 -- 172.21.15.160:0/524924538 shutdown_connections 2023-12-25T20:08:39.713 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:39.711+0000 7f94de15b700 1 --2- 172.21.15.160:0/524924538 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f94d8060280 0x7f94d80fb4f0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:08:39.713 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:39.711+0000 7f94de15b700 1 -- 172.21.15.160:0/524924538 >> 172.21.15.160:0/524924538 conn(0x7f94d80f0b70 msgr2=0x7f94d80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:08:39.713 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:39.711+0000 7f94de15b700 1 -- 172.21.15.160:0/524924538 shutdown_connections 2023-12-25T20:08:39.713 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:39.711+0000 7f94de15b700 1 -- 172.21.15.160:0/524924538 wait complete. 2023-12-25T20:08:39.713 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:39.712+0000 7f94de15b700 1 Processor -- start 2023-12-25T20:08:39.713 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:39.712+0000 7f94de15b700 1 -- start start 2023-12-25T20:08:39.713 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:39.712+0000 7f94de15b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f94d8060280 0x7f94d81088e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:08:39.714 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:39.712+0000 7f94de15b700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94d8108dd0 con 0x7f94d8060280 2023-12-25T20:08:39.714 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:39.712+0000 7f94d77fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f94d8060280 0x7f94d81088e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:08:39.714 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:39.712+0000 7f94d77fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f94d8060280 0x7f94d81088e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:33774/0 (socket says 172.21.15.160:33774) 2023-12-25T20:08:39.714 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:39.712+0000 7f94d77fe700 1 -- 172.21.15.160:0/4049171369 learned_addr learned my addr 172.21.15.160:0/4049171369 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:08:39.714 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:39.712+0000 7f94d77fe700 1 -- 172.21.15.160:0/4049171369 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94cc005040 con 0x7f94d8060280 2023-12-25T20:08:39.714 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:39.713+0000 7f94d77fe700 1 --2- 172.21.15.160:0/4049171369 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f94d8060280 0x7f94d81088e0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f94cc013510 tx=0x7f94cc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:08:39.714 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:39.713+0000 7f94bffff700 1 -- 172.21.15.160:0/4049171369 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f94cc009320 con 0x7f94d8060280 2023-12-25T20:08:39.714 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:39.713+0000 7f94de15b700 1 -- 172.21.15.160:0/4049171369 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94d8108fd0 con 0x7f94d8060280 2023-12-25T20:08:39.714 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:39.713+0000 7f94bffff700 1 -- 172.21.15.160:0/4049171369 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f94cc013f00 con 0x7f94d8060280 2023-12-25T20:08:39.714 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:39.713+0000 7f94bffff700 1 -- 172.21.15.160:0/4049171369 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f94cc014dc0 con 0x7f94d8060280 2023-12-25T20:08:39.714 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:39.713+0000 7f94de15b700 1 -- 172.21.15.160:0/4049171369 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94d8109460 con 0x7f94d8060280 2023-12-25T20:08:39.715 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:39.714+0000 7f94bffff700 1 -- 172.21.15.160:0/4049171369 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f94cc014420 con 0x7f94d8060280 2023-12-25T20:08:39.715 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:39.714+0000 7f94de15b700 1 -- 172.21.15.160:0/4049171369 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94d8045ad0 con 0x7f94d8060280 2023-12-25T20:08:39.715 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:39.714+0000 7f94bffff700 1 --2- 172.21.15.160:0/4049171369 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f94c002ff60 0x7f94c0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:08:39.715 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:39.714+0000 7f94bffff700 1 -- 172.21.15.160:0/4049171369 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f94cc041030 con 0x7f94d8060280 2023-12-25T20:08:39.715 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:39.714+0000 7f94d6ffd700 1 --2- 172.21.15.160:0/4049171369 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f94c002ff60 0x7f94c0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:08:39.716 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:39.715+0000 7f94d6ffd700 1 --2- 172.21.15.160:0/4049171369 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f94c002ff60 0x7f94c0032420 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f94c80096a0 tx=0x7f94c8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:08:39.718 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:39.719+0000 7f94bffff700 1 -- 172.21.15.160:0/4049171369 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f94cc0095f0 con 0x7f94d8060280 2023-12-25T20:08:39.926 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:39.926+0000 7f94de15b700 1 -- 172.21.15.160:0/4049171369 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f94d810a310 con 0x7f94d8060280 2023-12-25T20:08:39.927 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:39.927+0000 7f94bffff700 1 -- 172.21.15.160:0/4049171369 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f94cc01cb10 con 0x7f94d8060280 2023-12-25T20:08:39.927 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:08:39.927 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:08:39.928 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:39.928+0000 7f94de15b700 1 -- 172.21.15.160:0/4049171369 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f94c002ff60 msgr2=0x7f94c0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:08:39.928 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:39.928+0000 7f94de15b700 1 --2- 172.21.15.160:0/4049171369 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f94c002ff60 0x7f94c0032420 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f94c80096a0 tx=0x7f94c8006b40).stop 2023-12-25T20:08:39.928 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:39.928+0000 7f94de15b700 1 -- 172.21.15.160:0/4049171369 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f94d8060280 msgr2=0x7f94d81088e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:08:39.928 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:39.928+0000 7f94de15b700 1 --2- 172.21.15.160:0/4049171369 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f94d8060280 0x7f94d81088e0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f94cc013510 tx=0x7f94cc00b040).stop 2023-12-25T20:08:39.929 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:39.928+0000 7f94de15b700 1 -- 172.21.15.160:0/4049171369 shutdown_connections 2023-12-25T20:08:39.929 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:39.928+0000 7f94de15b700 1 --2- 172.21.15.160:0/4049171369 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f94c002ff60 0x7f94c0032420 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:08:39.929 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:39.928+0000 7f94de15b700 1 --2- 172.21.15.160:0/4049171369 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f94d8060280 0x7f94d81088e0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:08:39.929 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:39.928+0000 7f94de15b700 1 -- 172.21.15.160:0/4049171369 >> 172.21.15.160:0/4049171369 conn(0x7f94d80f0b70 msgr2=0x7f94d80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:08:39.929 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:39.928+0000 7f94de15b700 1 -- 172.21.15.160:0/4049171369 shutdown_connections 2023-12-25T20:08:39.929 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:39.928+0000 7f94de15b700 1 -- 172.21.15.160:0/4049171369 wait complete. 2023-12-25T20:08:39.929 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:08:40.468 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:08:40 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:08:38.634522+0000 mgr.smithi050.nbzhun (mgr.14162) 43 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:08:40.469 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:08:40 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:08:39.927365+0000 mon.smithi050 (mon.0) 139 : audit [DBG] from='client.? 172.21.15.160:0/4049171369' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:08:41.334 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:08:41.334 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:08:42.468 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:08:42 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:08:40.634902+0000 mgr.smithi050.nbzhun (mgr.14162) 44 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:08:43.539 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:43.539+0000 7f0cd1c08700 1 -- 172.21.15.160:0/2182941691 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0ccc0f5410 msgr2=0x7f0ccc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:08:43.540 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:43.539+0000 7f0cd1c08700 1 --2- 172.21.15.160:0/2182941691 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0ccc0f5410 0x7f0ccc0f5830 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f0cc0009fb0 tx=0x7f0cc0009b30).stop 2023-12-25T20:08:43.540 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:43.539+0000 7f0cd1c08700 1 -- 172.21.15.160:0/2182941691 shutdown_connections 2023-12-25T20:08:43.540 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:43.539+0000 7f0cd1c08700 1 --2- 172.21.15.160:0/2182941691 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0ccc0f5410 0x7f0ccc0f5830 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:08:43.540 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:43.539+0000 7f0cd1c08700 1 -- 172.21.15.160:0/2182941691 >> 172.21.15.160:0/2182941691 conn(0x7f0ccc0f0b70 msgr2=0x7f0ccc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:08:43.540 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:43.539+0000 7f0cd1c08700 1 -- 172.21.15.160:0/2182941691 shutdown_connections 2023-12-25T20:08:43.540 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:43.539+0000 7f0cd1c08700 1 -- 172.21.15.160:0/2182941691 wait complete. 2023-12-25T20:08:43.541 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:43.540+0000 7f0cd1c08700 1 Processor -- start 2023-12-25T20:08:43.541 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:43.540+0000 7f0cd1c08700 1 -- start start 2023-12-25T20:08:43.541 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:43.540+0000 7f0cd1c08700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0ccc0f5410 0x7f0ccc105600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:08:43.542 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:43.540+0000 7f0cd1c08700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ccc105af0 con 0x7f0ccc0f5410 2023-12-25T20:08:43.542 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:43.541+0000 7f0ccb7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0ccc0f5410 0x7f0ccc105600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:08:43.542 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:43.541+0000 7f0ccb7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0ccc0f5410 0x7f0ccc105600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:33796/0 (socket says 172.21.15.160:33796) 2023-12-25T20:08:43.542 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:43.541+0000 7f0ccb7fe700 1 -- 172.21.15.160:0/2539509279 learned_addr learned my addr 172.21.15.160:0/2539509279 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:08:43.542 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:43.541+0000 7f0ccb7fe700 1 -- 172.21.15.160:0/2539509279 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cc0005040 con 0x7f0ccc0f5410 2023-12-25T20:08:43.542 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:43.541+0000 7f0ccb7fe700 1 --2- 172.21.15.160:0/2539509279 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0ccc0f5410 0x7f0ccc105600 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f0cc0013510 tx=0x7f0cc000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:08:43.542 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:43.541+0000 7f0cb3fff700 1 -- 172.21.15.160:0/2539509279 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0cc0009320 con 0x7f0ccc0f5410 2023-12-25T20:08:43.542 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:43.541+0000 7f0cb3fff700 1 -- 172.21.15.160:0/2539509279 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0cc0013f00 con 0x7f0ccc0f5410 2023-12-25T20:08:43.542 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:43.541+0000 7f0cd1c08700 1 -- 172.21.15.160:0/2539509279 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ccc105cf0 con 0x7f0ccc0f5410 2023-12-25T20:08:43.542 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:43.541+0000 7f0cb3fff700 1 -- 172.21.15.160:0/2539509279 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0cc0014d90 con 0x7f0ccc0f5410 2023-12-25T20:08:43.542 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:43.541+0000 7f0cd1c08700 1 -- 172.21.15.160:0/2539509279 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ccc106180 con 0x7f0ccc0f5410 2023-12-25T20:08:43.543 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:43.542+0000 7f0cb3fff700 1 -- 172.21.15.160:0/2539509279 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0cc000f410 con 0x7f0ccc0f5410 2023-12-25T20:08:43.543 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:43.543+0000 7f0cb3fff700 1 --2- 172.21.15.160:0/2539509279 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f0cb402ff60 0x7f0cb4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:08:43.543 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:43.543+0000 7f0cb3fff700 1 -- 172.21.15.160:0/2539509279 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0cc0042d90 con 0x7f0ccc0f5410 2023-12-25T20:08:43.543 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:43.543+0000 7f0cd1c08700 1 -- 172.21.15.160:0/2539509279 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ccc045ad0 con 0x7f0ccc0f5410 2023-12-25T20:08:43.543 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:43.543+0000 7f0ccaffd700 1 --2- 172.21.15.160:0/2539509279 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f0cb402ff60 0x7f0cb4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:08:43.543 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:43.543+0000 7f0ccaffd700 1 --2- 172.21.15.160:0/2539509279 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f0cb402ff60 0x7f0cb4032420 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f0cbc0096a0 tx=0x7f0cbc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:08:43.547 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:43.547+0000 7f0cb3fff700 1 -- 172.21.15.160:0/2539509279 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0cc0012920 con 0x7f0ccc0f5410 2023-12-25T20:08:43.753 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:43.753+0000 7f0cd1c08700 1 -- 172.21.15.160:0/2539509279 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0ccc0faf50 con 0x7f0ccc0f5410 2023-12-25T20:08:43.753 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:43.753+0000 7f0cb3fff700 1 -- 172.21.15.160:0/2539509279 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0cc0012330 con 0x7f0ccc0f5410 2023-12-25T20:08:43.753 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:08:43.754 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:08:43.755 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:43.755+0000 7f0cd1c08700 1 -- 172.21.15.160:0/2539509279 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f0cb402ff60 msgr2=0x7f0cb4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:08:43.755 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:43.755+0000 7f0cd1c08700 1 --2- 172.21.15.160:0/2539509279 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f0cb402ff60 0x7f0cb4032420 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f0cbc0096a0 tx=0x7f0cbc006b40).stop 2023-12-25T20:08:43.755 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:43.755+0000 7f0cd1c08700 1 -- 172.21.15.160:0/2539509279 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0ccc0f5410 msgr2=0x7f0ccc105600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:08:43.755 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:43.755+0000 7f0cd1c08700 1 --2- 172.21.15.160:0/2539509279 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0ccc0f5410 0x7f0ccc105600 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f0cc0013510 tx=0x7f0cc000b040).stop 2023-12-25T20:08:43.755 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:43.755+0000 7f0cd1c08700 1 -- 172.21.15.160:0/2539509279 shutdown_connections 2023-12-25T20:08:43.755 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:43.755+0000 7f0cd1c08700 1 --2- 172.21.15.160:0/2539509279 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f0cb402ff60 0x7f0cb4032420 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:08:43.755 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:43.755+0000 7f0cd1c08700 1 --2- 172.21.15.160:0/2539509279 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0ccc0f5410 0x7f0ccc105600 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:08:43.755 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:43.755+0000 7f0cd1c08700 1 -- 172.21.15.160:0/2539509279 >> 172.21.15.160:0/2539509279 conn(0x7f0ccc0f0b70 msgr2=0x7f0ccc0f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:08:43.755 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:43.755+0000 7f0cd1c08700 1 -- 172.21.15.160:0/2539509279 shutdown_connections 2023-12-25T20:08:43.755 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:43.755+0000 7f0cd1c08700 1 -- 172.21.15.160:0/2539509279 wait complete. 2023-12-25T20:08:43.756 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:08:44.469 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:08:44 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:08:42.635204+0000 mgr.smithi050.nbzhun (mgr.14162) 45 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:08:44.469 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:08:44 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:08:43.753993+0000 mon.smithi050 (mon.0) 140 : audit [DBG] from='client.? 172.21.15.160:0/2539509279' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:08:45.184 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:08:45.184 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:08:46.386 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:46.385+0000 7f46453aa700 1 -- 172.21.15.160:0/2391064072 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4640060280 msgr2=0x7f46400f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:08:46.386 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:46.385+0000 7f46453aa700 1 --2- 172.21.15.160:0/2391064072 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4640060280 0x7f46400f90f0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f4634009fb0 tx=0x7f4634009b30).stop 2023-12-25T20:08:46.386 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:46.385+0000 7f46453aa700 1 -- 172.21.15.160:0/2391064072 shutdown_connections 2023-12-25T20:08:46.386 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:46.385+0000 7f46453aa700 1 --2- 172.21.15.160:0/2391064072 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4640060280 0x7f46400f90f0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:08:46.386 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:46.385+0000 7f46453aa700 1 -- 172.21.15.160:0/2391064072 >> 172.21.15.160:0/2391064072 conn(0x7f46400f0b70 msgr2=0x7f46400f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:08:46.386 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:46.385+0000 7f46453aa700 1 -- 172.21.15.160:0/2391064072 shutdown_connections 2023-12-25T20:08:46.386 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:46.385+0000 7f46453aa700 1 -- 172.21.15.160:0/2391064072 wait complete. 2023-12-25T20:08:46.386 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:46.385+0000 7f46453aa700 1 Processor -- start 2023-12-25T20:08:46.386 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:46.386+0000 7f46453aa700 1 -- start start 2023-12-25T20:08:46.387 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:46.386+0000 7f46453aa700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4640060280 0x7f46400f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:08:46.387 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:46.386+0000 7f46453aa700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46400f4ff0 con 0x7f4640060280 2023-12-25T20:08:46.388 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:46.386+0000 7f463effd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4640060280 0x7f46400f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:08:46.388 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:46.386+0000 7f463effd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4640060280 0x7f46400f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:33814/0 (socket says 172.21.15.160:33814) 2023-12-25T20:08:46.388 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:46.386+0000 7f463effd700 1 -- 172.21.15.160:0/1052496430 learned_addr learned my addr 172.21.15.160:0/1052496430 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:08:46.388 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:46.386+0000 7f463effd700 1 -- 172.21.15.160:0/1052496430 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4634005040 con 0x7f4640060280 2023-12-25T20:08:46.389 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:46.387+0000 7f463effd700 1 --2- 172.21.15.160:0/1052496430 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4640060280 0x7f46400f6b70 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f4634013510 tx=0x7f463400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:08:46.389 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:46.387+0000 7f46277fe700 1 -- 172.21.15.160:0/1052496430 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4634009320 con 0x7f4640060280 2023-12-25T20:08:46.389 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:46.387+0000 7f46453aa700 1 -- 172.21.15.160:0/1052496430 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46400f5190 con 0x7f4640060280 2023-12-25T20:08:46.389 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:46.387+0000 7f46277fe700 1 -- 172.21.15.160:0/1052496430 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4634013f00 con 0x7f4640060280 2023-12-25T20:08:46.389 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:46.387+0000 7f46277fe700 1 -- 172.21.15.160:0/1052496430 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4634014dc0 con 0x7f4640060280 2023-12-25T20:08:46.389 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:46.387+0000 7f46453aa700 1 -- 172.21.15.160:0/1052496430 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46400f5620 con 0x7f4640060280 2023-12-25T20:08:46.389 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:46.388+0000 7f46453aa700 1 -- 172.21.15.160:0/1052496430 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4640059100 con 0x7f4640060280 2023-12-25T20:08:46.389 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:46.388+0000 7f46277fe700 1 -- 172.21.15.160:0/1052496430 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4634014420 con 0x7f4640060280 2023-12-25T20:08:46.389 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:46.388+0000 7f46277fe700 1 --2- 172.21.15.160:0/1052496430 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f462802ffb0 0x7f4628032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:08:46.389 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:46.388+0000 7f46277fe700 1 -- 172.21.15.160:0/1052496430 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4634040fe0 con 0x7f4640060280 2023-12-25T20:08:46.389 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:46.388+0000 7f463e7fc700 1 --2- 172.21.15.160:0/1052496430 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f462802ffb0 0x7f4628032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:08:46.389 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:46.389+0000 7f463e7fc700 1 --2- 172.21.15.160:0/1052496430 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f462802ffb0 0x7f4628032470 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f46300096a0 tx=0x7f4630006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:08:46.392 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:46.393+0000 7f46277fe700 1 -- 172.21.15.160:0/1052496430 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f463406d2b0 con 0x7f4640060280 2023-12-25T20:08:46.469 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:08:46 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:08:44.635517+0000 mgr.smithi050.nbzhun (mgr.14162) 46 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:08:46.598 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:46.598+0000 7f46453aa700 1 -- 172.21.15.160:0/1052496430 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f46400fbf50 con 0x7f4640060280 2023-12-25T20:08:46.599 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:46.599+0000 7f46277fe700 1 -- 172.21.15.160:0/1052496430 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f46340095f0 con 0x7f4640060280 2023-12-25T20:08:46.599 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:08:46.599 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:08:46.600 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:46.600+0000 7f46453aa700 1 -- 172.21.15.160:0/1052496430 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f462802ffb0 msgr2=0x7f4628032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:08:46.600 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:46.600+0000 7f46453aa700 1 --2- 172.21.15.160:0/1052496430 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f462802ffb0 0x7f4628032470 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f46300096a0 tx=0x7f4630006b40).stop 2023-12-25T20:08:46.600 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:46.600+0000 7f46453aa700 1 -- 172.21.15.160:0/1052496430 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4640060280 msgr2=0x7f46400f6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:08:46.601 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:46.600+0000 7f46453aa700 1 --2- 172.21.15.160:0/1052496430 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4640060280 0x7f46400f6b70 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f4634013510 tx=0x7f463400b040).stop 2023-12-25T20:08:46.601 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:46.600+0000 7f46453aa700 1 -- 172.21.15.160:0/1052496430 shutdown_connections 2023-12-25T20:08:46.601 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:46.600+0000 7f46453aa700 1 --2- 172.21.15.160:0/1052496430 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f462802ffb0 0x7f4628032470 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:08:46.601 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:46.600+0000 7f46453aa700 1 --2- 172.21.15.160:0/1052496430 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4640060280 0x7f46400f6b70 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:08:46.601 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:46.600+0000 7f46453aa700 1 -- 172.21.15.160:0/1052496430 >> 172.21.15.160:0/1052496430 conn(0x7f46400f0b70 msgr2=0x7f46400f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:08:46.601 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:46.600+0000 7f46453aa700 1 -- 172.21.15.160:0/1052496430 shutdown_connections 2023-12-25T20:08:46.601 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:46.600+0000 7f46453aa700 1 -- 172.21.15.160:0/1052496430 wait complete. 2023-12-25T20:08:46.601 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:08:47.469 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:08:47 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:08:46.599459+0000 mon.smithi050 (mon.0) 141 : audit [DBG] from='client.? 172.21.15.160:0/1052496430' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:08:48.022 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:08:48.022 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:08:48.468 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:08:48 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:08:46.635810+0000 mgr.smithi050.nbzhun (mgr.14162) 47 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:08:49.220 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:49.218+0000 7f102a093700 1 -- 172.21.15.160:0/3700303100 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1024060140 msgr2=0x7f10240f8fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:08:49.220 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:49.218+0000 7f102a093700 1 --2- 172.21.15.160:0/3700303100 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1024060140 0x7f10240f8fc0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f1020009fb0 tx=0x7f1020009b30).stop 2023-12-25T20:08:49.220 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:49.218+0000 7f102a093700 1 -- 172.21.15.160:0/3700303100 shutdown_connections 2023-12-25T20:08:49.220 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:49.218+0000 7f102a093700 1 --2- 172.21.15.160:0/3700303100 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1024060140 0x7f10240f8fc0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:08:49.220 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:49.218+0000 7f102a093700 1 -- 172.21.15.160:0/3700303100 >> 172.21.15.160:0/3700303100 conn(0x7f10240f09a0 msgr2=0x7f10240f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:08:49.220 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:49.218+0000 7f102a093700 1 -- 172.21.15.160:0/3700303100 shutdown_connections 2023-12-25T20:08:49.220 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:49.218+0000 7f102a093700 1 -- 172.21.15.160:0/3700303100 wait complete. 2023-12-25T20:08:49.220 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:49.218+0000 7f102a093700 1 Processor -- start 2023-12-25T20:08:49.220 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:49.218+0000 7f102a093700 1 -- start start 2023-12-25T20:08:49.220 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:49.218+0000 7f102a093700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1024060140 0x7f1024106520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:08:49.220 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:49.219+0000 7f102a093700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1024106a10 con 0x7f1024060140 2023-12-25T20:08:49.221 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:49.219+0000 7f1029091700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1024060140 0x7f1024106520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:08:49.221 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:49.219+0000 7f1029091700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1024060140 0x7f1024106520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:39964/0 (socket says 172.21.15.160:39964) 2023-12-25T20:08:49.221 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:49.219+0000 7f1029091700 1 -- 172.21.15.160:0/2822322094 learned_addr learned my addr 172.21.15.160:0/2822322094 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:08:49.221 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:49.219+0000 7f1029091700 1 -- 172.21.15.160:0/2822322094 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1020005040 con 0x7f1024060140 2023-12-25T20:08:49.221 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:49.219+0000 7f1029091700 1 --2- 172.21.15.160:0/2822322094 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1024060140 0x7f1024106520 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f1020013510 tx=0x7f102000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:08:49.221 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:49.219+0000 7f1019ffb700 1 -- 172.21.15.160:0/2822322094 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1020009320 con 0x7f1024060140 2023-12-25T20:08:49.221 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:49.219+0000 7f1019ffb700 1 -- 172.21.15.160:0/2822322094 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1020013f00 con 0x7f1024060140 2023-12-25T20:08:49.221 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:49.219+0000 7f102a093700 1 -- 172.21.15.160:0/2822322094 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1024106c10 con 0x7f1024060140 2023-12-25T20:08:49.221 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:49.219+0000 7f1019ffb700 1 -- 172.21.15.160:0/2822322094 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1020014d90 con 0x7f1024060140 2023-12-25T20:08:49.221 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:49.220+0000 7f102a093700 1 -- 172.21.15.160:0/2822322094 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10241070a0 con 0x7f1024060140 2023-12-25T20:08:49.222 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:49.220+0000 7f1019ffb700 1 -- 172.21.15.160:0/2822322094 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f10200095f0 con 0x7f1024060140 2023-12-25T20:08:49.222 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:49.221+0000 7f102a093700 1 -- 172.21.15.160:0/2822322094 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1024045ad0 con 0x7f1024060140 2023-12-25T20:08:49.222 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:49.221+0000 7f1019ffb700 1 --2- 172.21.15.160:0/2822322094 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f101002ffb0 0x7f1010032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:08:49.222 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:49.221+0000 7f1019ffb700 1 -- 172.21.15.160:0/2822322094 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f10200410c0 con 0x7f1024060140 2023-12-25T20:08:49.222 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:49.221+0000 7f1028890700 1 --2- 172.21.15.160:0/2822322094 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f101002ffb0 0x7f1010032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:08:49.222 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:49.221+0000 7f1028890700 1 --2- 172.21.15.160:0/2822322094 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f101002ffb0 0x7f1010032470 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f10140096a0 tx=0x7f1014006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:08:49.225 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:49.226+0000 7f1019ffb700 1 -- 172.21.15.160:0/2822322094 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1020014420 con 0x7f1024060140 2023-12-25T20:08:49.430 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:49.430+0000 7f102a093700 1 -- 172.21.15.160:0/2822322094 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1024107f50 con 0x7f1024060140 2023-12-25T20:08:49.431 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:49.431+0000 7f1019ffb700 1 -- 172.21.15.160:0/2822322094 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f102001d7f0 con 0x7f1024060140 2023-12-25T20:08:49.431 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:08:49.431 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:08:49.433 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:49.433+0000 7f102a093700 1 -- 172.21.15.160:0/2822322094 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f101002ffb0 msgr2=0x7f1010032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:08:49.433 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:49.433+0000 7f102a093700 1 --2- 172.21.15.160:0/2822322094 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f101002ffb0 0x7f1010032470 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f10140096a0 tx=0x7f1014006b40).stop 2023-12-25T20:08:49.433 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:49.433+0000 7f102a093700 1 -- 172.21.15.160:0/2822322094 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1024060140 msgr2=0x7f1024106520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:08:49.433 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:49.433+0000 7f102a093700 1 --2- 172.21.15.160:0/2822322094 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1024060140 0x7f1024106520 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f1020013510 tx=0x7f102000b040).stop 2023-12-25T20:08:49.433 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:49.433+0000 7f102a093700 1 -- 172.21.15.160:0/2822322094 shutdown_connections 2023-12-25T20:08:49.433 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:49.433+0000 7f102a093700 1 --2- 172.21.15.160:0/2822322094 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f101002ffb0 0x7f1010032470 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:08:49.434 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:49.433+0000 7f102a093700 1 --2- 172.21.15.160:0/2822322094 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1024060140 0x7f1024106520 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:08:49.434 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:49.433+0000 7f102a093700 1 -- 172.21.15.160:0/2822322094 >> 172.21.15.160:0/2822322094 conn(0x7f10240f09a0 msgr2=0x7f10240f14d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:08:49.434 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:49.433+0000 7f102a093700 1 -- 172.21.15.160:0/2822322094 shutdown_connections 2023-12-25T20:08:49.434 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:49.433+0000 7f102a093700 1 -- 172.21.15.160:0/2822322094 wait complete. 2023-12-25T20:08:49.434 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:08:50.469 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:08:50 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:08:48.636183+0000 mgr.smithi050.nbzhun (mgr.14162) 48 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:08:50.469 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:08:50 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:08:49.431691+0000 mon.smithi050 (mon.0) 142 : audit [DBG] from='client.? 172.21.15.160:0/2822322094' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:08:50.846 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:08:50.846 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:08:52.062 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:52.060+0000 7f86dfda8700 1 -- 172.21.15.160:0/1305343309 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f86d80f5410 msgr2=0x7f86d80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:08:52.062 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:52.060+0000 7f86dfda8700 1 --2- 172.21.15.160:0/1305343309 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f86d80f5410 0x7f86d80f5830 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f86cc009fb0 tx=0x7f86cc009b30).stop 2023-12-25T20:08:52.062 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:52.060+0000 7f86dfda8700 1 -- 172.21.15.160:0/1305343309 shutdown_connections 2023-12-25T20:08:52.062 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:52.060+0000 7f86dfda8700 1 --2- 172.21.15.160:0/1305343309 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f86d80f5410 0x7f86d80f5830 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:08:52.062 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:52.060+0000 7f86dfda8700 1 -- 172.21.15.160:0/1305343309 >> 172.21.15.160:0/1305343309 conn(0x7f86d80f0b70 msgr2=0x7f86d80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:08:52.062 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:52.060+0000 7f86dfda8700 1 -- 172.21.15.160:0/1305343309 shutdown_connections 2023-12-25T20:08:52.062 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:52.060+0000 7f86dfda8700 1 -- 172.21.15.160:0/1305343309 wait complete. 2023-12-25T20:08:52.062 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:52.061+0000 7f86dfda8700 1 Processor -- start 2023-12-25T20:08:52.062 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:52.061+0000 7f86dfda8700 1 -- start start 2023-12-25T20:08:52.063 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:52.061+0000 7f86dfda8700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f86d80f5410 0x7f86d8109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:08:52.063 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:52.061+0000 7f86dfda8700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86d8060220 con 0x7f86d80f5410 2023-12-25T20:08:52.063 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:52.061+0000 7f86ddb44700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f86d80f5410 0x7f86d8109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:08:52.063 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:52.061+0000 7f86ddb44700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f86d80f5410 0x7f86d8109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:39982/0 (socket says 172.21.15.160:39982) 2023-12-25T20:08:52.063 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:52.061+0000 7f86ddb44700 1 -- 172.21.15.160:0/3617254703 learned_addr learned my addr 172.21.15.160:0/3617254703 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:08:52.063 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:52.062+0000 7f86ddb44700 1 -- 172.21.15.160:0/3617254703 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86cc005040 con 0x7f86d80f5410 2023-12-25T20:08:52.063 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:52.062+0000 7f86ddb44700 1 --2- 172.21.15.160:0/3617254703 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f86d80f5410 0x7f86d8109450 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f86cc013510 tx=0x7f86cc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:08:52.063 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:52.062+0000 7f86ca7fc700 1 -- 172.21.15.160:0/3617254703 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f86cc009320 con 0x7f86d80f5410 2023-12-25T20:08:52.063 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:52.062+0000 7f86dfda8700 1 -- 172.21.15.160:0/3617254703 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86d8060420 con 0x7f86d80f5410 2023-12-25T20:08:52.063 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:52.062+0000 7f86ca7fc700 1 -- 172.21.15.160:0/3617254703 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f86cc013f00 con 0x7f86d80f5410 2023-12-25T20:08:52.063 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:52.062+0000 7f86ca7fc700 1 -- 172.21.15.160:0/3617254703 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f86cc014d90 con 0x7f86d80f5410 2023-12-25T20:08:52.064 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:52.062+0000 7f86dfda8700 1 -- 172.21.15.160:0/3617254703 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86d8109b00 con 0x7f86d80f5410 2023-12-25T20:08:52.064 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:52.063+0000 7f86ca7fc700 1 -- 172.21.15.160:0/3617254703 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f86cc0263a0 con 0x7f86d80f5410 2023-12-25T20:08:52.064 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:52.063+0000 7f86ca7fc700 1 --2- 172.21.15.160:0/3617254703 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f86c402ff60 0x7f86c4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:08:52.064 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:52.063+0000 7f86ca7fc700 1 -- 172.21.15.160:0/3617254703 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f86cc040ff0 con 0x7f86d80f5410 2023-12-25T20:08:52.065 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:52.063+0000 7f86dfda8700 1 -- 172.21.15.160:0/3617254703 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86d8059100 con 0x7f86d80f5410 2023-12-25T20:08:52.065 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:52.063+0000 7f86dd343700 1 --2- 172.21.15.160:0/3617254703 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f86c402ff60 0x7f86c4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:08:52.065 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:52.063+0000 7f86dd343700 1 --2- 172.21.15.160:0/3617254703 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f86c402ff60 0x7f86c4032420 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f86d40096a0 tx=0x7f86d4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:08:52.067 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:52.068+0000 7f86ca7fc700 1 -- 172.21.15.160:0/3617254703 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f86cc012920 con 0x7f86d80f5410 2023-12-25T20:08:52.274 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:52.274+0000 7f86dfda8700 1 -- 172.21.15.160:0/3617254703 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f86d810a1d0 con 0x7f86d80f5410 2023-12-25T20:08:52.275 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:52.275+0000 7f86ca7fc700 1 -- 172.21.15.160:0/3617254703 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f86cc02b020 con 0x7f86d80f5410 2023-12-25T20:08:52.275 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:08:52.275 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:08:52.277 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:52.276+0000 7f86dfda8700 1 -- 172.21.15.160:0/3617254703 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f86c402ff60 msgr2=0x7f86c4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:08:52.277 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:52.276+0000 7f86dfda8700 1 --2- 172.21.15.160:0/3617254703 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f86c402ff60 0x7f86c4032420 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f86d40096a0 tx=0x7f86d4006b40).stop 2023-12-25T20:08:52.277 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:52.276+0000 7f86dfda8700 1 -- 172.21.15.160:0/3617254703 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f86d80f5410 msgr2=0x7f86d8109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:08:52.277 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:52.276+0000 7f86dfda8700 1 --2- 172.21.15.160:0/3617254703 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f86d80f5410 0x7f86d8109450 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f86cc013510 tx=0x7f86cc00b040).stop 2023-12-25T20:08:52.277 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:52.277+0000 7f86dfda8700 1 -- 172.21.15.160:0/3617254703 shutdown_connections 2023-12-25T20:08:52.277 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:52.277+0000 7f86dfda8700 1 --2- 172.21.15.160:0/3617254703 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f86c402ff60 0x7f86c4032420 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:08:52.277 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:52.277+0000 7f86dfda8700 1 --2- 172.21.15.160:0/3617254703 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f86d80f5410 0x7f86d8109450 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:08:52.277 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:52.277+0000 7f86dfda8700 1 -- 172.21.15.160:0/3617254703 >> 172.21.15.160:0/3617254703 conn(0x7f86d80f0b70 msgr2=0x7f86d80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:08:52.277 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:52.277+0000 7f86dfda8700 1 -- 172.21.15.160:0/3617254703 shutdown_connections 2023-12-25T20:08:52.277 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:52.277+0000 7f86dfda8700 1 -- 172.21.15.160:0/3617254703 wait complete. 2023-12-25T20:08:52.278 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:08:52.468 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:08:52 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:08:50.636469+0000 mgr.smithi050.nbzhun (mgr.14162) 49 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:08:53.468 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:08:53 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:08:52.275616+0000 mon.smithi050 (mon.0) 143 : audit [DBG] from='client.? 172.21.15.160:0/3617254703' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:08:53.838 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:08:53.839 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:08:54.468 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:08:54 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:08:52.636817+0000 mgr.smithi050.nbzhun (mgr.14162) 50 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:08:55.070 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:55.069+0000 7fa5fd70f700 1 -- 172.21.15.160:0/1718403002 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa5f80f5410 msgr2=0x7fa5f80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:08:55.071 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:55.069+0000 7fa5fd70f700 1 --2- 172.21.15.160:0/1718403002 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa5f80f5410 0x7fa5f80f5830 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7fa5ec009fb0 tx=0x7fa5ec009b30).stop 2023-12-25T20:08:55.071 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:55.069+0000 7fa5fd70f700 1 -- 172.21.15.160:0/1718403002 shutdown_connections 2023-12-25T20:08:55.071 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:55.069+0000 7fa5fd70f700 1 --2- 172.21.15.160:0/1718403002 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa5f80f5410 0x7fa5f80f5830 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:08:55.071 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:55.069+0000 7fa5fd70f700 1 -- 172.21.15.160:0/1718403002 >> 172.21.15.160:0/1718403002 conn(0x7fa5f80f0b70 msgr2=0x7fa5f80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:08:55.071 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:55.069+0000 7fa5fd70f700 1 -- 172.21.15.160:0/1718403002 shutdown_connections 2023-12-25T20:08:55.071 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:55.070+0000 7fa5fd70f700 1 -- 172.21.15.160:0/1718403002 wait complete. 2023-12-25T20:08:55.071 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:55.070+0000 7fa5fd70f700 1 Processor -- start 2023-12-25T20:08:55.071 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:55.070+0000 7fa5fd70f700 1 -- start start 2023-12-25T20:08:55.071 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:55.070+0000 7fa5fd70f700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa5f80f5410 0x7fa5f8109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:08:55.071 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:55.070+0000 7fa5fd70f700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5f8060240 con 0x7fa5f80f5410 2023-12-25T20:08:55.071 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:55.071+0000 7fa5f6ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa5f80f5410 0x7fa5f8109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:08:55.071 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:55.071+0000 7fa5f6ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa5f80f5410 0x7fa5f8109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:40000/0 (socket says 172.21.15.160:40000) 2023-12-25T20:08:55.071 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:55.071+0000 7fa5f6ffd700 1 -- 172.21.15.160:0/1498696781 learned_addr learned my addr 172.21.15.160:0/1498696781 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:08:55.072 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:55.071+0000 7fa5f6ffd700 1 -- 172.21.15.160:0/1498696781 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5ec005040 con 0x7fa5f80f5410 2023-12-25T20:08:55.073 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:55.071+0000 7fa5f6ffd700 1 --2- 172.21.15.160:0/1498696781 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa5f80f5410 0x7fa5f8109430 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7fa5ec014000 tx=0x7fa5ec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:08:55.073 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:55.071+0000 7fa5df7fe700 1 -- 172.21.15.160:0/1498696781 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa5ec0095f0 con 0x7fa5f80f5410 2023-12-25T20:08:55.073 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:55.071+0000 7fa5df7fe700 1 -- 172.21.15.160:0/1498696781 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa5ec007e70 con 0x7fa5f80f5410 2023-12-25T20:08:55.073 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:55.071+0000 7fa5fd70f700 1 -- 172.21.15.160:0/1498696781 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5f8060440 con 0x7fa5f80f5410 2023-12-25T20:08:55.073 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:55.071+0000 7fa5df7fe700 1 -- 172.21.15.160:0/1498696781 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa5ec014ea0 con 0x7fa5f80f5410 2023-12-25T20:08:55.074 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:55.071+0000 7fa5fd70f700 1 -- 172.21.15.160:0/1498696781 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5f8109ae0 con 0x7fa5f80f5410 2023-12-25T20:08:55.074 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:55.072+0000 7fa5df7fe700 1 -- 172.21.15.160:0/1498696781 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa5ec025070 con 0x7fa5f80f5410 2023-12-25T20:08:55.074 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:55.072+0000 7fa5fd70f700 1 -- 172.21.15.160:0/1498696781 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5f8045ad0 con 0x7fa5f80f5410 2023-12-25T20:08:55.074 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:55.072+0000 7fa5df7fe700 1 --2- 172.21.15.160:0/1498696781 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fa5e002ff60 0x7fa5e0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:08:55.074 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:55.072+0000 7fa5df7fe700 1 -- 172.21.15.160:0/1498696781 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa5ec040980 con 0x7fa5f80f5410 2023-12-25T20:08:55.074 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:55.073+0000 7fa5f67fc700 1 --2- 172.21.15.160:0/1498696781 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fa5e002ff60 0x7fa5e0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:08:55.074 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:55.073+0000 7fa5f67fc700 1 --2- 172.21.15.160:0/1498696781 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fa5e002ff60 0x7fa5e0032420 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7fa5e80096a0 tx=0x7fa5e8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:08:55.077 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:55.077+0000 7fa5df7fe700 1 -- 172.21.15.160:0/1498696781 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa5ec0079c0 con 0x7fa5f80f5410 2023-12-25T20:08:55.280 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:55.281+0000 7fa5fd70f700 1 -- 172.21.15.160:0/1498696781 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa5f80fae60 con 0x7fa5f80f5410 2023-12-25T20:08:55.281 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:55.281+0000 7fa5df7fe700 1 -- 172.21.15.160:0/1498696781 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa5ec00f300 con 0x7fa5f80f5410 2023-12-25T20:08:55.281 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:08:55.282 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:08:55.283 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:55.283+0000 7fa5fd70f700 1 -- 172.21.15.160:0/1498696781 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fa5e002ff60 msgr2=0x7fa5e0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:08:55.283 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:55.283+0000 7fa5fd70f700 1 --2- 172.21.15.160:0/1498696781 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fa5e002ff60 0x7fa5e0032420 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7fa5e80096a0 tx=0x7fa5e8006b40).stop 2023-12-25T20:08:55.283 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:55.283+0000 7fa5fd70f700 1 -- 172.21.15.160:0/1498696781 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa5f80f5410 msgr2=0x7fa5f8109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:08:55.283 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:55.283+0000 7fa5fd70f700 1 --2- 172.21.15.160:0/1498696781 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa5f80f5410 0x7fa5f8109430 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7fa5ec014000 tx=0x7fa5ec00b040).stop 2023-12-25T20:08:55.283 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:55.283+0000 7fa5fd70f700 1 -- 172.21.15.160:0/1498696781 shutdown_connections 2023-12-25T20:08:55.283 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:55.283+0000 7fa5fd70f700 1 --2- 172.21.15.160:0/1498696781 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fa5e002ff60 0x7fa5e0032420 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:08:55.283 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:55.283+0000 7fa5fd70f700 1 --2- 172.21.15.160:0/1498696781 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa5f80f5410 0x7fa5f8109430 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:08:55.283 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:55.283+0000 7fa5fd70f700 1 -- 172.21.15.160:0/1498696781 >> 172.21.15.160:0/1498696781 conn(0x7fa5f80f0b70 msgr2=0x7fa5f80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:08:55.283 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:55.283+0000 7fa5fd70f700 1 -- 172.21.15.160:0/1498696781 shutdown_connections 2023-12-25T20:08:55.284 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:55.283+0000 7fa5fd70f700 1 -- 172.21.15.160:0/1498696781 wait complete. 2023-12-25T20:08:55.284 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:08:56.469 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:08:56 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:08:54.637132+0000 mgr.smithi050.nbzhun (mgr.14162) 51 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:08:56.469 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:08:56 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:08:55.282112+0000 mon.smithi050 (mon.0) 144 : audit [DBG] from='client.? 172.21.15.160:0/1498696781' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:08:56.693 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:08:56.693 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:08:57.908 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:57.908+0000 7fdaf0e72700 1 -- 172.21.15.160:0/427559688 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdaec0f51a0 msgr2=0x7fdaec0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:08:57.908 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:57.908+0000 7fdaf0e72700 1 --2- 172.21.15.160:0/427559688 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdaec0f51a0 0x7fdaec0f55c0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7fdadc009fb0 tx=0x7fdadc009b30).stop 2023-12-25T20:08:57.908 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:57.908+0000 7fdaf0e72700 1 -- 172.21.15.160:0/427559688 shutdown_connections 2023-12-25T20:08:57.908 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:57.908+0000 7fdaf0e72700 1 --2- 172.21.15.160:0/427559688 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdaec0f51a0 0x7fdaec0f55c0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:08:57.909 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:57.908+0000 7fdaf0e72700 1 -- 172.21.15.160:0/427559688 >> 172.21.15.160:0/427559688 conn(0x7fdaec0f09a0 msgr2=0x7fdaec0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:08:57.909 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:57.908+0000 7fdaf0e72700 1 -- 172.21.15.160:0/427559688 shutdown_connections 2023-12-25T20:08:57.909 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:57.908+0000 7fdaf0e72700 1 -- 172.21.15.160:0/427559688 wait complete. 2023-12-25T20:08:57.910 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:57.909+0000 7fdaf0e72700 1 Processor -- start 2023-12-25T20:08:57.910 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:57.909+0000 7fdaf0e72700 1 -- start start 2023-12-25T20:08:57.910 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:57.909+0000 7fdaf0e72700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdaec0f51a0 0x7fdaec1091d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:08:57.910 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:57.909+0000 7fdaf0e72700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdaec060220 con 0x7fdaec0f51a0 2023-12-25T20:08:57.910 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:57.909+0000 7fdaeb7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdaec0f51a0 0x7fdaec1091d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:08:57.910 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:57.909+0000 7fdaeb7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdaec0f51a0 0x7fdaec1091d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:37540/0 (socket says 172.21.15.160:37540) 2023-12-25T20:08:57.911 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:57.909+0000 7fdaeb7fe700 1 -- 172.21.15.160:0/2117373134 learned_addr learned my addr 172.21.15.160:0/2117373134 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:08:57.911 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:57.910+0000 7fdaeb7fe700 1 -- 172.21.15.160:0/2117373134 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdadc005040 con 0x7fdaec0f51a0 2023-12-25T20:08:57.911 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:57.910+0000 7fdaeb7fe700 1 --2- 172.21.15.160:0/2117373134 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdaec0f51a0 0x7fdaec1091d0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7fdadc013510 tx=0x7fdadc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:08:57.911 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:57.910+0000 7fdae3fff700 1 -- 172.21.15.160:0/2117373134 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdadc009320 con 0x7fdaec0f51a0 2023-12-25T20:08:57.911 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:57.910+0000 7fdaf0e72700 1 -- 172.21.15.160:0/2117373134 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdaec060420 con 0x7fdaec0f51a0 2023-12-25T20:08:57.911 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:57.910+0000 7fdae3fff700 1 -- 172.21.15.160:0/2117373134 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdadc013f00 con 0x7fdaec0f51a0 2023-12-25T20:08:57.911 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:57.910+0000 7fdae3fff700 1 -- 172.21.15.160:0/2117373134 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdadc014dc0 con 0x7fdaec0f51a0 2023-12-25T20:08:57.911 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:57.910+0000 7fdaf0e72700 1 -- 172.21.15.160:0/2117373134 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdaec109880 con 0x7fdaec0f51a0 2023-12-25T20:08:57.912 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:57.911+0000 7fdaf0e72700 1 -- 172.21.15.160:0/2117373134 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdaec059100 con 0x7fdaec0f51a0 2023-12-25T20:08:57.912 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:57.911+0000 7fdae3fff700 1 -- 172.21.15.160:0/2117373134 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdadc0095f0 con 0x7fdaec0f51a0 2023-12-25T20:08:57.912 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:57.911+0000 7fdae3fff700 1 --2- 172.21.15.160:0/2117373134 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fdad802ff60 0x7fdad8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:08:57.912 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:57.911+0000 7fdae3fff700 1 -- 172.21.15.160:0/2117373134 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdadc041100 con 0x7fdaec0f51a0 2023-12-25T20:08:57.912 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:57.911+0000 7fdaeaffd700 1 --2- 172.21.15.160:0/2117373134 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fdad802ff60 0x7fdad8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:08:57.912 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:57.912+0000 7fdaeaffd700 1 --2- 172.21.15.160:0/2117373134 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fdad802ff60 0x7fdad8032420 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7fdad40096a0 tx=0x7fdad4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:08:57.915 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:57.916+0000 7fdae3fff700 1 -- 172.21.15.160:0/2117373134 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdadc014420 con 0x7fdaec0f51a0 2023-12-25T20:08:58.120 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:58.120+0000 7fdaf0e72700 1 -- 172.21.15.160:0/2117373134 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdaec045ad0 con 0x7fdaec0f51a0 2023-12-25T20:08:58.121 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:58.121+0000 7fdae3fff700 1 -- 172.21.15.160:0/2117373134 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fdadc010230 con 0x7fdaec0f51a0 2023-12-25T20:08:58.121 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:08:58.121 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:08:58.123 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:58.123+0000 7fdaf0e72700 1 -- 172.21.15.160:0/2117373134 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fdad802ff60 msgr2=0x7fdad8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:08:58.123 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:58.123+0000 7fdaf0e72700 1 --2- 172.21.15.160:0/2117373134 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fdad802ff60 0x7fdad8032420 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7fdad40096a0 tx=0x7fdad4006b40).stop 2023-12-25T20:08:58.123 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:58.123+0000 7fdaf0e72700 1 -- 172.21.15.160:0/2117373134 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdaec0f51a0 msgr2=0x7fdaec1091d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:08:58.123 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:58.123+0000 7fdaf0e72700 1 --2- 172.21.15.160:0/2117373134 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdaec0f51a0 0x7fdaec1091d0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7fdadc013510 tx=0x7fdadc00b040).stop 2023-12-25T20:08:58.123 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:58.123+0000 7fdaf0e72700 1 -- 172.21.15.160:0/2117373134 shutdown_connections 2023-12-25T20:08:58.123 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:58.123+0000 7fdaf0e72700 1 --2- 172.21.15.160:0/2117373134 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fdad802ff60 0x7fdad8032420 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:08:58.123 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:58.123+0000 7fdaf0e72700 1 --2- 172.21.15.160:0/2117373134 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdaec0f51a0 0x7fdaec1091d0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:08:58.123 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:58.123+0000 7fdaf0e72700 1 -- 172.21.15.160:0/2117373134 >> 172.21.15.160:0/2117373134 conn(0x7fdaec0f09a0 msgr2=0x7fdaec0f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:08:58.123 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:58.123+0000 7fdaf0e72700 1 -- 172.21.15.160:0/2117373134 shutdown_connections 2023-12-25T20:08:58.123 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:08:58.123+0000 7fdaf0e72700 1 -- 172.21.15.160:0/2117373134 wait complete. 2023-12-25T20:08:58.124 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:08:58.468 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:08:58 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:08:56.637480+0000 mgr.smithi050.nbzhun (mgr.14162) 52 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:08:59.469 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:08:59 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:08:58.121783+0000 mon.smithi050 (mon.0) 145 : audit [DBG] from='client.? 172.21.15.160:0/2117373134' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:08:59.509 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:08:59.509 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:09:00.468 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:09:00 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:08:58.637835+0000 mgr.smithi050.nbzhun (mgr.14162) 53 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:09:00.725 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:00.723+0000 7f8ace585700 1 -- 172.21.15.160:0/2619844761 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8ac80f51a0 msgr2=0x7f8ac80f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:09:00.726 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:00.723+0000 7f8ace585700 1 --2- 172.21.15.160:0/2619844761 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8ac80f51a0 0x7f8ac80f55c0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f8ab8009fb0 tx=0x7f8ab8009b30).stop 2023-12-25T20:09:00.726 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:00.724+0000 7f8ace585700 1 -- 172.21.15.160:0/2619844761 shutdown_connections 2023-12-25T20:09:00.726 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:00.724+0000 7f8ace585700 1 --2- 172.21.15.160:0/2619844761 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8ac80f51a0 0x7f8ac80f55c0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:09:00.726 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:00.724+0000 7f8ace585700 1 -- 172.21.15.160:0/2619844761 >> 172.21.15.160:0/2619844761 conn(0x7f8ac80f09a0 msgr2=0x7f8ac80f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:09:00.726 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:00.724+0000 7f8ace585700 1 -- 172.21.15.160:0/2619844761 shutdown_connections 2023-12-25T20:09:00.726 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:00.724+0000 7f8ace585700 1 -- 172.21.15.160:0/2619844761 wait complete. 2023-12-25T20:09:00.726 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:00.724+0000 7f8ace585700 1 Processor -- start 2023-12-25T20:09:00.726 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:00.724+0000 7f8ace585700 1 -- start start 2023-12-25T20:09:00.726 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:00.724+0000 7f8ace585700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8ac80f51a0 0x7f8ac80fd120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:09:00.726 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:00.724+0000 7f8ace585700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ac80fb5a0 con 0x7f8ac80f51a0 2023-12-25T20:09:00.727 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:00.725+0000 7f8acd583700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8ac80f51a0 0x7f8ac80fd120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:09:00.727 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:00.725+0000 7f8acd583700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8ac80f51a0 0x7f8ac80fd120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:37550/0 (socket says 172.21.15.160:37550) 2023-12-25T20:09:00.727 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:00.725+0000 7f8acd583700 1 -- 172.21.15.160:0/2582870692 learned_addr learned my addr 172.21.15.160:0/2582870692 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:09:00.727 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:00.725+0000 7f8acd583700 1 -- 172.21.15.160:0/2582870692 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ab8005040 con 0x7f8ac80f51a0 2023-12-25T20:09:00.727 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:00.725+0000 7f8acd583700 1 --2- 172.21.15.160:0/2582870692 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8ac80f51a0 0x7f8ac80fd120 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f8ab8009830 tx=0x7f8ab800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:09:00.727 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:00.725+0000 7f8abdffb700 1 -- 172.21.15.160:0/2582870692 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8ab8009320 con 0x7f8ac80f51a0 2023-12-25T20:09:00.727 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:00.725+0000 7f8abdffb700 1 -- 172.21.15.160:0/2582870692 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8ab80136c0 con 0x7f8ac80f51a0 2023-12-25T20:09:00.727 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:00.725+0000 7f8ace585700 1 -- 172.21.15.160:0/2582870692 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ac80fb740 con 0x7f8ac80f51a0 2023-12-25T20:09:00.727 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:00.725+0000 7f8abdffb700 1 -- 172.21.15.160:0/2582870692 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8ab801ee90 con 0x7f8ac80f51a0 2023-12-25T20:09:00.727 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:00.725+0000 7f8ace585700 1 -- 172.21.15.160:0/2582870692 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ac80fbbd0 con 0x7f8ac80f51a0 2023-12-25T20:09:00.728 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:00.726+0000 7f8abdffb700 1 -- 172.21.15.160:0/2582870692 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8ab801e420 con 0x7f8ac80f51a0 2023-12-25T20:09:00.728 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:00.726+0000 7f8ace585700 1 -- 172.21.15.160:0/2582870692 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ac8059100 con 0x7f8ac80f51a0 2023-12-25T20:09:00.728 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:00.727+0000 7f8abdffb700 1 --2- 172.21.15.160:0/2582870692 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f8ab402ff10 0x7f8ab40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:09:00.728 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:00.727+0000 7f8abdffb700 1 -- 172.21.15.160:0/2582870692 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8ab8040f40 con 0x7f8ac80f51a0 2023-12-25T20:09:00.728 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:00.727+0000 7f8accd82700 1 --2- 172.21.15.160:0/2582870692 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f8ab402ff10 0x7f8ab40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:09:00.728 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:00.727+0000 7f8accd82700 1 --2- 172.21.15.160:0/2582870692 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f8ab402ff10 0x7f8ab40323d0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f8ac40096a0 tx=0x7f8ac4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:09:00.731 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:00.731+0000 7f8abdffb700 1 -- 172.21.15.160:0/2582870692 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8ab806d2b0 con 0x7f8ac80f51a0 2023-12-25T20:09:00.937 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:00.936+0000 7f8ace585700 1 -- 172.21.15.160:0/2582870692 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8ac80fabe0 con 0x7f8ac80f51a0 2023-12-25T20:09:00.937 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:00.938+0000 7f8abdffb700 1 -- 172.21.15.160:0/2582870692 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8ab80095f0 con 0x7f8ac80f51a0 2023-12-25T20:09:00.938 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:09:00.938 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:09:00.939 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:00.939+0000 7f8ace585700 1 -- 172.21.15.160:0/2582870692 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f8ab402ff10 msgr2=0x7f8ab40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:09:00.939 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:00.939+0000 7f8ace585700 1 --2- 172.21.15.160:0/2582870692 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f8ab402ff10 0x7f8ab40323d0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f8ac40096a0 tx=0x7f8ac4006b40).stop 2023-12-25T20:09:00.939 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:00.939+0000 7f8ace585700 1 -- 172.21.15.160:0/2582870692 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8ac80f51a0 msgr2=0x7f8ac80fd120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:09:00.939 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:00.939+0000 7f8ace585700 1 --2- 172.21.15.160:0/2582870692 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8ac80f51a0 0x7f8ac80fd120 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f8ab8009830 tx=0x7f8ab800b040).stop 2023-12-25T20:09:00.939 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:00.939+0000 7f8ace585700 1 -- 172.21.15.160:0/2582870692 shutdown_connections 2023-12-25T20:09:00.939 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:00.939+0000 7f8ace585700 1 --2- 172.21.15.160:0/2582870692 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f8ab402ff10 0x7f8ab40323d0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:09:00.939 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:00.939+0000 7f8ace585700 1 --2- 172.21.15.160:0/2582870692 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8ac80f51a0 0x7f8ac80fd120 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:09:00.940 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:00.939+0000 7f8ace585700 1 -- 172.21.15.160:0/2582870692 >> 172.21.15.160:0/2582870692 conn(0x7f8ac80f09a0 msgr2=0x7f8ac805fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:09:00.940 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:00.939+0000 7f8ace585700 1 -- 172.21.15.160:0/2582870692 shutdown_connections 2023-12-25T20:09:00.940 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:00.939+0000 7f8ace585700 1 -- 172.21.15.160:0/2582870692 wait complete. 2023-12-25T20:09:00.940 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:09:02.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:09:01 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:09:00.638170+0000 mgr.smithi050.nbzhun (mgr.14162) 54 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:09:02.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:09:01 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:09:00.938271+0000 mon.smithi050 (mon.0) 146 : audit [DBG] from='client.? 172.21.15.160:0/2582870692' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:09:02.331 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:09:02.331 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:09:03.556 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:03.556+0000 7f1e10f34700 1 -- 172.21.15.160:0/612388322 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1e0c060220 msgr2=0x7f1e0c0f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:09:03.557 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:03.556+0000 7f1e10f34700 1 --2- 172.21.15.160:0/612388322 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1e0c060220 0x7f1e0c0f9150 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f1e00009fb0 tx=0x7f1e00009b30).stop 2023-12-25T20:09:03.557 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:03.556+0000 7f1e10f34700 1 -- 172.21.15.160:0/612388322 shutdown_connections 2023-12-25T20:09:03.557 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:03.556+0000 7f1e10f34700 1 --2- 172.21.15.160:0/612388322 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1e0c060220 0x7f1e0c0f9150 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:09:03.557 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:03.556+0000 7f1e10f34700 1 -- 172.21.15.160:0/612388322 >> 172.21.15.160:0/612388322 conn(0x7f1e0c0f0b70 msgr2=0x7f1e0c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:09:03.557 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:03.556+0000 7f1e10f34700 1 -- 172.21.15.160:0/612388322 shutdown_connections 2023-12-25T20:09:03.557 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:03.556+0000 7f1e10f34700 1 -- 172.21.15.160:0/612388322 wait complete. 2023-12-25T20:09:03.557 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:03.557+0000 7f1e10f34700 1 Processor -- start 2023-12-25T20:09:03.557 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:03.557+0000 7f1e10f34700 1 -- start start 2023-12-25T20:09:03.557 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:03.557+0000 7f1e10f34700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1e0c060220 0x7f1e0c0f6bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:09:03.557 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:03.557+0000 7f1e10f34700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e0c0f5050 con 0x7f1e0c060220 2023-12-25T20:09:03.559 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:03.557+0000 7f1e0a59c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1e0c060220 0x7f1e0c0f6bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:09:03.559 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:03.557+0000 7f1e0a59c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1e0c060220 0x7f1e0c0f6bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:37564/0 (socket says 172.21.15.160:37564) 2023-12-25T20:09:03.559 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:03.557+0000 7f1e0a59c700 1 -- 172.21.15.160:0/1808342539 learned_addr learned my addr 172.21.15.160:0/1808342539 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:09:03.559 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:03.558+0000 7f1e0a59c700 1 -- 172.21.15.160:0/1808342539 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e00005040 con 0x7f1e0c060220 2023-12-25T20:09:03.559 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:03.558+0000 7f1e0a59c700 1 --2- 172.21.15.160:0/1808342539 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1e0c060220 0x7f1e0c0f6bd0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f1e00013510 tx=0x7f1e0000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:09:03.559 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:03.558+0000 7f1dfaffd700 1 -- 172.21.15.160:0/1808342539 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1e000095f0 con 0x7f1e0c060220 2023-12-25T20:09:03.559 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:03.558+0000 7f1dfaffd700 1 -- 172.21.15.160:0/1808342539 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1e00007e70 con 0x7f1e0c060220 2023-12-25T20:09:03.560 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:03.558+0000 7f1dfaffd700 1 -- 172.21.15.160:0/1808342539 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1e00014da0 con 0x7f1e0c060220 2023-12-25T20:09:03.560 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:03.558+0000 7f1e10f34700 1 -- 172.21.15.160:0/1808342539 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e0c0f51f0 con 0x7f1e0c060220 2023-12-25T20:09:03.560 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:03.558+0000 7f1e10f34700 1 -- 172.21.15.160:0/1808342539 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e0c0f5680 con 0x7f1e0c060220 2023-12-25T20:09:03.560 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:03.559+0000 7f1dfaffd700 1 -- 172.21.15.160:0/1808342539 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1e00025070 con 0x7f1e0c060220 2023-12-25T20:09:03.560 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:03.559+0000 7f1e10f34700 1 -- 172.21.15.160:0/1808342539 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e0c059100 con 0x7f1e0c060220 2023-12-25T20:09:03.560 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:03.559+0000 7f1dfaffd700 1 --2- 172.21.15.160:0/1808342539 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f1df4030000 0x7f1df40324c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:09:03.560 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:03.559+0000 7f1dfaffd700 1 -- 172.21.15.160:0/1808342539 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1e00040a30 con 0x7f1e0c060220 2023-12-25T20:09:03.560 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:03.559+0000 7f1e09d9b700 1 --2- 172.21.15.160:0/1808342539 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f1df4030000 0x7f1df40324c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:09:03.560 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:03.560+0000 7f1e09d9b700 1 --2- 172.21.15.160:0/1808342539 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f1df4030000 0x7f1df40324c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f1dfc0096a0 tx=0x7f1dfc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:09:03.563 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:03.564+0000 7f1dfaffd700 1 -- 172.21.15.160:0/1808342539 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1e0000f770 con 0x7f1e0c060220 2023-12-25T20:09:03.767 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:03.768+0000 7f1e10f34700 1 -- 172.21.15.160:0/1808342539 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1e0c0fbf50 con 0x7f1e0c060220 2023-12-25T20:09:03.768 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:03.769+0000 7f1dfaffd700 1 -- 172.21.15.160:0/1808342539 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1e0000f770 con 0x7f1e0c060220 2023-12-25T20:09:03.769 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:09:03.769 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:09:03.770 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:03.770+0000 7f1e10f34700 1 -- 172.21.15.160:0/1808342539 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f1df4030000 msgr2=0x7f1df40324c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:09:03.770 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:03.770+0000 7f1e10f34700 1 --2- 172.21.15.160:0/1808342539 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f1df4030000 0x7f1df40324c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f1dfc0096a0 tx=0x7f1dfc006b40).stop 2023-12-25T20:09:03.770 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:03.770+0000 7f1e10f34700 1 -- 172.21.15.160:0/1808342539 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1e0c060220 msgr2=0x7f1e0c0f6bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:09:03.770 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:03.770+0000 7f1e10f34700 1 --2- 172.21.15.160:0/1808342539 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1e0c060220 0x7f1e0c0f6bd0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f1e00013510 tx=0x7f1e0000b040).stop 2023-12-25T20:09:03.771 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:03.770+0000 7f1e10f34700 1 -- 172.21.15.160:0/1808342539 shutdown_connections 2023-12-25T20:09:03.771 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:03.770+0000 7f1e10f34700 1 --2- 172.21.15.160:0/1808342539 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f1df4030000 0x7f1df40324c0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:09:03.771 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:03.770+0000 7f1e10f34700 1 --2- 172.21.15.160:0/1808342539 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1e0c060220 0x7f1e0c0f6bd0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:09:03.771 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:03.770+0000 7f1e10f34700 1 -- 172.21.15.160:0/1808342539 >> 172.21.15.160:0/1808342539 conn(0x7f1e0c0f0b70 msgr2=0x7f1e0c0f1700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:09:03.771 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:03.770+0000 7f1e10f34700 1 -- 172.21.15.160:0/1808342539 shutdown_connections 2023-12-25T20:09:03.771 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:03.770+0000 7f1e10f34700 1 -- 172.21.15.160:0/1808342539 wait complete. 2023-12-25T20:09:03.771 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:09:04.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:09:03 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:09:02.638569+0000 mgr.smithi050.nbzhun (mgr.14162) 55 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:09:05.169 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:09:05.169 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:09:05.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:09:04 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:09:03.768854+0000 mon.smithi050 (mon.0) 147 : audit [DBG] from='client.? 172.21.15.160:0/1808342539' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:09:06.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:09:05 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:09:04.638856+0000 mgr.smithi050.nbzhun (mgr.14162) 56 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:09:06.360 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:06.360+0000 7fc5dca18700 1 -- 172.21.15.160:0/1485293220 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc5d80f78e0 msgr2=0x7fc5d80f7d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:09:06.360 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:06.360+0000 7fc5dca18700 1 --2- 172.21.15.160:0/1485293220 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc5d80f78e0 0x7fc5d80f7d00 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7fc5c8009fb0 tx=0x7fc5c8009b30).stop 2023-12-25T20:09:06.360 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:06.360+0000 7fc5dca18700 1 -- 172.21.15.160:0/1485293220 shutdown_connections 2023-12-25T20:09:06.360 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:06.360+0000 7fc5dca18700 1 --2- 172.21.15.160:0/1485293220 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc5d80f78e0 0x7fc5d80f7d00 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:09:06.360 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:06.360+0000 7fc5dca18700 1 -- 172.21.15.160:0/1485293220 >> 172.21.15.160:0/1485293220 conn(0x7fc5d80f0120 msgr2=0x7fc5d80f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:09:06.361 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:06.360+0000 7fc5dca18700 1 -- 172.21.15.160:0/1485293220 shutdown_connections 2023-12-25T20:09:06.361 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:06.360+0000 7fc5dca18700 1 -- 172.21.15.160:0/1485293220 wait complete. 2023-12-25T20:09:06.361 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:06.360+0000 7fc5dca18700 1 Processor -- start 2023-12-25T20:09:06.361 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:06.360+0000 7fc5dca18700 1 -- start start 2023-12-25T20:09:06.361 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:06.360+0000 7fc5dca18700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc5d80f78e0 0x7fc5d81064e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:09:06.361 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:06.361+0000 7fc5dca18700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5d81069d0 con 0x7fc5d80f78e0 2023-12-25T20:09:06.362 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:06.361+0000 7fc5d77fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc5d80f78e0 0x7fc5d81064e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:09:06.363 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:06.361+0000 7fc5d77fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc5d80f78e0 0x7fc5d81064e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:37572/0 (socket says 172.21.15.160:37572) 2023-12-25T20:09:06.363 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:06.361+0000 7fc5d77fe700 1 -- 172.21.15.160:0/540630020 learned_addr learned my addr 172.21.15.160:0/540630020 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:09:06.363 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:06.361+0000 7fc5d77fe700 1 -- 172.21.15.160:0/540630020 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5c8005040 con 0x7fc5d80f78e0 2023-12-25T20:09:06.363 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:06.361+0000 7fc5d77fe700 1 --2- 172.21.15.160:0/540630020 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc5d80f78e0 0x7fc5d81064e0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7fc5c8013510 tx=0x7fc5c800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:09:06.363 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:06.361+0000 7fc5bffff700 1 -- 172.21.15.160:0/540630020 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc5c8009320 con 0x7fc5d80f78e0 2023-12-25T20:09:06.363 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:06.361+0000 7fc5dca18700 1 -- 172.21.15.160:0/540630020 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5d8106bd0 con 0x7fc5d80f78e0 2023-12-25T20:09:06.363 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:06.361+0000 7fc5bffff700 1 -- 172.21.15.160:0/540630020 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc5c8013f00 con 0x7fc5d80f78e0 2023-12-25T20:09:06.363 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:06.361+0000 7fc5bffff700 1 -- 172.21.15.160:0/540630020 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc5c8014dc0 con 0x7fc5d80f78e0 2023-12-25T20:09:06.363 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:06.362+0000 7fc5dca18700 1 -- 172.21.15.160:0/540630020 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5d8107060 con 0x7fc5d80f78e0 2023-12-25T20:09:06.363 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:06.362+0000 7fc5bffff700 1 -- 172.21.15.160:0/540630020 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc5c8014420 con 0x7fc5d80f78e0 2023-12-25T20:09:06.363 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:06.362+0000 7fc5dca18700 1 -- 172.21.15.160:0/540630020 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5d8045ad0 con 0x7fc5d80f78e0 2023-12-25T20:09:06.363 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:06.362+0000 7fc5bffff700 1 --2- 172.21.15.160:0/540630020 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fc5c002ff60 0x7fc5c0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:09:06.363 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:06.362+0000 7fc5bffff700 1 -- 172.21.15.160:0/540630020 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc5c80410d0 con 0x7fc5d80f78e0 2023-12-25T20:09:06.364 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:06.363+0000 7fc5d6ffd700 1 --2- 172.21.15.160:0/540630020 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fc5c002ff60 0x7fc5c0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:09:06.364 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:06.363+0000 7fc5d6ffd700 1 --2- 172.21.15.160:0/540630020 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fc5c002ff60 0x7fc5c0032420 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7fc5cc0096a0 tx=0x7fc5cc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:09:06.367 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:06.367+0000 7fc5bffff700 1 -- 172.21.15.160:0/540630020 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc5c80095f0 con 0x7fc5d80f78e0 2023-12-25T20:09:06.574 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:06.573+0000 7fc5dca18700 1 -- 172.21.15.160:0/540630020 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc5d8107f10 con 0x7fc5d80f78e0 2023-12-25T20:09:06.574 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:06.574+0000 7fc5bffff700 1 -- 172.21.15.160:0/540630020 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc5c801cb10 con 0x7fc5d80f78e0 2023-12-25T20:09:06.574 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:09:06.574 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:09:06.575 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:06.576+0000 7fc5dca18700 1 -- 172.21.15.160:0/540630020 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fc5c002ff60 msgr2=0x7fc5c0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:09:06.575 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:06.576+0000 7fc5dca18700 1 --2- 172.21.15.160:0/540630020 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fc5c002ff60 0x7fc5c0032420 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7fc5cc0096a0 tx=0x7fc5cc006b40).stop 2023-12-25T20:09:06.576 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:06.576+0000 7fc5dca18700 1 -- 172.21.15.160:0/540630020 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc5d80f78e0 msgr2=0x7fc5d81064e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:09:06.576 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:06.576+0000 7fc5dca18700 1 --2- 172.21.15.160:0/540630020 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc5d80f78e0 0x7fc5d81064e0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7fc5c8013510 tx=0x7fc5c800b040).stop 2023-12-25T20:09:06.576 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:06.576+0000 7fc5dca18700 1 -- 172.21.15.160:0/540630020 shutdown_connections 2023-12-25T20:09:06.576 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:06.576+0000 7fc5dca18700 1 --2- 172.21.15.160:0/540630020 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fc5c002ff60 0x7fc5c0032420 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:09:06.576 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:06.576+0000 7fc5dca18700 1 --2- 172.21.15.160:0/540630020 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc5d80f78e0 0x7fc5d81064e0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:09:06.576 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:06.576+0000 7fc5dca18700 1 -- 172.21.15.160:0/540630020 >> 172.21.15.160:0/540630020 conn(0x7fc5d80f0120 msgr2=0x7fc5d80f0ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:09:06.576 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:06.576+0000 7fc5dca18700 1 -- 172.21.15.160:0/540630020 shutdown_connections 2023-12-25T20:09:06.576 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:06.576+0000 7fc5dca18700 1 -- 172.21.15.160:0/540630020 wait complete. 2023-12-25T20:09:06.576 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:09:07.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:09:06 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:09:06.574453+0000 mon.smithi050 (mon.0) 148 : audit [DBG] from='client.? 172.21.15.160:0/540630020' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:09:07.961 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:09:07.961 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:09:08.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:09:07 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:09:06.639144+0000 mgr.smithi050.nbzhun (mgr.14162) 57 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:09:09.188 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:09.184+0000 7fa219aab700 1 -- 172.21.15.160:0/2114482043 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa2140f6510 msgr2=0x7fa2140f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:09:09.188 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:09.184+0000 7fa219aab700 1 --2- 172.21.15.160:0/2114482043 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa2140f6510 0x7fa2140f6930 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7fa208009fb0 tx=0x7fa208009b30).stop 2023-12-25T20:09:09.189 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:09.184+0000 7fa219aab700 1 -- 172.21.15.160:0/2114482043 shutdown_connections 2023-12-25T20:09:09.189 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:09.184+0000 7fa219aab700 1 --2- 172.21.15.160:0/2114482043 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa2140f6510 0x7fa2140f6930 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:09:09.189 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:09.184+0000 7fa219aab700 1 -- 172.21.15.160:0/2114482043 >> 172.21.15.160:0/2114482043 conn(0x7fa2140f1c70 msgr2=0x7fa2140f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:09:09.189 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:09.184+0000 7fa219aab700 1 -- 172.21.15.160:0/2114482043 shutdown_connections 2023-12-25T20:09:09.189 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:09.184+0000 7fa219aab700 1 -- 172.21.15.160:0/2114482043 wait complete. 2023-12-25T20:09:09.189 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:09.184+0000 7fa219aab700 1 Processor -- start 2023-12-25T20:09:09.189 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:09.185+0000 7fa219aab700 1 -- start start 2023-12-25T20:09:09.189 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:09.185+0000 7fa219aab700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa2140f6510 0x7fa2141066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:09:09.190 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:09.185+0000 7fa219aab700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa214106bc0 con 0x7fa2140f6510 2023-12-25T20:09:09.190 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:09.185+0000 7fa2137fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa2140f6510 0x7fa2141066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:09:09.190 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:09.185+0000 7fa2137fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa2140f6510 0x7fa2141066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:48090/0 (socket says 172.21.15.160:48090) 2023-12-25T20:09:09.190 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:09.185+0000 7fa2137fe700 1 -- 172.21.15.160:0/3861503294 learned_addr learned my addr 172.21.15.160:0/3861503294 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:09:09.190 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:09.185+0000 7fa2137fe700 1 -- 172.21.15.160:0/3861503294 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa208005040 con 0x7fa2140f6510 2023-12-25T20:09:09.190 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:09.186+0000 7fa2137fe700 1 --2- 172.21.15.160:0/3861503294 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa2140f6510 0x7fa2141066d0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7fa208013510 tx=0x7fa20800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:09:09.190 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:09.186+0000 7fa1fbfff700 1 -- 172.21.15.160:0/3861503294 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa208009320 con 0x7fa2140f6510 2023-12-25T20:09:09.190 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:09.186+0000 7fa1fbfff700 1 -- 172.21.15.160:0/3861503294 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa208013f00 con 0x7fa2140f6510 2023-12-25T20:09:09.190 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:09.186+0000 7fa219aab700 1 -- 172.21.15.160:0/3861503294 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa214106dc0 con 0x7fa2140f6510 2023-12-25T20:09:09.191 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:09.186+0000 7fa1fbfff700 1 -- 172.21.15.160:0/3861503294 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa208014d90 con 0x7fa2140f6510 2023-12-25T20:09:09.191 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:09.186+0000 7fa219aab700 1 -- 172.21.15.160:0/3861503294 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa214107250 con 0x7fa2140f6510 2023-12-25T20:09:09.191 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:09.187+0000 7fa1fbfff700 1 -- 172.21.15.160:0/3861503294 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa208014420 con 0x7fa2140f6510 2023-12-25T20:09:09.191 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:09.187+0000 7fa1fbfff700 1 --2- 172.21.15.160:0/3861503294 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fa1fc02ffb0 0x7fa1fc032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:09:09.191 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:09.187+0000 7fa1fbfff700 1 -- 172.21.15.160:0/3861503294 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa208040ff0 con 0x7fa2140f6510 2023-12-25T20:09:09.191 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:09.187+0000 7fa219aab700 1 -- 172.21.15.160:0/3861503294 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa214045ad0 con 0x7fa2140f6510 2023-12-25T20:09:09.191 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:09.187+0000 7fa212ffd700 1 --2- 172.21.15.160:0/3861503294 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fa1fc02ffb0 0x7fa1fc032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:09:09.191 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:09.187+0000 7fa212ffd700 1 --2- 172.21.15.160:0/3861503294 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fa1fc02ffb0 0x7fa1fc032470 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7fa2040096a0 tx=0x7fa204006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:09:09.192 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:09.192+0000 7fa1fbfff700 1 -- 172.21.15.160:0/3861503294 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa2080095f0 con 0x7fa2140f6510 2023-12-25T20:09:09.398 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:09.397+0000 7fa219aab700 1 -- 172.21.15.160:0/3861503294 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa214108100 con 0x7fa2140f6510 2023-12-25T20:09:09.399 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:09.398+0000 7fa1fbfff700 1 -- 172.21.15.160:0/3861503294 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa20802b020 con 0x7fa2140f6510 2023-12-25T20:09:09.399 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:09:09.399 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:09:09.401 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:09.400+0000 7fa219aab700 1 -- 172.21.15.160:0/3861503294 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fa1fc02ffb0 msgr2=0x7fa1fc032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:09:09.401 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:09.400+0000 7fa219aab700 1 --2- 172.21.15.160:0/3861503294 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fa1fc02ffb0 0x7fa1fc032470 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7fa2040096a0 tx=0x7fa204006b40).stop 2023-12-25T20:09:09.401 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:09.400+0000 7fa219aab700 1 -- 172.21.15.160:0/3861503294 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa2140f6510 msgr2=0x7fa2141066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:09:09.401 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:09.400+0000 7fa219aab700 1 --2- 172.21.15.160:0/3861503294 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa2140f6510 0x7fa2141066d0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7fa208013510 tx=0x7fa20800b040).stop 2023-12-25T20:09:09.401 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:09.400+0000 7fa219aab700 1 -- 172.21.15.160:0/3861503294 shutdown_connections 2023-12-25T20:09:09.401 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:09.400+0000 7fa219aab700 1 --2- 172.21.15.160:0/3861503294 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fa1fc02ffb0 0x7fa1fc032470 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:09:09.401 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:09.400+0000 7fa219aab700 1 --2- 172.21.15.160:0/3861503294 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa2140f6510 0x7fa2141066d0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:09:09.401 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:09.400+0000 7fa219aab700 1 -- 172.21.15.160:0/3861503294 >> 172.21.15.160:0/3861503294 conn(0x7fa2140f1c70 msgr2=0x7fa2140f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:09:09.402 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:09.400+0000 7fa219aab700 1 -- 172.21.15.160:0/3861503294 shutdown_connections 2023-12-25T20:09:09.402 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:09.400+0000 7fa219aab700 1 -- 172.21.15.160:0/3861503294 wait complete. 2023-12-25T20:09:09.402 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:09:10.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:09:09 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:09:08.639492+0000 mgr.smithi050.nbzhun (mgr.14162) 58 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:09:10.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:09:09 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:09:09.399775+0000 mon.smithi050 (mon.0) 149 : audit [DBG] from='client.? 172.21.15.160:0/3861503294' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:09:10.814 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:09:10.814 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:09:12.041 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:12.040+0000 7fc5bffff700 1 -- 172.21.15.160:0/2807463504 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc5c00f51a0 msgr2=0x7fc5c00f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:09:12.041 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:12.040+0000 7fc5bffff700 1 --2- 172.21.15.160:0/2807463504 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc5c00f51a0 0x7fc5c00f55c0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7fc5b0009fb0 tx=0x7fc5b0009b30).stop 2023-12-25T20:09:12.041 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:12.040+0000 7fc5bffff700 1 -- 172.21.15.160:0/2807463504 shutdown_connections 2023-12-25T20:09:12.042 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:12.040+0000 7fc5bffff700 1 --2- 172.21.15.160:0/2807463504 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc5c00f51a0 0x7fc5c00f55c0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:09:12.042 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:12.040+0000 7fc5bffff700 1 -- 172.21.15.160:0/2807463504 >> 172.21.15.160:0/2807463504 conn(0x7fc5c00f09a0 msgr2=0x7fc5c00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:09:12.042 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:12.040+0000 7fc5bffff700 1 -- 172.21.15.160:0/2807463504 shutdown_connections 2023-12-25T20:09:12.042 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:12.040+0000 7fc5bffff700 1 -- 172.21.15.160:0/2807463504 wait complete. 2023-12-25T20:09:12.042 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:12.040+0000 7fc5bffff700 1 Processor -- start 2023-12-25T20:09:12.042 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:12.040+0000 7fc5bffff700 1 -- start start 2023-12-25T20:09:12.042 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:12.041+0000 7fc5bffff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc5c00f51a0 0x7fc5c01064f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:09:12.042 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:12.041+0000 7fc5bffff700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5c01069e0 con 0x7fc5c00f51a0 2023-12-25T20:09:12.044 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:12.041+0000 7fc5beffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc5c00f51a0 0x7fc5c01064f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:09:12.044 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:12.041+0000 7fc5beffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc5c00f51a0 0x7fc5c01064f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:48116/0 (socket says 172.21.15.160:48116) 2023-12-25T20:09:12.044 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:12.041+0000 7fc5beffd700 1 -- 172.21.15.160:0/2580158011 learned_addr learned my addr 172.21.15.160:0/2580158011 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:09:12.044 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:12.041+0000 7fc5beffd700 1 -- 172.21.15.160:0/2580158011 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5b0005040 con 0x7fc5c00f51a0 2023-12-25T20:09:12.044 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:12.041+0000 7fc5beffd700 1 --2- 172.21.15.160:0/2580158011 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc5c00f51a0 0x7fc5c01064f0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7fc5b0009830 tx=0x7fc5b000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:09:12.044 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:12.041+0000 7fc5a77fe700 1 -- 172.21.15.160:0/2580158011 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc5b0009320 con 0x7fc5c00f51a0 2023-12-25T20:09:12.044 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:12.042+0000 7fc5bffff700 1 -- 172.21.15.160:0/2580158011 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5c0106be0 con 0x7fc5c00f51a0 2023-12-25T20:09:12.044 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:12.042+0000 7fc5a77fe700 1 -- 172.21.15.160:0/2580158011 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc5b00136c0 con 0x7fc5c00f51a0 2023-12-25T20:09:12.044 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:12.042+0000 7fc5a77fe700 1 -- 172.21.15.160:0/2580158011 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc5b001ee90 con 0x7fc5c00f51a0 2023-12-25T20:09:12.044 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:12.042+0000 7fc5bffff700 1 -- 172.21.15.160:0/2580158011 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5c0107070 con 0x7fc5c00f51a0 2023-12-25T20:09:12.044 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:12.042+0000 7fc5a77fe700 1 -- 172.21.15.160:0/2580158011 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc5b001e420 con 0x7fc5c00f51a0 2023-12-25T20:09:12.044 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:12.042+0000 7fc5bffff700 1 -- 172.21.15.160:0/2580158011 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5c0045ad0 con 0x7fc5c00f51a0 2023-12-25T20:09:12.044 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:12.042+0000 7fc5a77fe700 1 --2- 172.21.15.160:0/2580158011 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fc5a802ff10 0x7fc5a80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:09:12.045 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:12.042+0000 7fc5a77fe700 1 -- 172.21.15.160:0/2580158011 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc5b0040f40 con 0x7fc5c00f51a0 2023-12-25T20:09:12.045 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:12.043+0000 7fc5be7fc700 1 --2- 172.21.15.160:0/2580158011 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fc5a802ff10 0x7fc5a80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:09:12.045 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:12.043+0000 7fc5be7fc700 1 --2- 172.21.15.160:0/2580158011 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fc5a802ff10 0x7fc5a80323d0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7fc5b40096a0 tx=0x7fc5b4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:09:12.048 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:12.047+0000 7fc5a77fe700 1 -- 172.21.15.160:0/2580158011 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc5b006d2b0 con 0x7fc5c00f51a0 2023-12-25T20:09:12.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:09:11 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:09:10.639789+0000 mgr.smithi050.nbzhun (mgr.14162) 59 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:09:12.252 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:12.251+0000 7fc5bffff700 1 -- 172.21.15.160:0/2580158011 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc5c0108690 con 0x7fc5c00f51a0 2023-12-25T20:09:12.252 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:12.252+0000 7fc5a77fe700 1 -- 172.21.15.160:0/2580158011 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc5b00095f0 con 0x7fc5c00f51a0 2023-12-25T20:09:12.252 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:09:12.252 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:09:12.254 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:12.253+0000 7fc5bffff700 1 -- 172.21.15.160:0/2580158011 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fc5a802ff10 msgr2=0x7fc5a80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:09:12.254 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:12.253+0000 7fc5bffff700 1 --2- 172.21.15.160:0/2580158011 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fc5a802ff10 0x7fc5a80323d0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7fc5b40096a0 tx=0x7fc5b4006b40).stop 2023-12-25T20:09:12.254 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:12.253+0000 7fc5bffff700 1 -- 172.21.15.160:0/2580158011 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc5c00f51a0 msgr2=0x7fc5c01064f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:09:12.254 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:12.253+0000 7fc5bffff700 1 --2- 172.21.15.160:0/2580158011 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc5c00f51a0 0x7fc5c01064f0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7fc5b0009830 tx=0x7fc5b000b040).stop 2023-12-25T20:09:12.255 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:12.253+0000 7fc5bffff700 1 -- 172.21.15.160:0/2580158011 shutdown_connections 2023-12-25T20:09:12.255 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:12.253+0000 7fc5bffff700 1 --2- 172.21.15.160:0/2580158011 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fc5a802ff10 0x7fc5a80323d0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:09:12.255 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:12.253+0000 7fc5bffff700 1 --2- 172.21.15.160:0/2580158011 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc5c00f51a0 0x7fc5c01064f0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:09:12.255 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:12.253+0000 7fc5bffff700 1 -- 172.21.15.160:0/2580158011 >> 172.21.15.160:0/2580158011 conn(0x7fc5c00f09a0 msgr2=0x7fc5c005fea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:09:12.255 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:12.253+0000 7fc5bffff700 1 -- 172.21.15.160:0/2580158011 shutdown_connections 2023-12-25T20:09:12.255 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:12.253+0000 7fc5bffff700 1 -- 172.21.15.160:0/2580158011 wait complete. 2023-12-25T20:09:12.255 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:09:13.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:09:12 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:09:12.253306+0000 mon.smithi050 (mon.0) 150 : audit [DBG] from='client.? 172.21.15.160:0/2580158011' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:09:13.643 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:09:13.644 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:09:14.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:09:13 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:09:12.640120+0000 mgr.smithi050.nbzhun (mgr.14162) 60 : cluster [DBG] pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:09:15.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:09:14 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:09:14.763076+0000 mon.smithi050 (mon.0) 151 : audit [INF] from='mgr.14162 172.21.15.50:0/3178367813' entity='mgr.smithi050.nbzhun' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi050.nbzhun/mirror_snapshot_schedule"}]: dispatch 2023-12-25T20:09:15.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:09:14 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:09:14.764479+0000 mon.smithi050 (mon.0) 152 : audit [INF] from='mgr.14162 172.21.15.50:0/3178367813' entity='mgr.smithi050.nbzhun' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi050.nbzhun/trash_purge_schedule"}]: dispatch 2023-12-25T20:09:15.733 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:15.731+0000 7f57f9785700 1 -- 172.21.15.160:0/131812852 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f57f40f6510 msgr2=0x7f57f40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:09:15.733 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:15.731+0000 7f57f9785700 1 --2- 172.21.15.160:0/131812852 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f57f40f6510 0x7f57f40f6930 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f57e8009fb0 tx=0x7f57e8009b30).stop 2023-12-25T20:09:15.733 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:15.731+0000 7f57f9785700 1 -- 172.21.15.160:0/131812852 shutdown_connections 2023-12-25T20:09:15.733 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:15.731+0000 7f57f9785700 1 --2- 172.21.15.160:0/131812852 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f57f40f6510 0x7f57f40f6930 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:09:15.733 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:15.731+0000 7f57f9785700 1 -- 172.21.15.160:0/131812852 >> 172.21.15.160:0/131812852 conn(0x7f57f40f1c70 msgr2=0x7f57f40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:09:15.733 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:15.731+0000 7f57f9785700 1 -- 172.21.15.160:0/131812852 shutdown_connections 2023-12-25T20:09:15.733 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:15.732+0000 7f57f9785700 1 -- 172.21.15.160:0/131812852 wait complete. 2023-12-25T20:09:15.734 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:15.732+0000 7f57f9785700 1 Processor -- start 2023-12-25T20:09:15.735 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:15.732+0000 7f57f9785700 1 -- start start 2023-12-25T20:09:15.735 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:15.733+0000 7f57f9785700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f57f40f6510 0x7f57f4106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:09:15.735 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:15.733+0000 7f57f9785700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57f4106c20 con 0x7f57f40f6510 2023-12-25T20:09:15.735 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:15.733+0000 7f57f2ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f57f40f6510 0x7f57f4106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:09:15.735 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:15.733+0000 7f57f2ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f57f40f6510 0x7f57f4106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:48130/0 (socket says 172.21.15.160:48130) 2023-12-25T20:09:15.735 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:15.733+0000 7f57f2ffd700 1 -- 172.21.15.160:0/4185129977 learned_addr learned my addr 172.21.15.160:0/4185129977 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:09:15.735 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:15.734+0000 7f57f2ffd700 1 -- 172.21.15.160:0/4185129977 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57e8005040 con 0x7f57f40f6510 2023-12-25T20:09:15.736 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:15.735+0000 7f57f2ffd700 1 --2- 172.21.15.160:0/4185129977 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f57f40f6510 0x7f57f4106730 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f57e8013510 tx=0x7f57e800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:09:15.737 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:15.735+0000 7f57db7fe700 1 -- 172.21.15.160:0/4185129977 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f57e8009320 con 0x7f57f40f6510 2023-12-25T20:09:15.737 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:15.735+0000 7f57db7fe700 1 -- 172.21.15.160:0/4185129977 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f57e8013f00 con 0x7f57f40f6510 2023-12-25T20:09:15.737 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:15.735+0000 7f57db7fe700 1 -- 172.21.15.160:0/4185129977 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f57e8014d90 con 0x7f57f40f6510 2023-12-25T20:09:15.737 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:15.735+0000 7f57f9785700 1 -- 172.21.15.160:0/4185129977 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57f4106e20 con 0x7f57f40f6510 2023-12-25T20:09:15.737 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:15.736+0000 7f57f9785700 1 -- 172.21.15.160:0/4185129977 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57f41072b0 con 0x7f57f40f6510 2023-12-25T20:09:15.737 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:15.736+0000 7f57f9785700 1 -- 172.21.15.160:0/4185129977 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57f4045ad0 con 0x7f57f40f6510 2023-12-25T20:09:15.738 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:15.737+0000 7f57db7fe700 1 -- 172.21.15.160:0/4185129977 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f57e80095f0 con 0x7f57f40f6510 2023-12-25T20:09:15.738 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:15.737+0000 7f57db7fe700 1 --2- 172.21.15.160:0/4185129977 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f57dc02ffb0 0x7f57dc032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:09:15.738 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:15.737+0000 7f57db7fe700 1 -- 172.21.15.160:0/4185129977 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f57e8041210 con 0x7f57f40f6510 2023-12-25T20:09:15.739 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:15.738+0000 7f57f27fc700 1 --2- 172.21.15.160:0/4185129977 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f57dc02ffb0 0x7f57dc032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:09:15.739 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:15.738+0000 7f57f27fc700 1 --2- 172.21.15.160:0/4185129977 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f57dc02ffb0 0x7f57dc032470 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f57e40096a0 tx=0x7f57e4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:09:15.742 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:15.742+0000 7f57db7fe700 1 -- 172.21.15.160:0/4185129977 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f57e8014420 con 0x7f57f40f6510 2023-12-25T20:09:15.949 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:15.948+0000 7f57f9785700 1 -- 172.21.15.160:0/4185129977 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f57f40fc050 con 0x7f57f40f6510 2023-12-25T20:09:15.949 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:15.949+0000 7f57db7fe700 1 -- 172.21.15.160:0/4185129977 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f57e802b030 con 0x7f57f40f6510 2023-12-25T20:09:15.950 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:09:15.950 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:09:15.951 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:15.950+0000 7f57f9785700 1 -- 172.21.15.160:0/4185129977 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f57dc02ffb0 msgr2=0x7f57dc032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:09:15.951 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:15.950+0000 7f57f9785700 1 --2- 172.21.15.160:0/4185129977 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f57dc02ffb0 0x7f57dc032470 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f57e40096a0 tx=0x7f57e4006b40).stop 2023-12-25T20:09:15.951 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:15.950+0000 7f57f9785700 1 -- 172.21.15.160:0/4185129977 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f57f40f6510 msgr2=0x7f57f4106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:09:15.951 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:15.950+0000 7f57f9785700 1 --2- 172.21.15.160:0/4185129977 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f57f40f6510 0x7f57f4106730 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f57e8013510 tx=0x7f57e800b040).stop 2023-12-25T20:09:15.952 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:15.950+0000 7f57f9785700 1 -- 172.21.15.160:0/4185129977 shutdown_connections 2023-12-25T20:09:15.952 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:15.950+0000 7f57f9785700 1 --2- 172.21.15.160:0/4185129977 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f57dc02ffb0 0x7f57dc032470 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:09:15.952 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:15.950+0000 7f57f9785700 1 --2- 172.21.15.160:0/4185129977 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f57f40f6510 0x7f57f4106730 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:09:15.952 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:15.950+0000 7f57f9785700 1 -- 172.21.15.160:0/4185129977 >> 172.21.15.160:0/4185129977 conn(0x7f57f40f1c70 msgr2=0x7f57f40f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:09:15.952 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:15.950+0000 7f57f9785700 1 -- 172.21.15.160:0/4185129977 shutdown_connections 2023-12-25T20:09:15.952 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:15.950+0000 7f57f9785700 1 -- 172.21.15.160:0/4185129977 wait complete. 2023-12-25T20:09:15.952 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:09:16.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:09:15 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:09:14.640409+0000 mgr.smithi050.nbzhun (mgr.14162) 61 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:09:17.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:09:16 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:09:15.950270+0000 mon.smithi050 (mon.0) 153 : audit [DBG] from='client.? 172.21.15.160:0/4185129977' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:09:17.330 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:09:17.331 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:09:18.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:09:17 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:09:16.640744+0000 mgr.smithi050.nbzhun (mgr.14162) 62 : cluster [DBG] pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:09:18.567 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:18.565+0000 7fb8d0c19700 1 -- 172.21.15.160:0/2516384668 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb8cc0f6510 msgr2=0x7fb8cc0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:09:18.567 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:18.565+0000 7fb8d0c19700 1 --2- 172.21.15.160:0/2516384668 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb8cc0f6510 0x7fb8cc0f6930 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7fb8c0009fb0 tx=0x7fb8c0009b30).stop 2023-12-25T20:09:18.567 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:18.565+0000 7fb8d0c19700 1 -- 172.21.15.160:0/2516384668 shutdown_connections 2023-12-25T20:09:18.567 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:18.565+0000 7fb8d0c19700 1 --2- 172.21.15.160:0/2516384668 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb8cc0f6510 0x7fb8cc0f6930 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:09:18.567 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:18.565+0000 7fb8d0c19700 1 -- 172.21.15.160:0/2516384668 >> 172.21.15.160:0/2516384668 conn(0x7fb8cc0f1c70 msgr2=0x7fb8cc0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:09:18.567 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:18.565+0000 7fb8d0c19700 1 -- 172.21.15.160:0/2516384668 shutdown_connections 2023-12-25T20:09:18.567 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:18.565+0000 7fb8d0c19700 1 -- 172.21.15.160:0/2516384668 wait complete. 2023-12-25T20:09:18.567 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:18.565+0000 7fb8d0c19700 1 Processor -- start 2023-12-25T20:09:18.567 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:18.565+0000 7fb8d0c19700 1 -- start start 2023-12-25T20:09:18.567 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:18.565+0000 7fb8d0c19700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb8cc0f6510 0x7fb8cc106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:09:18.568 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:18.565+0000 7fb8d0c19700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8cc106c20 con 0x7fb8cc0f6510 2023-12-25T20:09:18.568 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:18.566+0000 7fb8ca59c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb8cc0f6510 0x7fb8cc106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:09:18.568 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:18.566+0000 7fb8ca59c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb8cc0f6510 0x7fb8cc106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:40476/0 (socket says 172.21.15.160:40476) 2023-12-25T20:09:18.568 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:18.566+0000 7fb8ca59c700 1 -- 172.21.15.160:0/2737458616 learned_addr learned my addr 172.21.15.160:0/2737458616 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:09:18.568 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:18.566+0000 7fb8ca59c700 1 -- 172.21.15.160:0/2737458616 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8c0005040 con 0x7fb8cc0f6510 2023-12-25T20:09:18.568 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:18.566+0000 7fb8ca59c700 1 --2- 172.21.15.160:0/2737458616 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb8cc0f6510 0x7fb8cc106730 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7fb8c0014000 tx=0x7fb8c000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:09:18.569 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:18.567+0000 7fb8baffd700 1 -- 172.21.15.160:0/2737458616 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb8c00095f0 con 0x7fb8cc0f6510 2023-12-25T20:09:18.569 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:18.567+0000 7fb8d0c19700 1 -- 172.21.15.160:0/2737458616 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8cc106e20 con 0x7fb8cc0f6510 2023-12-25T20:09:18.569 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:18.567+0000 7fb8d0c19700 1 -- 172.21.15.160:0/2737458616 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8cc1072b0 con 0x7fb8cc0f6510 2023-12-25T20:09:18.569 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:18.567+0000 7fb8baffd700 1 -- 172.21.15.160:0/2737458616 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb8c0007e70 con 0x7fb8cc0f6510 2023-12-25T20:09:18.569 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:18.567+0000 7fb8baffd700 1 -- 172.21.15.160:0/2737458616 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb8c0014ce0 con 0x7fb8cc0f6510 2023-12-25T20:09:18.569 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:18.567+0000 7fb8d0c19700 1 -- 172.21.15.160:0/2737458616 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8b0004fa0 con 0x7fb8cc0f6510 2023-12-25T20:09:18.570 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:18.568+0000 7fb8baffd700 1 -- 172.21.15.160:0/2737458616 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb8c0025070 con 0x7fb8cc0f6510 2023-12-25T20:09:18.570 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:18.568+0000 7fb8baffd700 1 --2- 172.21.15.160:0/2737458616 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fb8b402ff60 0x7fb8b4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:09:18.570 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:18.568+0000 7fb8baffd700 1 -- 172.21.15.160:0/2737458616 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb8c0040920 con 0x7fb8cc0f6510 2023-12-25T20:09:18.570 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:18.569+0000 7fb8c9d9b700 1 --2- 172.21.15.160:0/2737458616 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fb8b402ff60 0x7fb8b4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:09:18.571 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:18.571+0000 7fb8c9d9b700 1 --2- 172.21.15.160:0/2737458616 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fb8b402ff60 0x7fb8b4032420 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7fb8bc0096a0 tx=0x7fb8bc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:09:18.574 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:18.573+0000 7fb8baffd700 1 -- 172.21.15.160:0/2737458616 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb8c00079c0 con 0x7fb8cc0f6510 2023-12-25T20:09:18.781 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:18.779+0000 7fb8d0c19700 1 -- 172.21.15.160:0/2737458616 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb8b0005670 con 0x7fb8cc0f6510 2023-12-25T20:09:18.781 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:18.780+0000 7fb8baffd700 1 -- 172.21.15.160:0/2737458616 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb8c0025300 con 0x7fb8cc0f6510 2023-12-25T20:09:18.782 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:09:18.782 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:09:18.784 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:18.782+0000 7fb8d0c19700 1 -- 172.21.15.160:0/2737458616 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fb8b402ff60 msgr2=0x7fb8b4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:09:18.784 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:18.782+0000 7fb8d0c19700 1 --2- 172.21.15.160:0/2737458616 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fb8b402ff60 0x7fb8b4032420 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7fb8bc0096a0 tx=0x7fb8bc006b40).stop 2023-12-25T20:09:18.784 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:18.782+0000 7fb8d0c19700 1 -- 172.21.15.160:0/2737458616 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb8cc0f6510 msgr2=0x7fb8cc106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:09:18.784 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:18.782+0000 7fb8d0c19700 1 --2- 172.21.15.160:0/2737458616 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb8cc0f6510 0x7fb8cc106730 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7fb8c0014000 tx=0x7fb8c000b040).stop 2023-12-25T20:09:18.784 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:18.782+0000 7fb8d0c19700 1 -- 172.21.15.160:0/2737458616 shutdown_connections 2023-12-25T20:09:18.784 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:18.782+0000 7fb8d0c19700 1 --2- 172.21.15.160:0/2737458616 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fb8b402ff60 0x7fb8b4032420 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:09:18.784 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:18.782+0000 7fb8d0c19700 1 --2- 172.21.15.160:0/2737458616 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb8cc0f6510 0x7fb8cc106730 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:09:18.784 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:18.782+0000 7fb8d0c19700 1 -- 172.21.15.160:0/2737458616 >> 172.21.15.160:0/2737458616 conn(0x7fb8cc0f1c70 msgr2=0x7fb8cc0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:09:18.784 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:18.782+0000 7fb8d0c19700 1 -- 172.21.15.160:0/2737458616 shutdown_connections 2023-12-25T20:09:18.784 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:18.782+0000 7fb8d0c19700 1 -- 172.21.15.160:0/2737458616 wait complete. 2023-12-25T20:09:18.785 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:09:20.184 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:09:20.185 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:09:20.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:09:19 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:09:18.641133+0000 mgr.smithi050.nbzhun (mgr.14162) 63 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:09:20.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:09:19 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:09:18.782008+0000 mon.smithi050 (mon.0) 154 : audit [DBG] from='client.? 172.21.15.160:0/2737458616' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:09:21.399 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:21.398+0000 7f87d08d8700 1 -- 172.21.15.160:0/2192291776 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f87cc060280 msgr2=0x7f87cc0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:09:21.399 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:21.398+0000 7f87d08d8700 1 --2- 172.21.15.160:0/2192291776 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f87cc060280 0x7f87cc0f90f0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f87b4009fb0 tx=0x7f87b4009b30).stop 2023-12-25T20:09:21.399 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:21.398+0000 7f87d08d8700 1 -- 172.21.15.160:0/2192291776 shutdown_connections 2023-12-25T20:09:21.399 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:21.398+0000 7f87d08d8700 1 --2- 172.21.15.160:0/2192291776 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f87cc060280 0x7f87cc0f90f0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:09:21.399 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:21.398+0000 7f87d08d8700 1 -- 172.21.15.160:0/2192291776 >> 172.21.15.160:0/2192291776 conn(0x7f87cc0f0b70 msgr2=0x7f87cc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:09:21.399 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:21.398+0000 7f87d08d8700 1 -- 172.21.15.160:0/2192291776 shutdown_connections 2023-12-25T20:09:21.400 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:21.398+0000 7f87d08d8700 1 -- 172.21.15.160:0/2192291776 wait complete. 2023-12-25T20:09:21.401 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:21.399+0000 7f87d08d8700 1 Processor -- start 2023-12-25T20:09:21.401 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:21.399+0000 7f87d08d8700 1 -- start start 2023-12-25T20:09:21.401 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:21.399+0000 7f87d08d8700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f87cc060280 0x7f87cc0f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:09:21.401 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:21.399+0000 7f87d08d8700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87cc0f4ff0 con 0x7f87cc060280 2023-12-25T20:09:21.401 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:21.399+0000 7f87ca59c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f87cc060280 0x7f87cc0f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:09:21.401 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:21.399+0000 7f87ca59c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f87cc060280 0x7f87cc0f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:40480/0 (socket says 172.21.15.160:40480) 2023-12-25T20:09:21.401 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:21.399+0000 7f87ca59c700 1 -- 172.21.15.160:0/3226887094 learned_addr learned my addr 172.21.15.160:0/3226887094 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:09:21.401 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:21.400+0000 7f87ca59c700 1 -- 172.21.15.160:0/3226887094 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87b4005040 con 0x7f87cc060280 2023-12-25T20:09:21.401 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:21.400+0000 7f87ca59c700 1 --2- 172.21.15.160:0/3226887094 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f87cc060280 0x7f87cc0f6b70 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f87b4009830 tx=0x7f87b400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:09:21.401 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:21.400+0000 7f87c2ffd700 1 -- 172.21.15.160:0/3226887094 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f87b4009320 con 0x7f87cc060280 2023-12-25T20:09:21.402 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:21.400+0000 7f87d08d8700 1 -- 172.21.15.160:0/3226887094 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87cc0f5190 con 0x7f87cc060280 2023-12-25T20:09:21.402 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:21.400+0000 7f87c2ffd700 1 -- 172.21.15.160:0/3226887094 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f87b40136c0 con 0x7f87cc060280 2023-12-25T20:09:21.402 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:21.400+0000 7f87c2ffd700 1 -- 172.21.15.160:0/3226887094 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f87b401ee90 con 0x7f87cc060280 2023-12-25T20:09:21.402 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:21.400+0000 7f87d08d8700 1 -- 172.21.15.160:0/3226887094 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87cc0f5620 con 0x7f87cc060280 2023-12-25T20:09:21.403 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:21.401+0000 7f87c2ffd700 1 -- 172.21.15.160:0/3226887094 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f87b401e420 con 0x7f87cc060280 2023-12-25T20:09:21.403 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:21.401+0000 7f87d08d8700 1 -- 172.21.15.160:0/3226887094 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87cc059100 con 0x7f87cc060280 2023-12-25T20:09:21.403 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:21.401+0000 7f87c2ffd700 1 --2- 172.21.15.160:0/3226887094 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f87b802ff10 0x7f87b80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:09:21.403 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:21.401+0000 7f87c2ffd700 1 -- 172.21.15.160:0/3226887094 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f87b4040f40 con 0x7f87cc060280 2023-12-25T20:09:21.403 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:21.401+0000 7f87c9d9b700 1 --2- 172.21.15.160:0/3226887094 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f87b802ff10 0x7f87b80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:09:21.403 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:21.402+0000 7f87c9d9b700 1 --2- 172.21.15.160:0/3226887094 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f87b802ff10 0x7f87b80323d0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f87bc0096a0 tx=0x7f87bc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:09:21.406 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:21.406+0000 7f87c2ffd700 1 -- 172.21.15.160:0/3226887094 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f87b406d2b0 con 0x7f87cc060280 2023-12-25T20:09:21.610 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:21.609+0000 7f87d08d8700 1 -- 172.21.15.160:0/3226887094 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f87cc045ad0 con 0x7f87cc060280 2023-12-25T20:09:21.611 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:21.610+0000 7f87c2ffd700 1 -- 172.21.15.160:0/3226887094 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f87b40095f0 con 0x7f87cc060280 2023-12-25T20:09:21.611 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:09:21.611 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:09:21.613 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:21.612+0000 7f87d08d8700 1 -- 172.21.15.160:0/3226887094 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f87b802ff10 msgr2=0x7f87b80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:09:21.613 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:21.612+0000 7f87d08d8700 1 --2- 172.21.15.160:0/3226887094 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f87b802ff10 0x7f87b80323d0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f87bc0096a0 tx=0x7f87bc006b40).stop 2023-12-25T20:09:21.613 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:21.612+0000 7f87d08d8700 1 -- 172.21.15.160:0/3226887094 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f87cc060280 msgr2=0x7f87cc0f6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:09:21.613 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:21.612+0000 7f87d08d8700 1 --2- 172.21.15.160:0/3226887094 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f87cc060280 0x7f87cc0f6b70 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f87b4009830 tx=0x7f87b400b040).stop 2023-12-25T20:09:21.613 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:21.612+0000 7f87d08d8700 1 -- 172.21.15.160:0/3226887094 shutdown_connections 2023-12-25T20:09:21.613 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:21.612+0000 7f87d08d8700 1 --2- 172.21.15.160:0/3226887094 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f87b802ff10 0x7f87b80323d0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:09:21.613 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:21.612+0000 7f87d08d8700 1 --2- 172.21.15.160:0/3226887094 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f87cc060280 0x7f87cc0f6b70 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:09:21.613 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:21.612+0000 7f87d08d8700 1 -- 172.21.15.160:0/3226887094 >> 172.21.15.160:0/3226887094 conn(0x7f87cc0f0b70 msgr2=0x7f87cc0f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:09:21.613 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:21.612+0000 7f87d08d8700 1 -- 172.21.15.160:0/3226887094 shutdown_connections 2023-12-25T20:09:21.614 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:21.612+0000 7f87d08d8700 1 -- 172.21.15.160:0/3226887094 wait complete. 2023-12-25T20:09:21.614 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:09:22.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:09:21 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:09:20.641403+0000 mgr.smithi050.nbzhun (mgr.14162) 64 : cluster [DBG] pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:09:22.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:09:21 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:09:21.611923+0000 mon.smithi050 (mon.0) 155 : audit [DBG] from='client.? 172.21.15.160:0/3226887094' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:09:23.003 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:09:23.003 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:09:24.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:09:23 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:09:22.641707+0000 mgr.smithi050.nbzhun (mgr.14162) 65 : cluster [DBG] pgmap v58: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:09:24.231 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:24.229+0000 7fa46f849700 1 -- 172.21.15.160:0/3201404691 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa4680f5410 msgr2=0x7fa4680f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:09:24.231 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:24.229+0000 7fa46f849700 1 --2- 172.21.15.160:0/3201404691 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa4680f5410 0x7fa4680f5830 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7fa464009fb0 tx=0x7fa464009b30).stop 2023-12-25T20:09:24.231 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:24.229+0000 7fa46f849700 1 -- 172.21.15.160:0/3201404691 shutdown_connections 2023-12-25T20:09:24.231 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:24.229+0000 7fa46f849700 1 --2- 172.21.15.160:0/3201404691 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa4680f5410 0x7fa4680f5830 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:09:24.231 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:24.229+0000 7fa46f849700 1 -- 172.21.15.160:0/3201404691 >> 172.21.15.160:0/3201404691 conn(0x7fa4680f0b70 msgr2=0x7fa4680f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:09:24.232 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:24.230+0000 7fa46f849700 1 -- 172.21.15.160:0/3201404691 shutdown_connections 2023-12-25T20:09:24.232 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:24.230+0000 7fa46f849700 1 -- 172.21.15.160:0/3201404691 wait complete. 2023-12-25T20:09:24.232 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:24.230+0000 7fa46f849700 1 Processor -- start 2023-12-25T20:09:24.232 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:24.230+0000 7fa46f849700 1 -- start start 2023-12-25T20:09:24.232 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:24.230+0000 7fa46f849700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa4680f5410 0x7fa468105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:09:24.232 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:24.230+0000 7fa46f849700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa468105b20 con 0x7fa4680f5410 2023-12-25T20:09:24.232 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:24.232+0000 7fa46d5e5700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa4680f5410 0x7fa468105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:09:24.232 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:24.232+0000 7fa46d5e5700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa4680f5410 0x7fa468105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:40492/0 (socket says 172.21.15.160:40492) 2023-12-25T20:09:24.233 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:24.232+0000 7fa46d5e5700 1 -- 172.21.15.160:0/1616904497 learned_addr learned my addr 172.21.15.160:0/1616904497 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:09:24.233 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:24.232+0000 7fa46d5e5700 1 -- 172.21.15.160:0/1616904497 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa464005040 con 0x7fa4680f5410 2023-12-25T20:09:24.233 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:24.233+0000 7fa46d5e5700 1 --2- 172.21.15.160:0/1616904497 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa4680f5410 0x7fa468105630 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7fa464013510 tx=0x7fa46400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:09:24.234 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:24.233+0000 7fa45dffb700 1 -- 172.21.15.160:0/1616904497 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa464009320 con 0x7fa4680f5410 2023-12-25T20:09:24.234 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:24.233+0000 7fa46f849700 1 -- 172.21.15.160:0/1616904497 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa468105d20 con 0x7fa4680f5410 2023-12-25T20:09:24.234 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:24.233+0000 7fa45dffb700 1 -- 172.21.15.160:0/1616904497 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa464013f00 con 0x7fa4680f5410 2023-12-25T20:09:24.234 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:24.233+0000 7fa45dffb700 1 -- 172.21.15.160:0/1616904497 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa464014d90 con 0x7fa4680f5410 2023-12-25T20:09:24.234 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:24.233+0000 7fa46f849700 1 -- 172.21.15.160:0/1616904497 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4681061b0 con 0x7fa4680f5410 2023-12-25T20:09:24.235 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:24.234+0000 7fa45dffb700 1 -- 172.21.15.160:0/1616904497 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa4640095f0 con 0x7fa4680f5410 2023-12-25T20:09:24.235 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:24.234+0000 7fa46f849700 1 -- 172.21.15.160:0/1616904497 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa468045ad0 con 0x7fa4680f5410 2023-12-25T20:09:24.235 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:24.234+0000 7fa45dffb700 1 --2- 172.21.15.160:0/1616904497 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fa45402ffb0 0x7fa454032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:09:24.235 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:24.235+0000 7fa45dffb700 1 -- 172.21.15.160:0/1616904497 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa4640410a0 con 0x7fa4680f5410 2023-12-25T20:09:24.236 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:24.235+0000 7fa46cde4700 1 --2- 172.21.15.160:0/1616904497 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fa45402ffb0 0x7fa454032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:09:24.236 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:24.235+0000 7fa46cde4700 1 --2- 172.21.15.160:0/1616904497 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fa45402ffb0 0x7fa454032470 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7fa4580096a0 tx=0x7fa458006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:09:24.240 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:24.240+0000 7fa45dffb700 1 -- 172.21.15.160:0/1616904497 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa46406d2b0 con 0x7fa4680f5410 2023-12-25T20:09:24.449 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:24.445+0000 7fa46f849700 1 -- 172.21.15.160:0/1616904497 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa4680fae60 con 0x7fa4680f5410 2023-12-25T20:09:24.450 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:24.446+0000 7fa45dffb700 1 -- 172.21.15.160:0/1616904497 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa464014420 con 0x7fa4680f5410 2023-12-25T20:09:24.450 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:09:24.450 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:09:24.451 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:24.447+0000 7fa46f849700 1 -- 172.21.15.160:0/1616904497 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fa45402ffb0 msgr2=0x7fa454032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:09:24.451 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:24.447+0000 7fa46f849700 1 --2- 172.21.15.160:0/1616904497 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fa45402ffb0 0x7fa454032470 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7fa4580096a0 tx=0x7fa458006b40).stop 2023-12-25T20:09:24.451 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:24.447+0000 7fa46f849700 1 -- 172.21.15.160:0/1616904497 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa4680f5410 msgr2=0x7fa468105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:09:24.451 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:24.447+0000 7fa46f849700 1 --2- 172.21.15.160:0/1616904497 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa4680f5410 0x7fa468105630 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7fa464013510 tx=0x7fa46400b040).stop 2023-12-25T20:09:24.451 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:24.447+0000 7fa46f849700 1 -- 172.21.15.160:0/1616904497 shutdown_connections 2023-12-25T20:09:24.451 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:24.447+0000 7fa46f849700 1 --2- 172.21.15.160:0/1616904497 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fa45402ffb0 0x7fa454032470 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:09:24.452 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:24.447+0000 7fa46f849700 1 --2- 172.21.15.160:0/1616904497 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa4680f5410 0x7fa468105630 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:09:24.452 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:24.447+0000 7fa46f849700 1 -- 172.21.15.160:0/1616904497 >> 172.21.15.160:0/1616904497 conn(0x7fa4680f0b70 msgr2=0x7fa4680f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:09:24.452 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:24.447+0000 7fa46f849700 1 -- 172.21.15.160:0/1616904497 shutdown_connections 2023-12-25T20:09:24.452 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:24.447+0000 7fa46f849700 1 -- 172.21.15.160:0/1616904497 wait complete. 2023-12-25T20:09:24.452 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:09:25.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:09:24 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:09:24.447309+0000 mon.smithi050 (mon.0) 156 : audit [DBG] from='client.? 172.21.15.160:0/1616904497' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:09:25.835 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:09:25.836 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:09:26.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:09:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:09:24.642088+0000 mgr.smithi050.nbzhun (mgr.14162) 66 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:09:27.058 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:27.057+0000 7fb8c8bd6700 1 -- 172.21.15.160:0/680488339 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb8c40f47c0 msgr2=0x7fb8c40f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:09:27.059 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:27.057+0000 7fb8c8bd6700 1 --2- 172.21.15.160:0/680488339 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb8c40f47c0 0x7fb8c40f4be0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7fb8b4009960 tx=0x7fb8b4004d10).stop 2023-12-25T20:09:27.059 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:27.057+0000 7fb8c8bd6700 1 -- 172.21.15.160:0/680488339 shutdown_connections 2023-12-25T20:09:27.059 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:27.057+0000 7fb8c8bd6700 1 --2- 172.21.15.160:0/680488339 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb8c40f47c0 0x7fb8c40f4be0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:09:27.059 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:27.057+0000 7fb8c8bd6700 1 -- 172.21.15.160:0/680488339 >> 172.21.15.160:0/680488339 conn(0x7fb8c40f02e0 msgr2=0x7fb8c40f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:09:27.059 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:27.057+0000 7fb8c8bd6700 1 -- 172.21.15.160:0/680488339 shutdown_connections 2023-12-25T20:09:27.059 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:27.057+0000 7fb8c8bd6700 1 -- 172.21.15.160:0/680488339 wait complete. 2023-12-25T20:09:27.059 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:27.057+0000 7fb8c8bd6700 1 Processor -- start 2023-12-25T20:09:27.059 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:27.057+0000 7fb8c8bd6700 1 -- start start 2023-12-25T20:09:27.059 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:27.058+0000 7fb8c8bd6700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb8c40f47c0 0x7fb8c4106700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:09:27.059 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:27.058+0000 7fb8c8bd6700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8c4106bf0 con 0x7fb8c40f47c0 2023-12-25T20:09:27.060 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:27.059+0000 7fb8c259c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb8c40f47c0 0x7fb8c4106700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:09:27.060 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:27.059+0000 7fb8c259c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb8c40f47c0 0x7fb8c4106700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:40514/0 (socket says 172.21.15.160:40514) 2023-12-25T20:09:27.060 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:27.059+0000 7fb8c259c700 1 -- 172.21.15.160:0/3716982887 learned_addr learned my addr 172.21.15.160:0/3716982887 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:09:27.061 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:27.061+0000 7fb8c259c700 1 -- 172.21.15.160:0/3716982887 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8b4009640 con 0x7fb8c40f47c0 2023-12-25T20:09:27.062 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:27.062+0000 7fb8c259c700 1 --2- 172.21.15.160:0/3716982887 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb8c40f47c0 0x7fb8c4106700 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7fb8b4004b70 tx=0x7fb8b4004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:09:27.064 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:27.063+0000 7fb8baffd700 1 -- 172.21.15.160:0/3716982887 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb8b4018070 con 0x7fb8c40f47c0 2023-12-25T20:09:27.064 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:27.063+0000 7fb8baffd700 1 -- 172.21.15.160:0/3716982887 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb8b400b870 con 0x7fb8c40f47c0 2023-12-25T20:09:27.064 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:27.063+0000 7fb8c8bd6700 1 -- 172.21.15.160:0/3716982887 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8c4106df0 con 0x7fb8c40f47c0 2023-12-25T20:09:27.064 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:27.063+0000 7fb8baffd700 1 -- 172.21.15.160:0/3716982887 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb8b401c560 con 0x7fb8c40f47c0 2023-12-25T20:09:27.064 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:27.063+0000 7fb8c8bd6700 1 -- 172.21.15.160:0/3716982887 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8c4107280 con 0x7fb8c40f47c0 2023-12-25T20:09:27.064 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:27.064+0000 7fb8c8bd6700 1 -- 172.21.15.160:0/3716982887 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8c4045ad0 con 0x7fb8c40f47c0 2023-12-25T20:09:27.066 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:27.065+0000 7fb8baffd700 1 -- 172.21.15.160:0/3716982887 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb8b4013630 con 0x7fb8c40f47c0 2023-12-25T20:09:27.066 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:27.065+0000 7fb8baffd700 1 --2- 172.21.15.160:0/3716982887 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fb8b002ff60 0x7fb8b0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:09:27.066 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:27.066+0000 7fb8baffd700 1 -- 172.21.15.160:0/3716982887 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb8b40413d0 con 0x7fb8c40f47c0 2023-12-25T20:09:27.068 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:27.067+0000 7fb8c1d9b700 1 --2- 172.21.15.160:0/3716982887 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fb8b002ff60 0x7fb8b0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:09:27.070 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:27.070+0000 7fb8c1d9b700 1 --2- 172.21.15.160:0/3716982887 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fb8b002ff60 0x7fb8b0032420 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7fb8ac0096a0 tx=0x7fb8ac006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:09:27.071 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:27.070+0000 7fb8baffd700 1 -- 172.21.15.160:0/3716982887 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb8b4004810 con 0x7fb8c40f47c0 2023-12-25T20:09:27.273 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:27.272+0000 7fb8c8bd6700 1 -- 172.21.15.160:0/3716982887 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb8c4108130 con 0x7fb8c40f47c0 2023-12-25T20:09:27.274 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:27.273+0000 7fb8baffd700 1 -- 172.21.15.160:0/3716982887 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb8b401b320 con 0x7fb8c40f47c0 2023-12-25T20:09:27.274 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:09:27.274 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:09:27.276 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:27.274+0000 7fb8c8bd6700 1 -- 172.21.15.160:0/3716982887 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fb8b002ff60 msgr2=0x7fb8b0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:09:27.276 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:27.274+0000 7fb8c8bd6700 1 --2- 172.21.15.160:0/3716982887 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fb8b002ff60 0x7fb8b0032420 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7fb8ac0096a0 tx=0x7fb8ac006b40).stop 2023-12-25T20:09:27.276 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:27.274+0000 7fb8c8bd6700 1 -- 172.21.15.160:0/3716982887 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb8c40f47c0 msgr2=0x7fb8c4106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:09:27.276 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:27.274+0000 7fb8c8bd6700 1 --2- 172.21.15.160:0/3716982887 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb8c40f47c0 0x7fb8c4106700 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7fb8b4004b70 tx=0x7fb8b4004d10).stop 2023-12-25T20:09:27.276 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:27.275+0000 7fb8c8bd6700 1 -- 172.21.15.160:0/3716982887 shutdown_connections 2023-12-25T20:09:27.276 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:27.275+0000 7fb8c8bd6700 1 --2- 172.21.15.160:0/3716982887 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fb8b002ff60 0x7fb8b0032420 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:09:27.276 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:27.275+0000 7fb8c8bd6700 1 --2- 172.21.15.160:0/3716982887 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb8c40f47c0 0x7fb8c4106700 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:09:27.276 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:27.275+0000 7fb8c8bd6700 1 -- 172.21.15.160:0/3716982887 >> 172.21.15.160:0/3716982887 conn(0x7fb8c40f02e0 msgr2=0x7fb8c40f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:09:27.276 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:27.275+0000 7fb8c8bd6700 1 -- 172.21.15.160:0/3716982887 shutdown_connections 2023-12-25T20:09:27.276 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:27.275+0000 7fb8c8bd6700 1 -- 172.21.15.160:0/3716982887 wait complete. 2023-12-25T20:09:27.277 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:09:28.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:09:27 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:09:26.642389+0000 mgr.smithi050.nbzhun (mgr.14162) 67 : cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:09:28.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:09:27 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:09:27.274768+0000 mon.smithi050 (mon.0) 157 : audit [DBG] from='client.? 172.21.15.160:0/3716982887' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:09:28.687 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:09:28.687 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:09:29.915 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:29.912+0000 7fd30bec5700 1 -- 172.21.15.160:0/3615725498 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd3040f5410 msgr2=0x7fd3040f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:09:29.915 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:29.912+0000 7fd30bec5700 1 --2- 172.21.15.160:0/3615725498 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd3040f5410 0x7fd3040f5830 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7fd2f4009fb0 tx=0x7fd2f4009b30).stop 2023-12-25T20:09:29.915 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:29.912+0000 7fd30bec5700 1 -- 172.21.15.160:0/3615725498 shutdown_connections 2023-12-25T20:09:29.915 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:29.912+0000 7fd30bec5700 1 --2- 172.21.15.160:0/3615725498 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd3040f5410 0x7fd3040f5830 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:09:29.915 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:29.912+0000 7fd30bec5700 1 -- 172.21.15.160:0/3615725498 >> 172.21.15.160:0/3615725498 conn(0x7fd3040f0b70 msgr2=0x7fd3040f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:09:29.915 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:29.912+0000 7fd30bec5700 1 -- 172.21.15.160:0/3615725498 shutdown_connections 2023-12-25T20:09:29.915 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:29.912+0000 7fd30bec5700 1 -- 172.21.15.160:0/3615725498 wait complete. 2023-12-25T20:09:29.915 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:29.913+0000 7fd30bec5700 1 Processor -- start 2023-12-25T20:09:29.915 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:29.913+0000 7fd30bec5700 1 -- start start 2023-12-25T20:09:29.915 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:29.913+0000 7fd30bec5700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd3040f5410 0x7fd304106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:09:29.915 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:29.913+0000 7fd30bec5700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd304106c10 con 0x7fd3040f5410 2023-12-25T20:09:29.915 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:29.913+0000 7fd309c61700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd3040f5410 0x7fd304106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:09:29.916 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:29.913+0000 7fd309c61700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd3040f5410 0x7fd304106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:34606/0 (socket says 172.21.15.160:34606) 2023-12-25T20:09:29.916 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:29.913+0000 7fd309c61700 1 -- 172.21.15.160:0/2805928900 learned_addr learned my addr 172.21.15.160:0/2805928900 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:09:29.916 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:29.914+0000 7fd309c61700 1 -- 172.21.15.160:0/2805928900 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2f4005040 con 0x7fd3040f5410 2023-12-25T20:09:29.916 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:29.914+0000 7fd309c61700 1 --2- 172.21.15.160:0/2805928900 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd3040f5410 0x7fd304106720 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7fd2f4013510 tx=0x7fd2f400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:09:29.916 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:29.914+0000 7fd2fa7fc700 1 -- 172.21.15.160:0/2805928900 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd2f4009320 con 0x7fd3040f5410 2023-12-25T20:09:29.916 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:29.914+0000 7fd2fa7fc700 1 -- 172.21.15.160:0/2805928900 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd2f4013f00 con 0x7fd3040f5410 2023-12-25T20:09:29.916 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:29.914+0000 7fd30bec5700 1 -- 172.21.15.160:0/2805928900 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd304106e10 con 0x7fd3040f5410 2023-12-25T20:09:29.916 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:29.914+0000 7fd2fa7fc700 1 -- 172.21.15.160:0/2805928900 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd2f4014d90 con 0x7fd3040f5410 2023-12-25T20:09:29.916 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:29.914+0000 7fd30bec5700 1 -- 172.21.15.160:0/2805928900 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3041072a0 con 0x7fd3040f5410 2023-12-25T20:09:29.917 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:29.915+0000 7fd2fa7fc700 1 -- 172.21.15.160:0/2805928900 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd2f40095f0 con 0x7fd3040f5410 2023-12-25T20:09:29.917 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:29.915+0000 7fd30bec5700 1 -- 172.21.15.160:0/2805928900 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd304045ad0 con 0x7fd3040f5410 2023-12-25T20:09:29.917 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:29.915+0000 7fd2fa7fc700 1 --2- 172.21.15.160:0/2805928900 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fd2f002ff60 0x7fd2f0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:09:29.917 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:29.915+0000 7fd2fa7fc700 1 -- 172.21.15.160:0/2805928900 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd2f4040fe0 con 0x7fd3040f5410 2023-12-25T20:09:29.917 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:29.915+0000 7fd309460700 1 --2- 172.21.15.160:0/2805928900 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fd2f002ff60 0x7fd2f0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:09:29.917 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:29.915+0000 7fd309460700 1 --2- 172.21.15.160:0/2805928900 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fd2f002ff60 0x7fd2f0032420 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7fd3000096a0 tx=0x7fd300006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:09:29.920 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:29.919+0000 7fd2fa7fc700 1 -- 172.21.15.160:0/2805928900 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd2f401d7f0 con 0x7fd3040f5410 2023-12-25T20:09:30.124 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:30.124+0000 7fd30bec5700 1 -- 172.21.15.160:0/2805928900 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd304108150 con 0x7fd3040f5410 2023-12-25T20:09:30.125 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:30.124+0000 7fd2fa7fc700 1 -- 172.21.15.160:0/2805928900 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd2f4010230 con 0x7fd3040f5410 2023-12-25T20:09:30.125 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:09:30.125 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:09:30.127 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:30.126+0000 7fd30bec5700 1 -- 172.21.15.160:0/2805928900 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fd2f002ff60 msgr2=0x7fd2f0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:09:30.127 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:30.126+0000 7fd30bec5700 1 --2- 172.21.15.160:0/2805928900 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fd2f002ff60 0x7fd2f0032420 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7fd3000096a0 tx=0x7fd300006b40).stop 2023-12-25T20:09:30.127 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:30.126+0000 7fd30bec5700 1 -- 172.21.15.160:0/2805928900 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd3040f5410 msgr2=0x7fd304106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:09:30.127 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:30.126+0000 7fd30bec5700 1 --2- 172.21.15.160:0/2805928900 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd3040f5410 0x7fd304106720 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7fd2f4013510 tx=0x7fd2f400b040).stop 2023-12-25T20:09:30.127 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:30.126+0000 7fd30bec5700 1 -- 172.21.15.160:0/2805928900 shutdown_connections 2023-12-25T20:09:30.127 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:30.126+0000 7fd30bec5700 1 --2- 172.21.15.160:0/2805928900 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fd2f002ff60 0x7fd2f0032420 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:09:30.127 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:30.126+0000 7fd30bec5700 1 --2- 172.21.15.160:0/2805928900 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd3040f5410 0x7fd304106720 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:09:30.127 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:30.126+0000 7fd30bec5700 1 -- 172.21.15.160:0/2805928900 >> 172.21.15.160:0/2805928900 conn(0x7fd3040f0b70 msgr2=0x7fd30405fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:09:30.127 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:30.126+0000 7fd30bec5700 1 -- 172.21.15.160:0/2805928900 shutdown_connections 2023-12-25T20:09:30.127 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:30.126+0000 7fd30bec5700 1 -- 172.21.15.160:0/2805928900 wait complete. 2023-12-25T20:09:30.128 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:09:30.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:09:29 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:09:28.642722+0000 mgr.smithi050.nbzhun (mgr.14162) 68 : cluster [DBG] pgmap v61: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:09:31.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:09:30 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:09:30.125848+0000 mon.smithi050 (mon.0) 158 : audit [DBG] from='client.? 172.21.15.160:0/2805928900' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:09:31.517 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:09:31.517 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:09:32.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:09:31 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:09:30.643115+0000 mgr.smithi050.nbzhun (mgr.14162) 69 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:09:32.711 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:32.709+0000 7fa3d54a5700 1 -- 172.21.15.160:0/1329320247 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa3d00f59d0 msgr2=0x7fa3d00f5df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:09:32.711 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:32.709+0000 7fa3d54a5700 1 --2- 172.21.15.160:0/1329320247 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa3d00f59d0 0x7fa3d00f5df0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7fa3c4009fb0 tx=0x7fa3c4009b30).stop 2023-12-25T20:09:32.711 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:32.709+0000 7fa3d54a5700 1 -- 172.21.15.160:0/1329320247 shutdown_connections 2023-12-25T20:09:32.711 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:32.709+0000 7fa3d54a5700 1 --2- 172.21.15.160:0/1329320247 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa3d00f59d0 0x7fa3d00f5df0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:09:32.712 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:32.709+0000 7fa3d54a5700 1 -- 172.21.15.160:0/1329320247 >> 172.21.15.160:0/1329320247 conn(0x7fa3d00f02e0 msgr2=0x7fa3d00f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:09:32.712 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:32.709+0000 7fa3d54a5700 1 -- 172.21.15.160:0/1329320247 shutdown_connections 2023-12-25T20:09:32.712 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:32.709+0000 7fa3d54a5700 1 -- 172.21.15.160:0/1329320247 wait complete. 2023-12-25T20:09:32.712 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:32.710+0000 7fa3d54a5700 1 Processor -- start 2023-12-25T20:09:32.712 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:32.710+0000 7fa3d54a5700 1 -- start start 2023-12-25T20:09:32.712 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:32.710+0000 7fa3d54a5700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa3d00f59d0 0x7fa3d01093c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:09:32.712 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:32.710+0000 7fa3d54a5700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3d00f53b0 con 0x7fa3d00f59d0 2023-12-25T20:09:32.712 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:32.710+0000 7fa3ceffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa3d00f59d0 0x7fa3d01093c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:09:32.712 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:32.711+0000 7fa3ceffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa3d00f59d0 0x7fa3d01093c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:34620/0 (socket says 172.21.15.160:34620) 2023-12-25T20:09:32.712 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:32.711+0000 7fa3ceffd700 1 -- 172.21.15.160:0/575526017 learned_addr learned my addr 172.21.15.160:0/575526017 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:09:32.713 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:32.711+0000 7fa3ceffd700 1 -- 172.21.15.160:0/575526017 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3c4005040 con 0x7fa3d00f59d0 2023-12-25T20:09:32.714 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:32.711+0000 7fa3ceffd700 1 --2- 172.21.15.160:0/575526017 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa3d00f59d0 0x7fa3d01093c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7fa3c4013510 tx=0x7fa3c400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:09:32.714 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:32.711+0000 7fa3b77fe700 1 -- 172.21.15.160:0/575526017 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa3c4009320 con 0x7fa3d00f59d0 2023-12-25T20:09:32.714 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:32.711+0000 7fa3d54a5700 1 -- 172.21.15.160:0/575526017 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3d00f55b0 con 0x7fa3d00f59d0 2023-12-25T20:09:32.714 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:32.711+0000 7fa3b77fe700 1 -- 172.21.15.160:0/575526017 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa3c4013f00 con 0x7fa3d00f59d0 2023-12-25T20:09:32.714 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:32.711+0000 7fa3b77fe700 1 -- 172.21.15.160:0/575526017 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa3c4014d90 con 0x7fa3d00f59d0 2023-12-25T20:09:32.714 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:32.711+0000 7fa3d54a5700 1 -- 172.21.15.160:0/575526017 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3d0109a70 con 0x7fa3d00f59d0 2023-12-25T20:09:32.714 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:32.712+0000 7fa3b77fe700 1 -- 172.21.15.160:0/575526017 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa3c40263a0 con 0x7fa3d00f59d0 2023-12-25T20:09:32.714 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:32.712+0000 7fa3b77fe700 1 --2- 172.21.15.160:0/575526017 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fa3b802ffb0 0x7fa3b8032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:09:32.714 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:32.712+0000 7fa3b77fe700 1 -- 172.21.15.160:0/575526017 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa3c40410a0 con 0x7fa3d00f59d0 2023-12-25T20:09:32.714 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:32.712+0000 7fa3d54a5700 1 -- 172.21.15.160:0/575526017 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3d0045ad0 con 0x7fa3d00f59d0 2023-12-25T20:09:32.714 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:32.712+0000 7fa3ce7fc700 1 --2- 172.21.15.160:0/575526017 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fa3b802ffb0 0x7fa3b8032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:09:32.714 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:32.713+0000 7fa3ce7fc700 1 --2- 172.21.15.160:0/575526017 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fa3b802ffb0 0x7fa3b8032470 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7fa3c00096a0 tx=0x7fa3c0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:09:32.717 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:32.717+0000 7fa3b77fe700 1 -- 172.21.15.160:0/575526017 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa3c4012920 con 0x7fa3d00f59d0 2023-12-25T20:09:32.923 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:32.922+0000 7fa3d54a5700 1 -- 172.21.15.160:0/575526017 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa3d00fae50 con 0x7fa3d00f59d0 2023-12-25T20:09:32.923 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:32.923+0000 7fa3b77fe700 1 -- 172.21.15.160:0/575526017 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa3c402b020 con 0x7fa3d00f59d0 2023-12-25T20:09:32.924 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:09:32.924 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:09:32.925 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:32.924+0000 7fa3d54a5700 1 -- 172.21.15.160:0/575526017 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fa3b802ffb0 msgr2=0x7fa3b8032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:09:32.925 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:32.924+0000 7fa3d54a5700 1 --2- 172.21.15.160:0/575526017 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fa3b802ffb0 0x7fa3b8032470 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7fa3c00096a0 tx=0x7fa3c0006b40).stop 2023-12-25T20:09:32.925 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:32.924+0000 7fa3d54a5700 1 -- 172.21.15.160:0/575526017 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa3d00f59d0 msgr2=0x7fa3d01093c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:09:32.925 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:32.924+0000 7fa3d54a5700 1 --2- 172.21.15.160:0/575526017 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa3d00f59d0 0x7fa3d01093c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7fa3c4013510 tx=0x7fa3c400b040).stop 2023-12-25T20:09:32.925 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:32.924+0000 7fa3d54a5700 1 -- 172.21.15.160:0/575526017 shutdown_connections 2023-12-25T20:09:32.925 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:32.924+0000 7fa3d54a5700 1 --2- 172.21.15.160:0/575526017 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fa3b802ffb0 0x7fa3b8032470 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:09:32.926 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:32.924+0000 7fa3d54a5700 1 --2- 172.21.15.160:0/575526017 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa3d00f59d0 0x7fa3d01093c0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:09:32.926 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:32.924+0000 7fa3d54a5700 1 -- 172.21.15.160:0/575526017 >> 172.21.15.160:0/575526017 conn(0x7fa3d00f02e0 msgr2=0x7fa3d00f0dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:09:32.926 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:32.924+0000 7fa3d54a5700 1 -- 172.21.15.160:0/575526017 shutdown_connections 2023-12-25T20:09:32.926 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:32.924+0000 7fa3d54a5700 1 -- 172.21.15.160:0/575526017 wait complete. 2023-12-25T20:09:32.926 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:09:34.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:09:33 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:09:32.643490+0000 mgr.smithi050.nbzhun (mgr.14162) 70 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:09:34.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:09:33 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:09:32.924417+0000 mon.smithi050 (mon.0) 159 : audit [DBG] from='client.? 172.21.15.160:0/575526017' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:09:34.339 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:09:34.340 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:09:35.562 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:35.559+0000 7f7e5329d700 1 -- 172.21.15.160:0/3740940064 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7e4c0f51a0 msgr2=0x7f7e4c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:09:35.562 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:35.559+0000 7f7e5329d700 1 --2- 172.21.15.160:0/3740940064 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7e4c0f51a0 0x7f7e4c0f55c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f7e3c009960 tx=0x7f7e3c004d10).stop 2023-12-25T20:09:35.562 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:35.559+0000 7f7e5329d700 1 -- 172.21.15.160:0/3740940064 shutdown_connections 2023-12-25T20:09:35.562 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:35.559+0000 7f7e5329d700 1 --2- 172.21.15.160:0/3740940064 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7e4c0f51a0 0x7f7e4c0f55c0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:09:35.563 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:35.559+0000 7f7e5329d700 1 -- 172.21.15.160:0/3740940064 >> 172.21.15.160:0/3740940064 conn(0x7f7e4c0f09a0 msgr2=0x7f7e4c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:09:35.563 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:35.560+0000 7f7e5329d700 1 -- 172.21.15.160:0/3740940064 shutdown_connections 2023-12-25T20:09:35.563 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:35.560+0000 7f7e5329d700 1 -- 172.21.15.160:0/3740940064 wait complete. 2023-12-25T20:09:35.563 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:35.560+0000 7f7e5329d700 1 Processor -- start 2023-12-25T20:09:35.563 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:35.560+0000 7f7e5329d700 1 -- start start 2023-12-25T20:09:35.563 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:35.560+0000 7f7e5329d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7e4c0f51a0 0x7f7e4c1091d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:09:35.563 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:35.560+0000 7f7e5329d700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e4c060220 con 0x7f7e4c0f51a0 2023-12-25T20:09:35.563 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:35.560+0000 7f7e5229b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7e4c0f51a0 0x7f7e4c1091d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:09:35.563 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:35.560+0000 7f7e5229b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7e4c0f51a0 0x7f7e4c1091d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:34636/0 (socket says 172.21.15.160:34636) 2023-12-25T20:09:35.563 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:35.560+0000 7f7e5229b700 1 -- 172.21.15.160:0/2274979065 learned_addr learned my addr 172.21.15.160:0/2274979065 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:09:35.563 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:35.561+0000 7f7e5229b700 1 -- 172.21.15.160:0/2274979065 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e3c009640 con 0x7f7e4c0f51a0 2023-12-25T20:09:35.563 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:35.561+0000 7f7e5229b700 1 --2- 172.21.15.160:0/2274979065 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7e4c0f51a0 0x7f7e4c1091d0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f7e3c007bd0 tx=0x7f7e3c004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:09:35.563 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:35.561+0000 7f7e4affd700 1 -- 172.21.15.160:0/2274979065 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7e3c018070 con 0x7f7e4c0f51a0 2023-12-25T20:09:35.563 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:35.561+0000 7f7e4affd700 1 -- 172.21.15.160:0/2274979065 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7e3c00b870 con 0x7f7e4c0f51a0 2023-12-25T20:09:35.563 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:35.561+0000 7f7e5329d700 1 -- 172.21.15.160:0/2274979065 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e4c060420 con 0x7f7e4c0f51a0 2023-12-25T20:09:35.564 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:35.561+0000 7f7e4affd700 1 -- 172.21.15.160:0/2274979065 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7e3c01d420 con 0x7f7e4c0f51a0 2023-12-25T20:09:35.564 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:35.561+0000 7f7e5329d700 1 -- 172.21.15.160:0/2274979065 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e4c109880 con 0x7f7e4c0f51a0 2023-12-25T20:09:35.564 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:35.562+0000 7f7e4affd700 1 -- 172.21.15.160:0/2274979065 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7e3c024060 con 0x7f7e4c0f51a0 2023-12-25T20:09:35.564 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:35.562+0000 7f7e5329d700 1 -- 172.21.15.160:0/2274979065 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e4c059100 con 0x7f7e4c0f51a0 2023-12-25T20:09:35.565 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:35.562+0000 7f7e4affd700 1 --2- 172.21.15.160:0/2274979065 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f7e3402ff60 0x7f7e34032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:09:35.565 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:35.562+0000 7f7e4affd700 1 -- 172.21.15.160:0/2274979065 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7e3c041150 con 0x7f7e4c0f51a0 2023-12-25T20:09:35.565 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:35.562+0000 7f7e51a9a700 1 --2- 172.21.15.160:0/2274979065 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f7e3402ff60 0x7f7e34032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:09:35.565 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:35.563+0000 7f7e51a9a700 1 --2- 172.21.15.160:0/2274979065 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f7e3402ff60 0x7f7e34032420 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f7e400096a0 tx=0x7f7e40006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:09:35.567 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:35.567+0000 7f7e4affd700 1 -- 172.21.15.160:0/2274979065 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7e3c025520 con 0x7f7e4c0f51a0 2023-12-25T20:09:35.771 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:35.771+0000 7f7e5329d700 1 -- 172.21.15.160:0/2274979065 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7e4c045ad0 con 0x7f7e4c0f51a0 2023-12-25T20:09:35.772 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:35.771+0000 7f7e4affd700 1 -- 172.21.15.160:0/2274979065 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7e3c01a2d0 con 0x7f7e4c0f51a0 2023-12-25T20:09:35.772 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:09:35.772 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:09:35.774 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:35.773+0000 7f7e5329d700 1 -- 172.21.15.160:0/2274979065 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f7e3402ff60 msgr2=0x7f7e34032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:09:35.774 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:35.773+0000 7f7e5329d700 1 --2- 172.21.15.160:0/2274979065 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f7e3402ff60 0x7f7e34032420 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f7e400096a0 tx=0x7f7e40006b40).stop 2023-12-25T20:09:35.774 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:35.773+0000 7f7e5329d700 1 -- 172.21.15.160:0/2274979065 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7e4c0f51a0 msgr2=0x7f7e4c1091d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:09:35.774 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:35.773+0000 7f7e5329d700 1 --2- 172.21.15.160:0/2274979065 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7e4c0f51a0 0x7f7e4c1091d0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f7e3c007bd0 tx=0x7f7e3c004d10).stop 2023-12-25T20:09:35.774 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:35.773+0000 7f7e5329d700 1 -- 172.21.15.160:0/2274979065 shutdown_connections 2023-12-25T20:09:35.774 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:35.773+0000 7f7e5329d700 1 --2- 172.21.15.160:0/2274979065 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f7e3402ff60 0x7f7e34032420 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:09:35.774 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:35.773+0000 7f7e5329d700 1 --2- 172.21.15.160:0/2274979065 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7e4c0f51a0 0x7f7e4c1091d0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:09:35.774 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:35.773+0000 7f7e5329d700 1 -- 172.21.15.160:0/2274979065 >> 172.21.15.160:0/2274979065 conn(0x7f7e4c0f09a0 msgr2=0x7f7e4c0f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:09:35.775 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:35.773+0000 7f7e5329d700 1 -- 172.21.15.160:0/2274979065 shutdown_connections 2023-12-25T20:09:35.775 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:35.773+0000 7f7e5329d700 1 -- 172.21.15.160:0/2274979065 wait complete. 2023-12-25T20:09:35.775 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:09:36.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:09:35 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:09:34.643818+0000 mgr.smithi050.nbzhun (mgr.14162) 71 : cluster [DBG] pgmap v64: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:09:36.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:09:35 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:09:35.772998+0000 mon.smithi050 (mon.0) 160 : audit [DBG] from='client.? 172.21.15.160:0/2274979065' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:09:37.160 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:09:37.160 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:09:38.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:09:37 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:09:36.644185+0000 mgr.smithi050.nbzhun (mgr.14162) 72 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:09:38.415 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:38.413+0000 7f60e272f700 1 -- 172.21.15.160:0/2757561692 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f60dc060200 msgr2=0x7f60dc0fb570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:09:38.415 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:38.413+0000 7f60e272f700 1 --2- 172.21.15.160:0/2757561692 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f60dc060200 0x7f60dc0fb570 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f60cc009fb0 tx=0x7f60cc009b30).stop 2023-12-25T20:09:38.415 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:38.413+0000 7f60e272f700 1 -- 172.21.15.160:0/2757561692 shutdown_connections 2023-12-25T20:09:38.416 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:38.413+0000 7f60e272f700 1 --2- 172.21.15.160:0/2757561692 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f60dc060200 0x7f60dc0fb570 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:09:38.416 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:38.413+0000 7f60e272f700 1 -- 172.21.15.160:0/2757561692 >> 172.21.15.160:0/2757561692 conn(0x7f60dc0f0b70 msgr2=0x7f60dc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:09:38.416 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:38.413+0000 7f60e272f700 1 -- 172.21.15.160:0/2757561692 shutdown_connections 2023-12-25T20:09:38.416 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:38.413+0000 7f60e272f700 1 -- 172.21.15.160:0/2757561692 wait complete. 2023-12-25T20:09:38.416 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:38.414+0000 7f60e272f700 1 Processor -- start 2023-12-25T20:09:38.416 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:38.414+0000 7f60e272f700 1 -- start start 2023-12-25T20:09:38.416 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:38.414+0000 7f60e272f700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f60dc060200 0x7f60dc1088d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:09:38.416 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:38.414+0000 7f60e272f700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60dc108dc0 con 0x7f60dc060200 2023-12-25T20:09:38.416 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:38.414+0000 7f60dbfff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f60dc060200 0x7f60dc1088d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:09:38.416 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:38.414+0000 7f60dbfff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f60dc060200 0x7f60dc1088d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:57168/0 (socket says 172.21.15.160:57168) 2023-12-25T20:09:38.416 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:38.414+0000 7f60dbfff700 1 -- 172.21.15.160:0/1479781835 learned_addr learned my addr 172.21.15.160:0/1479781835 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:09:38.416 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:38.415+0000 7f60dbfff700 1 -- 172.21.15.160:0/1479781835 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60cc005040 con 0x7f60dc060200 2023-12-25T20:09:38.416 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:38.415+0000 7f60dbfff700 1 --2- 172.21.15.160:0/1479781835 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f60dc060200 0x7f60dc1088d0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f60cc009830 tx=0x7f60cc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:09:38.418 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:38.415+0000 7f60d8ff9700 1 -- 172.21.15.160:0/1479781835 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f60cc009320 con 0x7f60dc060200 2023-12-25T20:09:38.418 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:38.415+0000 7f60e272f700 1 -- 172.21.15.160:0/1479781835 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60dc108fc0 con 0x7f60dc060200 2023-12-25T20:09:38.418 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:38.415+0000 7f60d8ff9700 1 -- 172.21.15.160:0/1479781835 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f60cc0136c0 con 0x7f60dc060200 2023-12-25T20:09:38.418 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:38.415+0000 7f60d8ff9700 1 -- 172.21.15.160:0/1479781835 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f60cc01ee90 con 0x7f60dc060200 2023-12-25T20:09:38.418 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:38.415+0000 7f60e272f700 1 -- 172.21.15.160:0/1479781835 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60dc109450 con 0x7f60dc060200 2023-12-25T20:09:38.418 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:38.416+0000 7f60d8ff9700 1 -- 172.21.15.160:0/1479781835 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f60cc014440 con 0x7f60dc060200 2023-12-25T20:09:38.418 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:38.416+0000 7f60e272f700 1 -- 172.21.15.160:0/1479781835 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60dc045ad0 con 0x7f60dc060200 2023-12-25T20:09:38.418 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:38.416+0000 7f60d8ff9700 1 --2- 172.21.15.160:0/1479781835 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f60c802ff10 0x7f60c80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:09:38.418 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:38.416+0000 7f60d8ff9700 1 -- 172.21.15.160:0/1479781835 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f60cc041300 con 0x7f60dc060200 2023-12-25T20:09:38.418 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:38.416+0000 7f60db7fe700 1 --2- 172.21.15.160:0/1479781835 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f60c802ff10 0x7f60c80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:09:38.418 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:38.417+0000 7f60db7fe700 1 --2- 172.21.15.160:0/1479781835 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f60c802ff10 0x7f60c80323d0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f60c40096a0 tx=0x7f60c4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:09:38.421 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:38.421+0000 7f60d8ff9700 1 -- 172.21.15.160:0/1479781835 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f60cc01c320 con 0x7f60dc060200 2023-12-25T20:09:38.625 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:38.625+0000 7f60e272f700 1 -- 172.21.15.160:0/1479781835 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f60dc10a300 con 0x7f60dc060200 2023-12-25T20:09:38.626 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:38.625+0000 7f60d8ff9700 1 -- 172.21.15.160:0/1479781835 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f60cc01c320 con 0x7f60dc060200 2023-12-25T20:09:38.626 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:09:38.626 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:09:38.628 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:38.627+0000 7f60e272f700 1 -- 172.21.15.160:0/1479781835 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f60c802ff10 msgr2=0x7f60c80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:09:38.628 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:38.627+0000 7f60e272f700 1 --2- 172.21.15.160:0/1479781835 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f60c802ff10 0x7f60c80323d0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f60c40096a0 tx=0x7f60c4006b40).stop 2023-12-25T20:09:38.628 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:38.627+0000 7f60e272f700 1 -- 172.21.15.160:0/1479781835 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f60dc060200 msgr2=0x7f60dc1088d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:09:38.628 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:38.627+0000 7f60e272f700 1 --2- 172.21.15.160:0/1479781835 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f60dc060200 0x7f60dc1088d0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f60cc009830 tx=0x7f60cc00b040).stop 2023-12-25T20:09:38.628 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:38.627+0000 7f60e272f700 1 -- 172.21.15.160:0/1479781835 shutdown_connections 2023-12-25T20:09:38.628 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:38.627+0000 7f60e272f700 1 --2- 172.21.15.160:0/1479781835 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f60c802ff10 0x7f60c80323d0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:09:38.628 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:38.627+0000 7f60e272f700 1 --2- 172.21.15.160:0/1479781835 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f60dc060200 0x7f60dc1088d0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:09:38.629 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:38.627+0000 7f60e272f700 1 -- 172.21.15.160:0/1479781835 >> 172.21.15.160:0/1479781835 conn(0x7f60dc0f0b70 msgr2=0x7f60dc0f1670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:09:38.629 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:38.627+0000 7f60e272f700 1 -- 172.21.15.160:0/1479781835 shutdown_connections 2023-12-25T20:09:38.629 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:38.627+0000 7f60e272f700 1 -- 172.21.15.160:0/1479781835 wait complete. 2023-12-25T20:09:38.629 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:09:39.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:09:38 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:09:38.627027+0000 mon.smithi050 (mon.0) 161 : audit [DBG] from='client.? 172.21.15.160:0/1479781835' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:09:40.028 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:09:40.029 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:09:40.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:09:39 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:09:38.644491+0000 mgr.smithi050.nbzhun (mgr.14162) 73 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:09:41.196 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:41.195+0000 7f3fe4e1e700 1 -- 172.21.15.160:0/840000868 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3fdc0138a0 con 0x7f3fe0060280 2023-12-25T20:09:41.196 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:41.195+0000 7f3fe8084700 1 -- 172.21.15.160:0/840000868 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3fe0060280 msgr2=0x7f3fe00f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:09:41.196 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:41.195+0000 7f3fe8084700 1 --2- 172.21.15.160:0/840000868 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3fe0060280 0x7f3fe00f90f0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f3fdc009fb0 tx=0x7f3fdc009b30).stop 2023-12-25T20:09:41.196 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:41.195+0000 7f3fe8084700 1 -- 172.21.15.160:0/840000868 shutdown_connections 2023-12-25T20:09:41.196 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:41.195+0000 7f3fe8084700 1 --2- 172.21.15.160:0/840000868 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3fe0060280 0x7f3fe00f90f0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:09:41.197 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:41.195+0000 7f3fe8084700 1 -- 172.21.15.160:0/840000868 >> 172.21.15.160:0/840000868 conn(0x7f3fe00f0b70 msgr2=0x7f3fe00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:09:41.197 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:41.195+0000 7f3fe8084700 1 -- 172.21.15.160:0/840000868 shutdown_connections 2023-12-25T20:09:41.197 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:41.195+0000 7f3fe8084700 1 -- 172.21.15.160:0/840000868 wait complete. 2023-12-25T20:09:41.198 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:41.196+0000 7f3fe8084700 1 Processor -- start 2023-12-25T20:09:41.198 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:41.196+0000 7f3fe8084700 1 -- start start 2023-12-25T20:09:41.198 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:41.196+0000 7f3fe8084700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3fe0060280 0x7f3fe01066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:09:41.198 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:41.196+0000 7f3fe8084700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fe0106bd0 con 0x7f3fe0060280 2023-12-25T20:09:41.198 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:41.197+0000 7f3fe5e20700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3fe0060280 0x7f3fe01066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:09:41.198 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:41.197+0000 7f3fe5e20700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3fe0060280 0x7f3fe01066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:57194/0 (socket says 172.21.15.160:57194) 2023-12-25T20:09:41.199 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:41.197+0000 7f3fe5e20700 1 -- 172.21.15.160:0/4194452938 learned_addr learned my addr 172.21.15.160:0/4194452938 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:09:41.199 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:41.197+0000 7f3fe5e20700 1 -- 172.21.15.160:0/4194452938 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fdc005040 con 0x7f3fe0060280 2023-12-25T20:09:41.199 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:41.197+0000 7f3fe5e20700 1 --2- 172.21.15.160:0/4194452938 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3fe0060280 0x7f3fe01066e0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f3fdc013510 tx=0x7f3fdc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:09:41.199 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:41.197+0000 7f3fd67fc700 1 -- 172.21.15.160:0/4194452938 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3fdc009320 con 0x7f3fe0060280 2023-12-25T20:09:41.199 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:41.197+0000 7f3fd67fc700 1 -- 172.21.15.160:0/4194452938 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3fdc013f00 con 0x7f3fe0060280 2023-12-25T20:09:41.199 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:41.197+0000 7f3fe8084700 1 -- 172.21.15.160:0/4194452938 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fe0106dd0 con 0x7f3fe0060280 2023-12-25T20:09:41.199 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:41.197+0000 7f3fd67fc700 1 -- 172.21.15.160:0/4194452938 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3fdc014dc0 con 0x7f3fe0060280 2023-12-25T20:09:41.199 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:41.197+0000 7f3fe8084700 1 -- 172.21.15.160:0/4194452938 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fe0107260 con 0x7f3fe0060280 2023-12-25T20:09:41.200 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:41.198+0000 7f3fd67fc700 1 -- 172.21.15.160:0/4194452938 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3fdc0095f0 con 0x7f3fe0060280 2023-12-25T20:09:41.200 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:41.198+0000 7f3fe8084700 1 -- 172.21.15.160:0/4194452938 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fe0045ad0 con 0x7f3fe0060280 2023-12-25T20:09:41.200 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:41.198+0000 7f3fd67fc700 1 --2- 172.21.15.160:0/4194452938 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f3fcc02ff60 0x7f3fcc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:09:41.200 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:41.198+0000 7f3fd67fc700 1 -- 172.21.15.160:0/4194452938 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3fdc041000 con 0x7f3fe0060280 2023-12-25T20:09:41.200 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:41.198+0000 7f3fe561f700 1 --2- 172.21.15.160:0/4194452938 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f3fcc02ff60 0x7f3fcc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:09:41.200 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:41.199+0000 7f3fe561f700 1 --2- 172.21.15.160:0/4194452938 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f3fcc02ff60 0x7f3fcc032420 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f3fd00096a0 tx=0x7f3fd0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:09:41.203 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:41.204+0000 7f3fd67fc700 1 -- 172.21.15.160:0/4194452938 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3fdc014420 con 0x7f3fe0060280 2023-12-25T20:09:41.411 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:41.410+0000 7f3fe8084700 1 -- 172.21.15.160:0/4194452938 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3fe0108110 con 0x7f3fe0060280 2023-12-25T20:09:41.412 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:41.411+0000 7f3fd67fc700 1 -- 172.21.15.160:0/4194452938 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3fdc010230 con 0x7f3fe0060280 2023-12-25T20:09:41.412 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:09:41.412 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:09:41.413 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:41.413+0000 7f3fe8084700 1 -- 172.21.15.160:0/4194452938 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f3fcc02ff60 msgr2=0x7f3fcc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:09:41.414 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:41.413+0000 7f3fe8084700 1 --2- 172.21.15.160:0/4194452938 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f3fcc02ff60 0x7f3fcc032420 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f3fd00096a0 tx=0x7f3fd0006b40).stop 2023-12-25T20:09:41.414 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:41.413+0000 7f3fe8084700 1 -- 172.21.15.160:0/4194452938 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3fe0060280 msgr2=0x7f3fe01066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:09:41.414 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:41.413+0000 7f3fe8084700 1 --2- 172.21.15.160:0/4194452938 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3fe0060280 0x7f3fe01066e0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f3fdc013510 tx=0x7f3fdc00b040).stop 2023-12-25T20:09:41.414 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:41.413+0000 7f3fe8084700 1 -- 172.21.15.160:0/4194452938 shutdown_connections 2023-12-25T20:09:41.414 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:41.413+0000 7f3fe8084700 1 --2- 172.21.15.160:0/4194452938 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f3fcc02ff60 0x7f3fcc032420 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:09:41.414 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:41.413+0000 7f3fe8084700 1 --2- 172.21.15.160:0/4194452938 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3fe0060280 0x7f3fe01066e0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:09:41.414 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:41.413+0000 7f3fe8084700 1 -- 172.21.15.160:0/4194452938 >> 172.21.15.160:0/4194452938 conn(0x7f3fe00f0b70 msgr2=0x7f3fe00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:09:41.414 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:41.413+0000 7f3fe8084700 1 -- 172.21.15.160:0/4194452938 shutdown_connections 2023-12-25T20:09:41.414 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:41.413+0000 7f3fe8084700 1 -- 172.21.15.160:0/4194452938 wait complete. 2023-12-25T20:09:41.414 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:09:42.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:09:41 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:09:40.644813+0000 mgr.smithi050.nbzhun (mgr.14162) 74 : cluster [DBG] pgmap v67: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:09:42.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:09:41 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:09:41.412743+0000 mon.smithi050 (mon.0) 162 : audit [DBG] from='client.? 172.21.15.160:0/4194452938' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:09:42.781 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:09:42.782 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:09:43.985 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:43.984+0000 7f89bd144700 1 -- 172.21.15.160:0/3167737608 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f89b80f6510 msgr2=0x7f89b80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:09:43.985 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:43.984+0000 7f89bd144700 1 --2- 172.21.15.160:0/3167737608 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f89b80f6510 0x7f89b80f6930 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f89ac009fb0 tx=0x7f89ac009b30).stop 2023-12-25T20:09:43.986 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:43.984+0000 7f89bd144700 1 -- 172.21.15.160:0/3167737608 shutdown_connections 2023-12-25T20:09:43.986 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:43.984+0000 7f89bd144700 1 --2- 172.21.15.160:0/3167737608 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f89b80f6510 0x7f89b80f6930 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:09:43.986 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:43.984+0000 7f89bd144700 1 -- 172.21.15.160:0/3167737608 >> 172.21.15.160:0/3167737608 conn(0x7f89b80f1c70 msgr2=0x7f89b80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:09:43.986 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:43.984+0000 7f89bd144700 1 -- 172.21.15.160:0/3167737608 shutdown_connections 2023-12-25T20:09:43.986 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:43.984+0000 7f89bd144700 1 -- 172.21.15.160:0/3167737608 wait complete. 2023-12-25T20:09:43.986 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:43.984+0000 7f89bd144700 1 Processor -- start 2023-12-25T20:09:43.986 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:43.984+0000 7f89bd144700 1 -- start start 2023-12-25T20:09:43.986 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:43.985+0000 7f89bd144700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f89b80f6510 0x7f89b8106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:09:43.986 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:43.985+0000 7f89bd144700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89b8106c20 con 0x7f89b80f6510 2023-12-25T20:09:43.987 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:43.985+0000 7f89b6d9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f89b80f6510 0x7f89b8106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:09:43.987 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:43.985+0000 7f89b6d9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f89b80f6510 0x7f89b8106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:57218/0 (socket says 172.21.15.160:57218) 2023-12-25T20:09:43.988 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:43.985+0000 7f89b6d9d700 1 -- 172.21.15.160:0/786216629 learned_addr learned my addr 172.21.15.160:0/786216629 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:09:43.988 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:43.985+0000 7f89b6d9d700 1 -- 172.21.15.160:0/786216629 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89ac005040 con 0x7f89b80f6510 2023-12-25T20:09:43.988 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:43.985+0000 7f89b6d9d700 1 --2- 172.21.15.160:0/786216629 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f89b80f6510 0x7f89b8106730 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f89ac013510 tx=0x7f89ac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:09:43.988 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:43.986+0000 7f899f7fe700 1 -- 172.21.15.160:0/786216629 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f89ac009320 con 0x7f89b80f6510 2023-12-25T20:09:43.988 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:43.986+0000 7f899f7fe700 1 -- 172.21.15.160:0/786216629 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f89ac013f00 con 0x7f89b80f6510 2023-12-25T20:09:43.988 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:43.986+0000 7f89bd144700 1 -- 172.21.15.160:0/786216629 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89b8106e20 con 0x7f89b80f6510 2023-12-25T20:09:43.988 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:43.986+0000 7f899f7fe700 1 -- 172.21.15.160:0/786216629 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f89ac014d90 con 0x7f89b80f6510 2023-12-25T20:09:43.988 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:43.986+0000 7f89bd144700 1 -- 172.21.15.160:0/786216629 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89b81072b0 con 0x7f89b80f6510 2023-12-25T20:09:43.988 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:43.987+0000 7f899f7fe700 1 -- 172.21.15.160:0/786216629 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f89ac0263a0 con 0x7f89b80f6510 2023-12-25T20:09:43.988 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:43.987+0000 7f899f7fe700 1 --2- 172.21.15.160:0/786216629 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f89a002ff60 0x7f89a0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:09:43.988 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:43.987+0000 7f899f7fe700 1 -- 172.21.15.160:0/786216629 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f89ac041f00 con 0x7f89b80f6510 2023-12-25T20:09:43.988 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:43.987+0000 7f89bd144700 1 -- 172.21.15.160:0/786216629 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89b8059100 con 0x7f89b80f6510 2023-12-25T20:09:43.988 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:43.987+0000 7f89b659c700 1 --2- 172.21.15.160:0/786216629 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f89a002ff60 0x7f89a0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:09:43.989 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:43.987+0000 7f89b659c700 1 --2- 172.21.15.160:0/786216629 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f89a002ff60 0x7f89a0032420 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f89a80096a0 tx=0x7f89a8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:09:43.992 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:43.992+0000 7f899f7fe700 1 -- 172.21.15.160:0/786216629 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f89ac012920 con 0x7f89b80f6510 2023-12-25T20:09:44.196 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:44.195+0000 7f89bd144700 1 -- 172.21.15.160:0/786216629 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f89b80fbf60 con 0x7f89b80f6510 2023-12-25T20:09:44.196 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:44.196+0000 7f899f7fe700 1 -- 172.21.15.160:0/786216629 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f89ac02b020 con 0x7f89b80f6510 2023-12-25T20:09:44.197 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:09:44.197 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:09:44.198 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:44.197+0000 7f89bd144700 1 -- 172.21.15.160:0/786216629 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f89a002ff60 msgr2=0x7f89a0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:09:44.198 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:44.197+0000 7f89bd144700 1 --2- 172.21.15.160:0/786216629 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f89a002ff60 0x7f89a0032420 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f89a80096a0 tx=0x7f89a8006b40).stop 2023-12-25T20:09:44.198 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:44.197+0000 7f89bd144700 1 -- 172.21.15.160:0/786216629 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f89b80f6510 msgr2=0x7f89b8106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:09:44.198 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:44.197+0000 7f89bd144700 1 --2- 172.21.15.160:0/786216629 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f89b80f6510 0x7f89b8106730 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f89ac013510 tx=0x7f89ac00b040).stop 2023-12-25T20:09:44.199 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:44.197+0000 7f89bd144700 1 -- 172.21.15.160:0/786216629 shutdown_connections 2023-12-25T20:09:44.199 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:44.197+0000 7f89bd144700 1 --2- 172.21.15.160:0/786216629 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f89a002ff60 0x7f89a0032420 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:09:44.199 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:44.197+0000 7f89bd144700 1 --2- 172.21.15.160:0/786216629 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f89b80f6510 0x7f89b8106730 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:09:44.199 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:44.197+0000 7f89bd144700 1 -- 172.21.15.160:0/786216629 >> 172.21.15.160:0/786216629 conn(0x7f89b80f1c70 msgr2=0x7f89b80f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:09:44.199 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:44.197+0000 7f89bd144700 1 -- 172.21.15.160:0/786216629 shutdown_connections 2023-12-25T20:09:44.199 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:44.197+0000 7f89bd144700 1 -- 172.21.15.160:0/786216629 wait complete. 2023-12-25T20:09:44.199 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:09:44.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:09:43 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:09:42.645105+0000 mgr.smithi050.nbzhun (mgr.14162) 75 : cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:09:45.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:09:44 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:09:44.197284+0000 mon.smithi050 (mon.0) 163 : audit [DBG] from='client.? 172.21.15.160:0/786216629' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:09:45.635 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:09:45.635 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:09:46.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:09:45 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:09:44.645436+0000 mgr.smithi050.nbzhun (mgr.14162) 76 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:09:47.305 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:47.303+0000 7fea1bb66700 1 -- 172.21.15.160:0/4105824214 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fea140f5410 msgr2=0x7fea140f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:09:47.305 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:47.303+0000 7fea1bb66700 1 --2- 172.21.15.160:0/4105824214 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fea140f5410 0x7fea140f5830 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7fea04009fb0 tx=0x7fea04009b30).stop 2023-12-25T20:09:47.305 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:47.303+0000 7fea1bb66700 1 -- 172.21.15.160:0/4105824214 shutdown_connections 2023-12-25T20:09:47.305 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:47.303+0000 7fea1bb66700 1 --2- 172.21.15.160:0/4105824214 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fea140f5410 0x7fea140f5830 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:09:47.305 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:47.303+0000 7fea1bb66700 1 -- 172.21.15.160:0/4105824214 >> 172.21.15.160:0/4105824214 conn(0x7fea140f0b70 msgr2=0x7fea140f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:09:47.305 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:47.303+0000 7fea1bb66700 1 -- 172.21.15.160:0/4105824214 shutdown_connections 2023-12-25T20:09:47.305 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:47.303+0000 7fea1bb66700 1 -- 172.21.15.160:0/4105824214 wait complete. 2023-12-25T20:09:47.305 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:47.304+0000 7fea1bb66700 1 Processor -- start 2023-12-25T20:09:47.305 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:47.304+0000 7fea1bb66700 1 -- start start 2023-12-25T20:09:47.305 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:47.304+0000 7fea1bb66700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fea140f5410 0x7fea14109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:09:47.306 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:47.304+0000 7fea1bb66700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea14060220 con 0x7fea140f5410 2023-12-25T20:09:47.306 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:47.304+0000 7fea19902700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fea140f5410 0x7fea14109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:09:47.306 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:47.304+0000 7fea19902700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fea140f5410 0x7fea14109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:51428/0 (socket says 172.21.15.160:51428) 2023-12-25T20:09:47.306 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:47.304+0000 7fea19902700 1 -- 172.21.15.160:0/3868181823 learned_addr learned my addr 172.21.15.160:0/3868181823 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:09:47.306 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:47.305+0000 7fea19902700 1 -- 172.21.15.160:0/3868181823 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea04005040 con 0x7fea140f5410 2023-12-25T20:09:47.306 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:47.305+0000 7fea19902700 1 --2- 172.21.15.160:0/3868181823 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fea140f5410 0x7fea14109450 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7fea04013510 tx=0x7fea0400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:09:47.306 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:47.305+0000 7fea0a7fc700 1 -- 172.21.15.160:0/3868181823 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fea04009320 con 0x7fea140f5410 2023-12-25T20:09:47.306 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:47.305+0000 7fea1bb66700 1 -- 172.21.15.160:0/3868181823 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea14060420 con 0x7fea140f5410 2023-12-25T20:09:47.306 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:47.305+0000 7fea0a7fc700 1 -- 172.21.15.160:0/3868181823 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fea04013f00 con 0x7fea140f5410 2023-12-25T20:09:47.306 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:47.305+0000 7fea0a7fc700 1 -- 172.21.15.160:0/3868181823 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fea04014d90 con 0x7fea140f5410 2023-12-25T20:09:47.306 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:47.305+0000 7fea1bb66700 1 -- 172.21.15.160:0/3868181823 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea14109b00 con 0x7fea140f5410 2023-12-25T20:09:47.307 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:47.306+0000 7fea0a7fc700 1 -- 172.21.15.160:0/3868181823 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fea040095f0 con 0x7fea140f5410 2023-12-25T20:09:47.307 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:47.306+0000 7fea1bb66700 1 -- 172.21.15.160:0/3868181823 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea14059100 con 0x7fea140f5410 2023-12-25T20:09:47.307 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:47.306+0000 7fea0a7fc700 1 --2- 172.21.15.160:0/3868181823 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fea0002ff60 0x7fea00032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:09:47.307 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:47.306+0000 7fea0a7fc700 1 -- 172.21.15.160:0/3868181823 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fea04041000 con 0x7fea140f5410 2023-12-25T20:09:47.307 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:47.306+0000 7fea19101700 1 --2- 172.21.15.160:0/3868181823 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fea0002ff60 0x7fea00032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:09:47.307 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:47.307+0000 7fea19101700 1 --2- 172.21.15.160:0/3868181823 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fea0002ff60 0x7fea00032420 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7fea100096a0 tx=0x7fea10006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:09:47.311 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:47.311+0000 7fea0a7fc700 1 -- 172.21.15.160:0/3868181823 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fea04014420 con 0x7fea140f5410 2023-12-25T20:09:47.516 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:47.516+0000 7fea1bb66700 1 -- 172.21.15.160:0/3868181823 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fea14045ad0 con 0x7fea140f5410 2023-12-25T20:09:47.517 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:47.517+0000 7fea0a7fc700 1 -- 172.21.15.160:0/3868181823 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fea04010230 con 0x7fea140f5410 2023-12-25T20:09:47.517 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:09:47.517 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:09:47.518 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:47.518+0000 7fea1bb66700 1 -- 172.21.15.160:0/3868181823 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fea0002ff60 msgr2=0x7fea00032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:09:47.518 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:47.518+0000 7fea1bb66700 1 --2- 172.21.15.160:0/3868181823 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fea0002ff60 0x7fea00032420 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7fea100096a0 tx=0x7fea10006b40).stop 2023-12-25T20:09:47.518 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:47.518+0000 7fea1bb66700 1 -- 172.21.15.160:0/3868181823 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fea140f5410 msgr2=0x7fea14109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:09:47.519 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:47.518+0000 7fea1bb66700 1 --2- 172.21.15.160:0/3868181823 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fea140f5410 0x7fea14109450 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7fea04013510 tx=0x7fea0400b040).stop 2023-12-25T20:09:47.519 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:47.519+0000 7fea1bb66700 1 -- 172.21.15.160:0/3868181823 shutdown_connections 2023-12-25T20:09:47.519 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:47.519+0000 7fea1bb66700 1 --2- 172.21.15.160:0/3868181823 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fea0002ff60 0x7fea00032420 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:09:47.519 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:47.519+0000 7fea1bb66700 1 --2- 172.21.15.160:0/3868181823 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fea140f5410 0x7fea14109450 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:09:47.519 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:47.519+0000 7fea1bb66700 1 -- 172.21.15.160:0/3868181823 >> 172.21.15.160:0/3868181823 conn(0x7fea140f0b70 msgr2=0x7fea140f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:09:47.519 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:47.519+0000 7fea1bb66700 1 -- 172.21.15.160:0/3868181823 shutdown_connections 2023-12-25T20:09:47.519 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:47.519+0000 7fea1bb66700 1 -- 172.21.15.160:0/3868181823 wait complete. 2023-12-25T20:09:47.519 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:09:48.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:09:47 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:09:46.645752+0000 mgr.smithi050.nbzhun (mgr.14162) 77 : cluster [DBG] pgmap v70: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:09:48.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:09:47 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:09:47.517500+0000 mon.smithi050 (mon.0) 164 : audit [DBG] from='client.? 172.21.15.160:0/3868181823' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:09:48.916 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:09:48.916 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:09:50.209 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:50.208+0000 7f9016cfd700 1 -- 172.21.15.160:0/2005365082 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f90100f5410 msgr2=0x7f90100f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:09:50.209 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:50.208+0000 7f9016cfd700 1 --2- 172.21.15.160:0/2005365082 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f90100f5410 0x7f90100f5830 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f9004009fb0 tx=0x7f9004009b30).stop 2023-12-25T20:09:50.209 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:50.208+0000 7f9016cfd700 1 -- 172.21.15.160:0/2005365082 shutdown_connections 2023-12-25T20:09:50.209 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:50.208+0000 7f9016cfd700 1 --2- 172.21.15.160:0/2005365082 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f90100f5410 0x7f90100f5830 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:09:50.209 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:50.208+0000 7f9016cfd700 1 -- 172.21.15.160:0/2005365082 >> 172.21.15.160:0/2005365082 conn(0x7f90100f0b70 msgr2=0x7f90100f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:09:50.209 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:50.208+0000 7f9016cfd700 1 -- 172.21.15.160:0/2005365082 shutdown_connections 2023-12-25T20:09:50.209 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:50.208+0000 7f9016cfd700 1 -- 172.21.15.160:0/2005365082 wait complete. 2023-12-25T20:09:50.210 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:50.209+0000 7f9016cfd700 1 Processor -- start 2023-12-25T20:09:50.210 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:50.209+0000 7f9016cfd700 1 -- start start 2023-12-25T20:09:50.210 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:50.209+0000 7f9016cfd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f90100f5410 0x7f901010b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:09:50.210 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:50.209+0000 7f9016cfd700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90101071e0 con 0x7f90100f5410 2023-12-25T20:09:50.211 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:50.210+0000 7f9014a99700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f90100f5410 0x7f901010b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:09:50.211 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:50.210+0000 7f9014a99700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f90100f5410 0x7f901010b7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:51446/0 (socket says 172.21.15.160:51446) 2023-12-25T20:09:50.211 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:50.210+0000 7f9014a99700 1 -- 172.21.15.160:0/793811522 learned_addr learned my addr 172.21.15.160:0/793811522 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:09:50.211 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:50.210+0000 7f9014a99700 1 -- 172.21.15.160:0/793811522 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9004005040 con 0x7f90100f5410 2023-12-25T20:09:50.211 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:50.210+0000 7f9014a99700 1 --2- 172.21.15.160:0/793811522 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f90100f5410 0x7f901010b7f0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f9004013510 tx=0x7f900400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:09:50.211 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:50.210+0000 7f900d7fa700 1 -- 172.21.15.160:0/793811522 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9004009320 con 0x7f90100f5410 2023-12-25T20:09:50.211 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:50.210+0000 7f9016cfd700 1 -- 172.21.15.160:0/793811522 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9010060220 con 0x7f90100f5410 2023-12-25T20:09:50.212 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:50.210+0000 7f900d7fa700 1 -- 172.21.15.160:0/793811522 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9004013f00 con 0x7f90100f5410 2023-12-25T20:09:50.212 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:50.210+0000 7f900d7fa700 1 -- 172.21.15.160:0/793811522 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9004014d90 con 0x7f90100f5410 2023-12-25T20:09:50.212 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:50.210+0000 7f9016cfd700 1 -- 172.21.15.160:0/793811522 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f901010bce0 con 0x7f90100f5410 2023-12-25T20:09:50.212 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:50.211+0000 7f900d7fa700 1 -- 172.21.15.160:0/793811522 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f90040095f0 con 0x7f90100f5410 2023-12-25T20:09:50.212 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:50.211+0000 7f9016cfd700 1 -- 172.21.15.160:0/793811522 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9010045ad0 con 0x7f90100f5410 2023-12-25T20:09:50.212 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:50.211+0000 7f900d7fa700 1 --2- 172.21.15.160:0/793811522 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f900002ff60 0x7f9000032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:09:50.212 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:50.211+0000 7f900d7fa700 1 -- 172.21.15.160:0/793811522 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9004041000 con 0x7f90100f5410 2023-12-25T20:09:50.212 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:50.213+0000 7f900ffff700 1 --2- 172.21.15.160:0/793811522 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f900002ff60 0x7f9000032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:09:50.215 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:50.215+0000 7f900ffff700 1 --2- 172.21.15.160:0/793811522 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f900002ff60 0x7f9000032420 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f8ffc0096a0 tx=0x7f8ffc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:09:50.216 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:50.216+0000 7f900d7fa700 1 -- 172.21.15.160:0/793811522 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9004014420 con 0x7f90100f5410 2023-12-25T20:09:50.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:09:49 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:09:48.646077+0000 mgr.smithi050.nbzhun (mgr.14162) 78 : cluster [DBG] pgmap v71: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:09:50.420 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:50.420+0000 7f9016cfd700 1 -- 172.21.15.160:0/793811522 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f901010c4a0 con 0x7f90100f5410 2023-12-25T20:09:50.421 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:50.422+0000 7f900d7fa700 1 -- 172.21.15.160:0/793811522 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9004010230 con 0x7f90100f5410 2023-12-25T20:09:50.422 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:09:50.422 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:09:50.423 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:50.423+0000 7f9016cfd700 1 -- 172.21.15.160:0/793811522 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f900002ff60 msgr2=0x7f9000032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:09:50.423 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:50.423+0000 7f9016cfd700 1 --2- 172.21.15.160:0/793811522 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f900002ff60 0x7f9000032420 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f8ffc0096a0 tx=0x7f8ffc006b40).stop 2023-12-25T20:09:50.423 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:50.423+0000 7f9016cfd700 1 -- 172.21.15.160:0/793811522 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f90100f5410 msgr2=0x7f901010b7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:09:50.423 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:50.423+0000 7f9016cfd700 1 --2- 172.21.15.160:0/793811522 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f90100f5410 0x7f901010b7f0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f9004013510 tx=0x7f900400b040).stop 2023-12-25T20:09:50.424 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:50.423+0000 7f9016cfd700 1 -- 172.21.15.160:0/793811522 shutdown_connections 2023-12-25T20:09:50.424 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:50.423+0000 7f9016cfd700 1 --2- 172.21.15.160:0/793811522 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f900002ff60 0x7f9000032420 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:09:50.424 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:50.423+0000 7f9016cfd700 1 --2- 172.21.15.160:0/793811522 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f90100f5410 0x7f901010b7f0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:09:50.424 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:50.423+0000 7f9016cfd700 1 -- 172.21.15.160:0/793811522 >> 172.21.15.160:0/793811522 conn(0x7f90100f0b70 msgr2=0x7f90100f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:09:50.424 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:50.424+0000 7f9016cfd700 1 -- 172.21.15.160:0/793811522 shutdown_connections 2023-12-25T20:09:50.424 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:50.424+0000 7f9016cfd700 1 -- 172.21.15.160:0/793811522 wait complete. 2023-12-25T20:09:50.424 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:09:51.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:09:50 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:09:50.421811+0000 mon.smithi050 (mon.0) 165 : audit [DBG] from='client.? 172.21.15.160:0/793811522' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:09:51.858 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:09:51.858 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:09:52.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:09:51 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:09:50.646424+0000 mgr.smithi050.nbzhun (mgr.14162) 79 : cluster [DBG] pgmap v72: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:09:53.051 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:53.051+0000 7f94c61b2700 1 -- 172.21.15.160:0/4144810370 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f94c00f51a0 msgr2=0x7f94c00f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:09:53.052 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:53.051+0000 7f94c61b2700 1 --2- 172.21.15.160:0/4144810370 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f94c00f51a0 0x7f94c00f55c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f94b0009fb0 tx=0x7f94b0009b30).stop 2023-12-25T20:09:53.052 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:53.051+0000 7f94c61b2700 1 -- 172.21.15.160:0/4144810370 shutdown_connections 2023-12-25T20:09:53.052 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:53.051+0000 7f94c61b2700 1 --2- 172.21.15.160:0/4144810370 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f94c00f51a0 0x7f94c00f55c0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:09:53.052 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:53.051+0000 7f94c61b2700 1 -- 172.21.15.160:0/4144810370 >> 172.21.15.160:0/4144810370 conn(0x7f94c00f09a0 msgr2=0x7f94c00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:09:53.052 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:53.051+0000 7f94c61b2700 1 -- 172.21.15.160:0/4144810370 shutdown_connections 2023-12-25T20:09:53.052 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:53.051+0000 7f94c61b2700 1 -- 172.21.15.160:0/4144810370 wait complete. 2023-12-25T20:09:53.052 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:53.051+0000 7f94c61b2700 1 Processor -- start 2023-12-25T20:09:53.052 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:53.052+0000 7f94c61b2700 1 -- start start 2023-12-25T20:09:53.052 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:53.052+0000 7f94c61b2700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f94c00f51a0 0x7f94c01053e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:09:53.052 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:53.052+0000 7f94c61b2700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94c01058d0 con 0x7f94c00f51a0 2023-12-25T20:09:53.053 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:53.053+0000 7f94c51b0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f94c00f51a0 0x7f94c01053e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:09:53.053 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:53.053+0000 7f94c51b0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f94c00f51a0 0x7f94c01053e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:51452/0 (socket says 172.21.15.160:51452) 2023-12-25T20:09:53.053 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:53.053+0000 7f94c51b0700 1 -- 172.21.15.160:0/3481031185 learned_addr learned my addr 172.21.15.160:0/3481031185 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:09:53.054 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:53.055+0000 7f94c51b0700 1 -- 172.21.15.160:0/3481031185 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94b0005040 con 0x7f94c00f51a0 2023-12-25T20:09:53.055 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:53.055+0000 7f94c51b0700 1 --2- 172.21.15.160:0/3481031185 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f94c00f51a0 0x7f94c01053e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f94b0013510 tx=0x7f94b000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:09:53.056 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:53.056+0000 7f94b5ffb700 1 -- 172.21.15.160:0/3481031185 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f94b0009320 con 0x7f94c00f51a0 2023-12-25T20:09:53.056 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:53.056+0000 7f94b5ffb700 1 -- 172.21.15.160:0/3481031185 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f94b0013f00 con 0x7f94c00f51a0 2023-12-25T20:09:53.056 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:53.056+0000 7f94c61b2700 1 -- 172.21.15.160:0/3481031185 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94c0105ad0 con 0x7f94c00f51a0 2023-12-25T20:09:53.056 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:53.056+0000 7f94b5ffb700 1 -- 172.21.15.160:0/3481031185 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f94b0014dc0 con 0x7f94c00f51a0 2023-12-25T20:09:53.056 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:53.056+0000 7f94c61b2700 1 -- 172.21.15.160:0/3481031185 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94c0105f60 con 0x7f94c00f51a0 2023-12-25T20:09:53.057 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:53.057+0000 7f94c61b2700 1 -- 172.21.15.160:0/3481031185 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94c0045ad0 con 0x7f94c00f51a0 2023-12-25T20:09:53.057 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:53.058+0000 7f94b5ffb700 1 -- 172.21.15.160:0/3481031185 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f94b00095f0 con 0x7f94c00f51a0 2023-12-25T20:09:53.057 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:53.058+0000 7f94b5ffb700 1 --2- 172.21.15.160:0/3481031185 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f94ac030000 0x7f94ac0324c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:09:53.058 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:53.058+0000 7f94b5ffb700 1 -- 172.21.15.160:0/3481031185 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f94b00431b0 con 0x7f94c00f51a0 2023-12-25T20:09:53.058 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:53.058+0000 7f94c49af700 1 --2- 172.21.15.160:0/3481031185 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f94ac030000 0x7f94ac0324c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:09:53.060 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:53.061+0000 7f94c49af700 1 --2- 172.21.15.160:0/3481031185 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f94ac030000 0x7f94ac0324c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f94bc0096a0 tx=0x7f94bc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:09:53.062 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:53.063+0000 7f94b5ffb700 1 -- 172.21.15.160:0/3481031185 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f94b001e970 con 0x7f94c00f51a0 2023-12-25T20:09:53.268 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:53.268+0000 7f94c61b2700 1 -- 172.21.15.160:0/3481031185 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f94c0106db0 con 0x7f94c00f51a0 2023-12-25T20:09:53.269 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:53.269+0000 7f94b5ffb700 1 -- 172.21.15.160:0/3481031185 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f94b001e790 con 0x7f94c00f51a0 2023-12-25T20:09:53.269 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:09:53.269 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:09:53.271 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:53.271+0000 7f94c61b2700 1 -- 172.21.15.160:0/3481031185 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f94ac030000 msgr2=0x7f94ac0324c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:09:53.271 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:53.271+0000 7f94c61b2700 1 --2- 172.21.15.160:0/3481031185 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f94ac030000 0x7f94ac0324c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f94bc0096a0 tx=0x7f94bc006b40).stop 2023-12-25T20:09:53.271 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:53.271+0000 7f94c61b2700 1 -- 172.21.15.160:0/3481031185 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f94c00f51a0 msgr2=0x7f94c01053e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:09:53.271 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:53.271+0000 7f94c61b2700 1 --2- 172.21.15.160:0/3481031185 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f94c00f51a0 0x7f94c01053e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f94b0013510 tx=0x7f94b000b040).stop 2023-12-25T20:09:53.271 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:53.271+0000 7f94c61b2700 1 -- 172.21.15.160:0/3481031185 shutdown_connections 2023-12-25T20:09:53.271 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:53.271+0000 7f94c61b2700 1 --2- 172.21.15.160:0/3481031185 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f94ac030000 0x7f94ac0324c0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:09:53.271 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:53.271+0000 7f94c61b2700 1 --2- 172.21.15.160:0/3481031185 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f94c00f51a0 0x7f94c01053e0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:09:53.271 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:53.271+0000 7f94c61b2700 1 -- 172.21.15.160:0/3481031185 >> 172.21.15.160:0/3481031185 conn(0x7f94c00f09a0 msgr2=0x7f94c00f13e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:09:53.271 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:53.271+0000 7f94c61b2700 1 -- 172.21.15.160:0/3481031185 shutdown_connections 2023-12-25T20:09:53.271 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:53.271+0000 7f94c61b2700 1 -- 172.21.15.160:0/3481031185 wait complete. 2023-12-25T20:09:53.272 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:09:54.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:09:53 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:09:52.646727+0000 mgr.smithi050.nbzhun (mgr.14162) 80 : cluster [DBG] pgmap v73: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:09:54.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:09:53 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:09:53.269933+0000 mon.smithi050 (mon.0) 166 : audit [DBG] from='client.? 172.21.15.160:0/3481031185' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:09:54.662 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:09:54.662 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:09:55.892 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:55.892+0000 7f732c92e700 1 -- 172.21.15.160:0/2614118519 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7328060280 msgr2=0x7f73280f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:09:55.893 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:55.892+0000 7f732c92e700 1 --2- 172.21.15.160:0/2614118519 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7328060280 0x7f73280f90f0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f7310009fb0 tx=0x7f7310009b30).stop 2023-12-25T20:09:55.893 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:55.892+0000 7f732c92e700 1 -- 172.21.15.160:0/2614118519 shutdown_connections 2023-12-25T20:09:55.893 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:55.892+0000 7f732c92e700 1 --2- 172.21.15.160:0/2614118519 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7328060280 0x7f73280f90f0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:09:55.893 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:55.892+0000 7f732c92e700 1 -- 172.21.15.160:0/2614118519 >> 172.21.15.160:0/2614118519 conn(0x7f73280f0b70 msgr2=0x7f73280f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:09:55.893 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:55.892+0000 7f732c92e700 1 -- 172.21.15.160:0/2614118519 shutdown_connections 2023-12-25T20:09:55.893 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:55.892+0000 7f732c92e700 1 -- 172.21.15.160:0/2614118519 wait complete. 2023-12-25T20:09:55.893 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:55.893+0000 7f732c92e700 1 Processor -- start 2023-12-25T20:09:55.893 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:55.893+0000 7f732c92e700 1 -- start start 2023-12-25T20:09:55.895 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:55.893+0000 7f732c92e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7328060280 0x7f73281066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:09:55.895 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:55.893+0000 7f732c92e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7328106bd0 con 0x7f7328060280 2023-12-25T20:09:55.895 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:55.894+0000 7f732659c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7328060280 0x7f73281066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:09:55.895 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:55.894+0000 7f732659c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7328060280 0x7f73281066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:51466/0 (socket says 172.21.15.160:51466) 2023-12-25T20:09:55.895 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:55.894+0000 7f732659c700 1 -- 172.21.15.160:0/534605034 learned_addr learned my addr 172.21.15.160:0/534605034 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:09:55.895 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:55.894+0000 7f732659c700 1 -- 172.21.15.160:0/534605034 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7310005040 con 0x7f7328060280 2023-12-25T20:09:55.895 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:55.894+0000 7f732659c700 1 --2- 172.21.15.160:0/534605034 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7328060280 0x7f73281066e0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f7310013510 tx=0x7f731000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:09:55.895 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:55.894+0000 7f731effd700 1 -- 172.21.15.160:0/534605034 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7310009320 con 0x7f7328060280 2023-12-25T20:09:55.896 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:55.894+0000 7f732c92e700 1 -- 172.21.15.160:0/534605034 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7328106dd0 con 0x7f7328060280 2023-12-25T20:09:55.896 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:55.894+0000 7f731effd700 1 -- 172.21.15.160:0/534605034 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7310013f00 con 0x7f7328060280 2023-12-25T20:09:55.896 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:55.894+0000 7f731effd700 1 -- 172.21.15.160:0/534605034 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7310014dc0 con 0x7f7328060280 2023-12-25T20:09:55.896 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:55.894+0000 7f732c92e700 1 -- 172.21.15.160:0/534605034 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7328107260 con 0x7f7328060280 2023-12-25T20:09:55.896 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:55.895+0000 7f731effd700 1 -- 172.21.15.160:0/534605034 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f731001d3d0 con 0x7f7328060280 2023-12-25T20:09:55.896 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:55.895+0000 7f731effd700 1 --2- 172.21.15.160:0/534605034 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f731402ff60 0x7f7314032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:09:55.896 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:55.895+0000 7f731effd700 1 -- 172.21.15.160:0/534605034 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7310041f60 con 0x7f7328060280 2023-12-25T20:09:55.896 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:55.895+0000 7f732c92e700 1 -- 172.21.15.160:0/534605034 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7328059100 con 0x7f7328060280 2023-12-25T20:09:55.896 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:55.895+0000 7f7325d9b700 1 --2- 172.21.15.160:0/534605034 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f731402ff60 0x7f7314032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:09:55.896 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:55.896+0000 7f7325d9b700 1 --2- 172.21.15.160:0/534605034 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f731402ff60 0x7f7314032420 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f73180096a0 tx=0x7f7318006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:09:55.900 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:55.900+0000 7f731effd700 1 -- 172.21.15.160:0/534605034 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f73100682b0 con 0x7f7328060280 2023-12-25T20:09:56.106 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:56.106+0000 7f732c92e700 1 -- 172.21.15.160:0/534605034 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7328045ad0 con 0x7f7328060280 2023-12-25T20:09:56.106 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:56.107+0000 7f731effd700 1 -- 172.21.15.160:0/534605034 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f731002b020 con 0x7f7328060280 2023-12-25T20:09:56.107 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:09:56.107 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:09:56.108 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:56.108+0000 7f732c92e700 1 -- 172.21.15.160:0/534605034 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f731402ff60 msgr2=0x7f7314032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:09:56.108 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:56.108+0000 7f732c92e700 1 --2- 172.21.15.160:0/534605034 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f731402ff60 0x7f7314032420 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f73180096a0 tx=0x7f7318006b40).stop 2023-12-25T20:09:56.108 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:56.108+0000 7f732c92e700 1 -- 172.21.15.160:0/534605034 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7328060280 msgr2=0x7f73281066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:09:56.108 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:56.108+0000 7f732c92e700 1 --2- 172.21.15.160:0/534605034 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7328060280 0x7f73281066e0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f7310013510 tx=0x7f731000b040).stop 2023-12-25T20:09:56.109 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:56.109+0000 7f732c92e700 1 -- 172.21.15.160:0/534605034 shutdown_connections 2023-12-25T20:09:56.109 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:56.109+0000 7f732c92e700 1 --2- 172.21.15.160:0/534605034 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f731402ff60 0x7f7314032420 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:09:56.109 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:56.109+0000 7f732c92e700 1 --2- 172.21.15.160:0/534605034 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7328060280 0x7f73281066e0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:09:56.109 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:56.109+0000 7f732c92e700 1 -- 172.21.15.160:0/534605034 >> 172.21.15.160:0/534605034 conn(0x7f73280f0b70 msgr2=0x7f73280f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:09:56.109 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:56.109+0000 7f732c92e700 1 -- 172.21.15.160:0/534605034 shutdown_connections 2023-12-25T20:09:56.109 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:56.109+0000 7f732c92e700 1 -- 172.21.15.160:0/534605034 wait complete. 2023-12-25T20:09:56.109 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:09:56.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:09:55 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:09:54.646935+0000 mgr.smithi050.nbzhun (mgr.14162) 81 : cluster [DBG] pgmap v74: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:09:57.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:09:56 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:09:56.107365+0000 mon.smithi050 (mon.0) 167 : audit [DBG] from='client.? 172.21.15.160:0/534605034' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:09:57.537 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:09:57.538 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:09:58.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:09:57 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:09:56.647262+0000 mgr.smithi050.nbzhun (mgr.14162) 82 : cluster [DBG] pgmap v75: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:09:58.763 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:58.761+0000 7f8ab3f16700 1 -- 172.21.15.160:0/2451896458 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8aac0f6510 msgr2=0x7f8aac0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:09:58.763 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:58.761+0000 7f8ab3f16700 1 --2- 172.21.15.160:0/2451896458 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8aac0f6510 0x7f8aac0f6930 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f8aa8009fb0 tx=0x7f8aa8009b30).stop 2023-12-25T20:09:58.763 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:58.761+0000 7f8ab3f16700 1 -- 172.21.15.160:0/2451896458 shutdown_connections 2023-12-25T20:09:58.763 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:58.761+0000 7f8ab3f16700 1 --2- 172.21.15.160:0/2451896458 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8aac0f6510 0x7f8aac0f6930 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:09:58.763 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:58.761+0000 7f8ab3f16700 1 -- 172.21.15.160:0/2451896458 >> 172.21.15.160:0/2451896458 conn(0x7f8aac0f1c70 msgr2=0x7f8aac0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:09:58.763 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:58.762+0000 7f8ab3f16700 1 -- 172.21.15.160:0/2451896458 shutdown_connections 2023-12-25T20:09:58.763 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:58.762+0000 7f8ab3f16700 1 -- 172.21.15.160:0/2451896458 wait complete. 2023-12-25T20:09:58.763 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:58.762+0000 7f8ab3f16700 1 Processor -- start 2023-12-25T20:09:58.763 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:58.762+0000 7f8ab3f16700 1 -- start start 2023-12-25T20:09:58.764 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:58.762+0000 7f8ab3f16700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8aac0f6510 0x7f8aac106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:09:58.764 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:58.762+0000 7f8ab3f16700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8aac106c20 con 0x7f8aac0f6510 2023-12-25T20:09:58.764 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:58.762+0000 7f8ab1cb2700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8aac0f6510 0x7f8aac106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:09:58.764 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:58.762+0000 7f8ab1cb2700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8aac0f6510 0x7f8aac106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:40564/0 (socket says 172.21.15.160:40564) 2023-12-25T20:09:58.764 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:58.762+0000 7f8ab1cb2700 1 -- 172.21.15.160:0/1770154045 learned_addr learned my addr 172.21.15.160:0/1770154045 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:09:58.764 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:58.763+0000 7f8ab1cb2700 1 -- 172.21.15.160:0/1770154045 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8aa8005040 con 0x7f8aac0f6510 2023-12-25T20:09:58.764 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:58.763+0000 7f8ab1cb2700 1 --2- 172.21.15.160:0/1770154045 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8aac0f6510 0x7f8aac106730 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f8aa8013510 tx=0x7f8aa800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:09:58.764 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:58.763+0000 7f8aa27fc700 1 -- 172.21.15.160:0/1770154045 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8aa8009320 con 0x7f8aac0f6510 2023-12-25T20:09:58.764 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:58.763+0000 7f8ab3f16700 1 -- 172.21.15.160:0/1770154045 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8aac106e20 con 0x7f8aac0f6510 2023-12-25T20:09:58.764 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:58.763+0000 7f8aa27fc700 1 -- 172.21.15.160:0/1770154045 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8aa8013f00 con 0x7f8aac0f6510 2023-12-25T20:09:58.764 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:58.763+0000 7f8aa27fc700 1 -- 172.21.15.160:0/1770154045 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8aa8014d90 con 0x7f8aac0f6510 2023-12-25T20:09:58.764 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:58.763+0000 7f8ab3f16700 1 -- 172.21.15.160:0/1770154045 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8aac1072b0 con 0x7f8aac0f6510 2023-12-25T20:09:58.765 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:58.764+0000 7f8aa27fc700 1 -- 172.21.15.160:0/1770154045 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8aa80095f0 con 0x7f8aac0f6510 2023-12-25T20:09:58.765 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:58.764+0000 7f8ab3f16700 1 -- 172.21.15.160:0/1770154045 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8aac045ad0 con 0x7f8aac0f6510 2023-12-25T20:09:58.765 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:58.764+0000 7f8aa27fc700 1 --2- 172.21.15.160:0/1770154045 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f8a9802ff60 0x7f8a98032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:09:58.765 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:58.764+0000 7f8aa27fc700 1 -- 172.21.15.160:0/1770154045 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8aa8041000 con 0x7f8aac0f6510 2023-12-25T20:09:58.766 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:58.764+0000 7f8ab14b1700 1 --2- 172.21.15.160:0/1770154045 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f8a9802ff60 0x7f8a98032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:09:58.766 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:58.765+0000 7f8ab14b1700 1 --2- 172.21.15.160:0/1770154045 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f8a9802ff60 0x7f8a98032420 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f8a9c0096a0 tx=0x7f8a9c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:09:58.769 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:58.769+0000 7f8aa27fc700 1 -- 172.21.15.160:0/1770154045 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8aa8014420 con 0x7f8aac0f6510 2023-12-25T20:09:58.971 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:58.972+0000 7f8ab3f16700 1 -- 172.21.15.160:0/1770154045 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8aac0fbf60 con 0x7f8aac0f6510 2023-12-25T20:09:58.972 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:58.972+0000 7f8aa27fc700 1 -- 172.21.15.160:0/1770154045 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8aa801d7f0 con 0x7f8aac0f6510 2023-12-25T20:09:58.972 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:09:58.972 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:09:58.974 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:58.974+0000 7f8ab3f16700 1 -- 172.21.15.160:0/1770154045 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f8a9802ff60 msgr2=0x7f8a98032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:09:58.974 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:58.974+0000 7f8ab3f16700 1 --2- 172.21.15.160:0/1770154045 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f8a9802ff60 0x7f8a98032420 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f8a9c0096a0 tx=0x7f8a9c006b40).stop 2023-12-25T20:09:58.974 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:58.974+0000 7f8ab3f16700 1 -- 172.21.15.160:0/1770154045 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8aac0f6510 msgr2=0x7f8aac106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:09:58.974 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:58.974+0000 7f8ab3f16700 1 --2- 172.21.15.160:0/1770154045 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8aac0f6510 0x7f8aac106730 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f8aa8013510 tx=0x7f8aa800b040).stop 2023-12-25T20:09:58.974 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:58.974+0000 7f8ab3f16700 1 -- 172.21.15.160:0/1770154045 shutdown_connections 2023-12-25T20:09:58.974 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:58.974+0000 7f8ab3f16700 1 --2- 172.21.15.160:0/1770154045 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f8a9802ff60 0x7f8a98032420 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:09:58.974 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:58.974+0000 7f8ab3f16700 1 --2- 172.21.15.160:0/1770154045 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8aac0f6510 0x7f8aac106730 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:09:58.974 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:58.974+0000 7f8ab3f16700 1 -- 172.21.15.160:0/1770154045 >> 172.21.15.160:0/1770154045 conn(0x7f8aac0f1c70 msgr2=0x7f8aac0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:09:58.974 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:58.974+0000 7f8ab3f16700 1 -- 172.21.15.160:0/1770154045 shutdown_connections 2023-12-25T20:09:58.974 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:09:58.974+0000 7f8ab3f16700 1 -- 172.21.15.160:0/1770154045 wait complete. 2023-12-25T20:09:58.975 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:10:00.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:09:59 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:09:58.647551+0000 mgr.smithi050.nbzhun (mgr.14162) 83 : cluster [DBG] pgmap v76: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:10:00.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:09:59 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:09:58.973023+0000 mon.smithi050 (mon.0) 168 : audit [DBG] from='client.? 172.21.15.160:0/1770154045' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:10:00.395 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:10:00.395 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:10:01.595 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:01.594+0000 7f65ad34f700 1 -- 172.21.15.160:0/832930874 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f65a80f5410 msgr2=0x7f65a80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:10:01.595 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:01.594+0000 7f65ad34f700 1 --2- 172.21.15.160:0/832930874 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f65a80f5410 0x7f65a80f5830 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f659c009fb0 tx=0x7f659c009b30).stop 2023-12-25T20:10:01.595 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:01.594+0000 7f65ad34f700 1 -- 172.21.15.160:0/832930874 shutdown_connections 2023-12-25T20:10:01.595 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:01.594+0000 7f65ad34f700 1 --2- 172.21.15.160:0/832930874 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f65a80f5410 0x7f65a80f5830 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:10:01.595 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:01.594+0000 7f65ad34f700 1 -- 172.21.15.160:0/832930874 >> 172.21.15.160:0/832930874 conn(0x7f65a80f0b70 msgr2=0x7f65a80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:10:01.595 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:01.594+0000 7f65ad34f700 1 -- 172.21.15.160:0/832930874 shutdown_connections 2023-12-25T20:10:01.595 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:01.594+0000 7f65ad34f700 1 -- 172.21.15.160:0/832930874 wait complete. 2023-12-25T20:10:01.596 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:01.595+0000 7f65ad34f700 1 Processor -- start 2023-12-25T20:10:01.596 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:01.595+0000 7f65ad34f700 1 -- start start 2023-12-25T20:10:01.596 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:01.595+0000 7f65ad34f700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f65a80f5410 0x7f65a81066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:10:01.596 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:01.595+0000 7f65ad34f700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65a8106bc0 con 0x7f65a80f5410 2023-12-25T20:10:01.597 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:01.596+0000 7f65a6ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f65a80f5410 0x7f65a81066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:10:01.598 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:01.596+0000 7f65a6ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f65a80f5410 0x7f65a81066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:40582/0 (socket says 172.21.15.160:40582) 2023-12-25T20:10:01.598 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:01.596+0000 7f65a6ffd700 1 -- 172.21.15.160:0/3563438392 learned_addr learned my addr 172.21.15.160:0/3563438392 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:10:01.598 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:01.596+0000 7f65a6ffd700 1 -- 172.21.15.160:0/3563438392 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f659c005040 con 0x7f65a80f5410 2023-12-25T20:10:01.598 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:01.596+0000 7f65a6ffd700 1 --2- 172.21.15.160:0/3563438392 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f65a80f5410 0x7f65a81066d0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f659c013510 tx=0x7f659c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:10:01.598 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:01.596+0000 7f658f7fe700 1 -- 172.21.15.160:0/3563438392 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f659c009320 con 0x7f65a80f5410 2023-12-25T20:10:01.598 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:01.596+0000 7f658f7fe700 1 -- 172.21.15.160:0/3563438392 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f659c013f00 con 0x7f65a80f5410 2023-12-25T20:10:01.598 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:01.596+0000 7f65ad34f700 1 -- 172.21.15.160:0/3563438392 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65a8106dc0 con 0x7f65a80f5410 2023-12-25T20:10:01.598 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:01.596+0000 7f658f7fe700 1 -- 172.21.15.160:0/3563438392 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f659c014d90 con 0x7f65a80f5410 2023-12-25T20:10:01.598 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:01.596+0000 7f65ad34f700 1 -- 172.21.15.160:0/3563438392 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65a8107250 con 0x7f65a80f5410 2023-12-25T20:10:01.598 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:01.597+0000 7f658f7fe700 1 -- 172.21.15.160:0/3563438392 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f659c0263a0 con 0x7f65a80f5410 2023-12-25T20:10:01.598 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:01.597+0000 7f658f7fe700 1 --2- 172.21.15.160:0/3563438392 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f659002ffb0 0x7f6590032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:10:01.598 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:01.597+0000 7f65ad34f700 1 -- 172.21.15.160:0/3563438392 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65a8045ad0 con 0x7f65a80f5410 2023-12-25T20:10:01.598 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:01.597+0000 7f658f7fe700 1 -- 172.21.15.160:0/3563438392 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f659c040fe0 con 0x7f65a80f5410 2023-12-25T20:10:01.598 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:01.598+0000 7f65a67fc700 1 --2- 172.21.15.160:0/3563438392 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f659002ffb0 0x7f6590032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:10:01.599 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:01.598+0000 7f65a67fc700 1 --2- 172.21.15.160:0/3563438392 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f659002ffb0 0x7f6590032470 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f65980096a0 tx=0x7f6598006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:10:01.602 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:01.602+0000 7f658f7fe700 1 -- 172.21.15.160:0/3563438392 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f659c012920 con 0x7f65a80f5410 2023-12-25T20:10:01.807 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:01.807+0000 7f65ad34f700 1 -- 172.21.15.160:0/3563438392 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f65a8108100 con 0x7f65a80f5410 2023-12-25T20:10:01.807 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:01.807+0000 7f658f7fe700 1 -- 172.21.15.160:0/3563438392 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f659c040070 con 0x7f65a80f5410 2023-12-25T20:10:01.807 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:10:01.808 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:10:01.809 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:01.809+0000 7f65ad34f700 1 -- 172.21.15.160:0/3563438392 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f659002ffb0 msgr2=0x7f6590032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:10:01.809 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:01.809+0000 7f65ad34f700 1 --2- 172.21.15.160:0/3563438392 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f659002ffb0 0x7f6590032470 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f65980096a0 tx=0x7f6598006b40).stop 2023-12-25T20:10:01.809 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:01.809+0000 7f65ad34f700 1 -- 172.21.15.160:0/3563438392 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f65a80f5410 msgr2=0x7f65a81066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:10:01.809 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:01.809+0000 7f65ad34f700 1 --2- 172.21.15.160:0/3563438392 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f65a80f5410 0x7f65a81066d0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f659c013510 tx=0x7f659c00b040).stop 2023-12-25T20:10:01.809 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:01.809+0000 7f65ad34f700 1 -- 172.21.15.160:0/3563438392 shutdown_connections 2023-12-25T20:10:01.809 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:01.809+0000 7f65ad34f700 1 --2- 172.21.15.160:0/3563438392 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f659002ffb0 0x7f6590032470 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:10:01.809 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:01.809+0000 7f65ad34f700 1 --2- 172.21.15.160:0/3563438392 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f65a80f5410 0x7f65a81066d0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:10:01.809 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:01.809+0000 7f65ad34f700 1 -- 172.21.15.160:0/3563438392 >> 172.21.15.160:0/3563438392 conn(0x7f65a80f0b70 msgr2=0x7f65a805fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:10:01.809 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:01.809+0000 7f65ad34f700 1 -- 172.21.15.160:0/3563438392 shutdown_connections 2023-12-25T20:10:01.810 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:01.809+0000 7f65ad34f700 1 -- 172.21.15.160:0/3563438392 wait complete. 2023-12-25T20:10:01.810 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:10:02.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:10:01 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:10:00.647837+0000 mgr.smithi050.nbzhun (mgr.14162) 84 : cluster [DBG] pgmap v77: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:10:02.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:10:01 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:10:01.808188+0000 mon.smithi050 (mon.0) 169 : audit [DBG] from='client.? 172.21.15.160:0/3563438392' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:10:03.249 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:10:03.249 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:10:04.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:10:03 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:10:02.648205+0000 mgr.smithi050.nbzhun (mgr.14162) 85 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:10:04.474 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:04.474+0000 7f4e001a5700 1 -- 172.21.15.160:0/1172495624 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4df80f5410 msgr2=0x7f4df80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:10:04.474 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:04.474+0000 7f4e001a5700 1 --2- 172.21.15.160:0/1172495624 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4df80f5410 0x7f4df80f5830 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f4de8009fb0 tx=0x7f4de8009b30).stop 2023-12-25T20:10:04.474 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:04.474+0000 7f4e001a5700 1 -- 172.21.15.160:0/1172495624 shutdown_connections 2023-12-25T20:10:04.474 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:04.474+0000 7f4e001a5700 1 --2- 172.21.15.160:0/1172495624 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4df80f5410 0x7f4df80f5830 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:10:04.475 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:04.474+0000 7f4e001a5700 1 -- 172.21.15.160:0/1172495624 >> 172.21.15.160:0/1172495624 conn(0x7f4df80f0b70 msgr2=0x7f4df80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:10:04.475 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:04.474+0000 7f4e001a5700 1 -- 172.21.15.160:0/1172495624 shutdown_connections 2023-12-25T20:10:04.475 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:04.474+0000 7f4e001a5700 1 -- 172.21.15.160:0/1172495624 wait complete. 2023-12-25T20:10:04.475 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:04.474+0000 7f4e001a5700 1 Processor -- start 2023-12-25T20:10:04.475 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:04.474+0000 7f4e001a5700 1 -- start start 2023-12-25T20:10:04.475 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:04.475+0000 7f4e001a5700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4df80f5410 0x7f4df81066b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:10:04.475 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:04.475+0000 7f4e001a5700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4df8106ba0 con 0x7f4df80f5410 2023-12-25T20:10:04.477 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:04.475+0000 7f4dfdf41700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4df80f5410 0x7f4df81066b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:10:04.477 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:04.475+0000 7f4dfdf41700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4df80f5410 0x7f4df81066b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:40596/0 (socket says 172.21.15.160:40596) 2023-12-25T20:10:04.477 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:04.475+0000 7f4dfdf41700 1 -- 172.21.15.160:0/453451057 learned_addr learned my addr 172.21.15.160:0/453451057 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:10:04.477 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:04.475+0000 7f4dfdf41700 1 -- 172.21.15.160:0/453451057 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4de8005040 con 0x7f4df80f5410 2023-12-25T20:10:04.477 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:04.475+0000 7f4dfdf41700 1 --2- 172.21.15.160:0/453451057 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4df80f5410 0x7f4df81066b0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f4de8014000 tx=0x7f4de800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:10:04.477 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:04.476+0000 7f4dee7fc700 1 -- 172.21.15.160:0/453451057 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4de80095f0 con 0x7f4df80f5410 2023-12-25T20:10:04.477 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:04.476+0000 7f4dee7fc700 1 -- 172.21.15.160:0/453451057 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4de8007e70 con 0x7f4df80f5410 2023-12-25T20:10:04.477 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:04.476+0000 7f4e001a5700 1 -- 172.21.15.160:0/453451057 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4df8106da0 con 0x7f4df80f5410 2023-12-25T20:10:04.477 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:04.476+0000 7f4dee7fc700 1 -- 172.21.15.160:0/453451057 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4de8014ea0 con 0x7f4df80f5410 2023-12-25T20:10:04.477 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:04.476+0000 7f4e001a5700 1 -- 172.21.15.160:0/453451057 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4df8107230 con 0x7f4df80f5410 2023-12-25T20:10:04.477 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:04.476+0000 7f4dee7fc700 1 -- 172.21.15.160:0/453451057 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4de8025070 con 0x7f4df80f5410 2023-12-25T20:10:04.477 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:04.476+0000 7f4e001a5700 1 -- 172.21.15.160:0/453451057 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4df8045ad0 con 0x7f4df80f5410 2023-12-25T20:10:04.478 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:04.477+0000 7f4dee7fc700 1 --2- 172.21.15.160:0/453451057 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f4de402ffb0 0x7f4de4032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:10:04.478 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:04.477+0000 7f4dee7fc700 1 -- 172.21.15.160:0/453451057 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4de8040900 con 0x7f4df80f5410 2023-12-25T20:10:04.478 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:04.477+0000 7f4dfd740700 1 --2- 172.21.15.160:0/453451057 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f4de402ffb0 0x7f4de4032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:10:04.478 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:04.477+0000 7f4dfd740700 1 --2- 172.21.15.160:0/453451057 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f4de402ffb0 0x7f4de4032470 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f4df40096a0 tx=0x7f4df4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:10:04.481 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:04.482+0000 7f4dee7fc700 1 -- 172.21.15.160:0/453451057 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4de80079c0 con 0x7f4df80f5410 2023-12-25T20:10:04.686 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:04.686+0000 7f4e001a5700 1 -- 172.21.15.160:0/453451057 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4df81080e0 con 0x7f4df80f5410 2023-12-25T20:10:04.687 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:04.688+0000 7f4dee7fc700 1 -- 172.21.15.160:0/453451057 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4de800f300 con 0x7f4df80f5410 2023-12-25T20:10:04.688 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:10:04.688 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:10:04.689 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:04.689+0000 7f4e001a5700 1 -- 172.21.15.160:0/453451057 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f4de402ffb0 msgr2=0x7f4de4032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:10:04.689 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:04.689+0000 7f4e001a5700 1 --2- 172.21.15.160:0/453451057 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f4de402ffb0 0x7f4de4032470 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f4df40096a0 tx=0x7f4df4006b40).stop 2023-12-25T20:10:04.689 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:04.689+0000 7f4e001a5700 1 -- 172.21.15.160:0/453451057 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4df80f5410 msgr2=0x7f4df81066b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:10:04.689 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:04.689+0000 7f4e001a5700 1 --2- 172.21.15.160:0/453451057 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4df80f5410 0x7f4df81066b0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f4de8014000 tx=0x7f4de800b040).stop 2023-12-25T20:10:04.690 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:04.689+0000 7f4e001a5700 1 -- 172.21.15.160:0/453451057 shutdown_connections 2023-12-25T20:10:04.690 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:04.689+0000 7f4e001a5700 1 --2- 172.21.15.160:0/453451057 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f4de402ffb0 0x7f4de4032470 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:10:04.690 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:04.689+0000 7f4e001a5700 1 --2- 172.21.15.160:0/453451057 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4df80f5410 0x7f4df81066b0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:10:04.690 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:04.689+0000 7f4e001a5700 1 -- 172.21.15.160:0/453451057 >> 172.21.15.160:0/453451057 conn(0x7f4df80f0b70 msgr2=0x7f4df80f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:10:04.690 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:04.689+0000 7f4e001a5700 1 -- 172.21.15.160:0/453451057 shutdown_connections 2023-12-25T20:10:04.755 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:04.689+0000 7f4e001a5700 1 -- 172.21.15.160:0/453451057 wait complete. 2023-12-25T20:10:04.755 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:10:05.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:10:04 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:10:04.687543+0000 mon.smithi050 (mon.0) 170 : audit [DBG] from='client.? 172.21.15.160:0/453451057' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:10:06.121 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:10:06.121 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:10:06.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:10:05 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:10:04.648452+0000 mgr.smithi050.nbzhun (mgr.14162) 86 : cluster [DBG] pgmap v79: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:10:07.344 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:07.343+0000 7f5cbbc31700 1 -- 172.21.15.160:0/2244378499 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5cb40f5410 msgr2=0x7f5cb40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:10:07.344 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:07.343+0000 7f5cbbc31700 1 --2- 172.21.15.160:0/2244378499 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5cb40f5410 0x7f5cb40f5830 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f5ca4009fb0 tx=0x7f5ca4009b30).stop 2023-12-25T20:10:07.344 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:07.343+0000 7f5cbbc31700 1 -- 172.21.15.160:0/2244378499 shutdown_connections 2023-12-25T20:10:07.344 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:07.343+0000 7f5cbbc31700 1 --2- 172.21.15.160:0/2244378499 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5cb40f5410 0x7f5cb40f5830 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:10:07.344 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:07.343+0000 7f5cbbc31700 1 -- 172.21.15.160:0/2244378499 >> 172.21.15.160:0/2244378499 conn(0x7f5cb40f0b70 msgr2=0x7f5cb40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:10:07.344 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:07.343+0000 7f5cbbc31700 1 -- 172.21.15.160:0/2244378499 shutdown_connections 2023-12-25T20:10:07.345 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:07.343+0000 7f5cbbc31700 1 -- 172.21.15.160:0/2244378499 wait complete. 2023-12-25T20:10:07.345 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:07.344+0000 7f5cbbc31700 1 Processor -- start 2023-12-25T20:10:07.345 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:07.344+0000 7f5cbbc31700 1 -- start start 2023-12-25T20:10:07.345 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:07.344+0000 7f5cbbc31700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5cb40f5410 0x7f5cb40fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:10:07.345 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:07.344+0000 7f5cbbc31700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cb40fb820 con 0x7f5cb40f5410 2023-12-25T20:10:07.345 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:07.344+0000 7f5cb99cd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5cb40f5410 0x7f5cb40fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:10:07.345 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:07.344+0000 7f5cb99cd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5cb40f5410 0x7f5cb40fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:45218/0 (socket says 172.21.15.160:45218) 2023-12-25T20:10:07.345 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:07.344+0000 7f5cb99cd700 1 -- 172.21.15.160:0/3431000470 learned_addr learned my addr 172.21.15.160:0/3431000470 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:10:07.347 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:07.345+0000 7f5cb99cd700 1 -- 172.21.15.160:0/3431000470 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ca4005040 con 0x7f5cb40f5410 2023-12-25T20:10:07.347 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:07.345+0000 7f5cb99cd700 1 --2- 172.21.15.160:0/3431000470 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5cb40f5410 0x7f5cb40fd3a0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f5ca4014000 tx=0x7f5ca400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:10:07.347 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:07.345+0000 7f5caa7fc700 1 -- 172.21.15.160:0/3431000470 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5ca40095f0 con 0x7f5cb40f5410 2023-12-25T20:10:07.347 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:07.345+0000 7f5cbbc31700 1 -- 172.21.15.160:0/3431000470 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cb40fb9c0 con 0x7f5cb40f5410 2023-12-25T20:10:07.347 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:07.345+0000 7f5caa7fc700 1 -- 172.21.15.160:0/3431000470 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5ca4007e70 con 0x7f5cb40f5410 2023-12-25T20:10:07.347 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:07.345+0000 7f5caa7fc700 1 -- 172.21.15.160:0/3431000470 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5ca4014ea0 con 0x7f5cb40f5410 2023-12-25T20:10:07.347 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:07.345+0000 7f5cbbc31700 1 -- 172.21.15.160:0/3431000470 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cb40fbe50 con 0x7f5cb40f5410 2023-12-25T20:10:07.347 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:07.346+0000 7f5caa7fc700 1 -- 172.21.15.160:0/3431000470 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5ca4025070 con 0x7f5cb40f5410 2023-12-25T20:10:07.347 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:07.346+0000 7f5cbbc31700 1 -- 172.21.15.160:0/3431000470 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5cb4059100 con 0x7f5cb40f5410 2023-12-25T20:10:07.347 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:07.346+0000 7f5caa7fc700 1 --2- 172.21.15.160:0/3431000470 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f5ca002ffb0 0x7f5ca0032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:10:07.348 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:07.346+0000 7f5caa7fc700 1 -- 172.21.15.160:0/3431000470 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5ca4040930 con 0x7f5cb40f5410 2023-12-25T20:10:07.348 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:07.346+0000 7f5cb91cc700 1 --2- 172.21.15.160:0/3431000470 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f5ca002ffb0 0x7f5ca0032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:10:07.348 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:07.347+0000 7f5cb91cc700 1 --2- 172.21.15.160:0/3431000470 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f5ca002ffb0 0x7f5ca0032470 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f5cb00096a0 tx=0x7f5cb0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:10:07.350 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:07.351+0000 7f5caa7fc700 1 -- 172.21.15.160:0/3431000470 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5ca40079c0 con 0x7f5cb40f5410 2023-12-25T20:10:07.552 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:07.552+0000 7f5cbbc31700 1 -- 172.21.15.160:0/3431000470 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5cb4045ad0 con 0x7f5cb40f5410 2023-12-25T20:10:07.552 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:07.552+0000 7f5caa7fc700 1 -- 172.21.15.160:0/3431000470 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5ca400f300 con 0x7f5cb40f5410 2023-12-25T20:10:07.552 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:10:07.553 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:10:07.554 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:07.554+0000 7f5cbbc31700 1 -- 172.21.15.160:0/3431000470 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f5ca002ffb0 msgr2=0x7f5ca0032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:10:07.554 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:07.554+0000 7f5cbbc31700 1 --2- 172.21.15.160:0/3431000470 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f5ca002ffb0 0x7f5ca0032470 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f5cb00096a0 tx=0x7f5cb0006b40).stop 2023-12-25T20:10:07.554 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:07.554+0000 7f5cbbc31700 1 -- 172.21.15.160:0/3431000470 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5cb40f5410 msgr2=0x7f5cb40fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:10:07.554 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:07.554+0000 7f5cbbc31700 1 --2- 172.21.15.160:0/3431000470 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5cb40f5410 0x7f5cb40fd3a0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f5ca4014000 tx=0x7f5ca400b040).stop 2023-12-25T20:10:07.554 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:07.554+0000 7f5cbbc31700 1 -- 172.21.15.160:0/3431000470 shutdown_connections 2023-12-25T20:10:07.554 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:07.554+0000 7f5cbbc31700 1 --2- 172.21.15.160:0/3431000470 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f5ca002ffb0 0x7f5ca0032470 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:10:07.554 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:07.554+0000 7f5cbbc31700 1 --2- 172.21.15.160:0/3431000470 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5cb40f5410 0x7f5cb40fd3a0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:10:07.554 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:07.554+0000 7f5cbbc31700 1 -- 172.21.15.160:0/3431000470 >> 172.21.15.160:0/3431000470 conn(0x7f5cb40f0b70 msgr2=0x7f5cb405fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:10:07.554 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:07.554+0000 7f5cbbc31700 1 -- 172.21.15.160:0/3431000470 shutdown_connections 2023-12-25T20:10:07.554 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:07.554+0000 7f5cbbc31700 1 -- 172.21.15.160:0/3431000470 wait complete. 2023-12-25T20:10:07.555 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:10:08.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:10:07 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:10:06.648676+0000 mgr.smithi050.nbzhun (mgr.14162) 87 : cluster [DBG] pgmap v80: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:10:08.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:10:07 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:10:07.553084+0000 mon.smithi050 (mon.0) 171 : audit [DBG] from='client.? 172.21.15.160:0/3431000470' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:10:08.988 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:10:08.988 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:10:10.175 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:10.175+0000 7f9abcc68700 1 -- 172.21.15.160:0/551093023 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9ab80f6510 msgr2=0x7f9ab80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:10:10.175 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:10.175+0000 7f9abcc68700 1 --2- 172.21.15.160:0/551093023 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9ab80f6510 0x7f9ab80f6930 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f9aac009fb0 tx=0x7f9aac009b30).stop 2023-12-25T20:10:10.176 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:10.175+0000 7f9abcc68700 1 -- 172.21.15.160:0/551093023 shutdown_connections 2023-12-25T20:10:10.176 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:10.175+0000 7f9abcc68700 1 --2- 172.21.15.160:0/551093023 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9ab80f6510 0x7f9ab80f6930 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:10:10.176 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:10.175+0000 7f9abcc68700 1 -- 172.21.15.160:0/551093023 >> 172.21.15.160:0/551093023 conn(0x7f9ab80f1c70 msgr2=0x7f9ab80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:10:10.176 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:10.175+0000 7f9abcc68700 1 -- 172.21.15.160:0/551093023 shutdown_connections 2023-12-25T20:10:10.176 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:10.175+0000 7f9abcc68700 1 -- 172.21.15.160:0/551093023 wait complete. 2023-12-25T20:10:10.176 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:10.175+0000 7f9abcc68700 1 Processor -- start 2023-12-25T20:10:10.176 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:10.175+0000 7f9abcc68700 1 -- start start 2023-12-25T20:10:10.176 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:10.175+0000 7f9abcc68700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9ab80f6510 0x7f9ab8106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:10:10.176 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:10.175+0000 7f9abcc68700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ab8106c20 con 0x7f9ab80f6510 2023-12-25T20:10:10.177 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:10.176+0000 7f9ab659c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9ab80f6510 0x7f9ab8106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:10:10.177 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:10.176+0000 7f9ab659c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9ab80f6510 0x7f9ab8106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:45228/0 (socket says 172.21.15.160:45228) 2023-12-25T20:10:10.177 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:10.176+0000 7f9ab659c700 1 -- 172.21.15.160:0/1907102488 learned_addr learned my addr 172.21.15.160:0/1907102488 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:10:10.177 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:10.177+0000 7f9ab659c700 1 -- 172.21.15.160:0/1907102488 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9aac005040 con 0x7f9ab80f6510 2023-12-25T20:10:10.177 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:10.177+0000 7f9ab659c700 1 --2- 172.21.15.160:0/1907102488 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9ab80f6510 0x7f9ab8106730 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f9aac014000 tx=0x7f9aac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:10:10.177 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:10.177+0000 7f9aa6ffd700 1 -- 172.21.15.160:0/1907102488 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9aac0095f0 con 0x7f9ab80f6510 2023-12-25T20:10:10.178 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:10.177+0000 7f9abcc68700 1 -- 172.21.15.160:0/1907102488 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ab8106e20 con 0x7f9ab80f6510 2023-12-25T20:10:10.178 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:10.178+0000 7f9abcc68700 1 -- 172.21.15.160:0/1907102488 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ab81072b0 con 0x7f9ab80f6510 2023-12-25T20:10:10.178 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:10.178+0000 7f9aa6ffd700 1 -- 172.21.15.160:0/1907102488 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9aac007e70 con 0x7f9ab80f6510 2023-12-25T20:10:10.178 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:10.178+0000 7f9aa6ffd700 1 -- 172.21.15.160:0/1907102488 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9aac014e50 con 0x7f9ab80f6510 2023-12-25T20:10:10.179 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:10.178+0000 7f9abcc68700 1 -- 172.21.15.160:0/1907102488 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a9c004fa0 con 0x7f9ab80f6510 2023-12-25T20:10:10.179 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:10.178+0000 7f9aa6ffd700 1 -- 172.21.15.160:0/1907102488 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9aac025070 con 0x7f9ab80f6510 2023-12-25T20:10:10.179 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:10.179+0000 7f9aa6ffd700 1 --2- 172.21.15.160:0/1907102488 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f9aa002ff10 0x7f9aa00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:10:10.179 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:10.179+0000 7f9aa6ffd700 1 -- 172.21.15.160:0/1907102488 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9aac040a30 con 0x7f9ab80f6510 2023-12-25T20:10:10.179 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:10.179+0000 7f9ab5d9b700 1 --2- 172.21.15.160:0/1907102488 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f9aa002ff10 0x7f9aa00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:10:10.179 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:10.179+0000 7f9ab5d9b700 1 --2- 172.21.15.160:0/1907102488 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f9aa002ff10 0x7f9aa00323d0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f9aa80096a0 tx=0x7f9aa8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:10:10.183 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:10.183+0000 7f9aa6ffd700 1 -- 172.21.15.160:0/1907102488 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9aac0079c0 con 0x7f9ab80f6510 2023-12-25T20:10:10.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:10:09 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:10:08.648971+0000 mgr.smithi050.nbzhun (mgr.14162) 88 : cluster [DBG] pgmap v81: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:10:10.388 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:10.388+0000 7f9abcc68700 1 -- 172.21.15.160:0/1907102488 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9a9c005670 con 0x7f9ab80f6510 2023-12-25T20:10:10.388 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:10.389+0000 7f9aa6ffd700 1 -- 172.21.15.160:0/1907102488 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9aac00f300 con 0x7f9ab80f6510 2023-12-25T20:10:10.388 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:10:10.388 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:10:10.390 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:10.390+0000 7f9abcc68700 1 -- 172.21.15.160:0/1907102488 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f9aa002ff10 msgr2=0x7f9aa00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:10:10.390 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:10.390+0000 7f9abcc68700 1 --2- 172.21.15.160:0/1907102488 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f9aa002ff10 0x7f9aa00323d0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f9aa80096a0 tx=0x7f9aa8006b40).stop 2023-12-25T20:10:10.390 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:10.390+0000 7f9abcc68700 1 -- 172.21.15.160:0/1907102488 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9ab80f6510 msgr2=0x7f9ab8106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:10:10.390 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:10.390+0000 7f9abcc68700 1 --2- 172.21.15.160:0/1907102488 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9ab80f6510 0x7f9ab8106730 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f9aac014000 tx=0x7f9aac00b040).stop 2023-12-25T20:10:10.390 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:10.390+0000 7f9abcc68700 1 -- 172.21.15.160:0/1907102488 shutdown_connections 2023-12-25T20:10:10.390 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:10.390+0000 7f9abcc68700 1 --2- 172.21.15.160:0/1907102488 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f9aa002ff10 0x7f9aa00323d0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:10:10.390 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:10.390+0000 7f9abcc68700 1 --2- 172.21.15.160:0/1907102488 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9ab80f6510 0x7f9ab8106730 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:10:10.390 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:10.390+0000 7f9abcc68700 1 -- 172.21.15.160:0/1907102488 >> 172.21.15.160:0/1907102488 conn(0x7f9ab80f1c70 msgr2=0x7f9ab80f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:10:10.390 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:10.390+0000 7f9abcc68700 1 -- 172.21.15.160:0/1907102488 shutdown_connections 2023-12-25T20:10:10.391 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:10.390+0000 7f9abcc68700 1 -- 172.21.15.160:0/1907102488 wait complete. 2023-12-25T20:10:10.391 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:10:11.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:10:10 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:10:10.389246+0000 mon.smithi050 (mon.0) 172 : audit [DBG] from='client.? 172.21.15.160:0/1907102488' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:10:11.821 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:10:11.821 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:10:12.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:10:11 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:10:10.649321+0000 mgr.smithi050.nbzhun (mgr.14162) 89 : cluster [DBG] pgmap v82: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:10:13.011 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:13.010+0000 7f85bf096700 1 -- 172.21.15.160:0/570356043 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f85b8060220 msgr2=0x7f85b80f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:10:13.011 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:13.010+0000 7f85bf096700 1 --2- 172.21.15.160:0/570356043 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f85b8060220 0x7f85b80f9150 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f85ac009fb0 tx=0x7f85ac009b30).stop 2023-12-25T20:10:13.011 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:13.010+0000 7f85bf096700 1 -- 172.21.15.160:0/570356043 shutdown_connections 2023-12-25T20:10:13.011 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:13.010+0000 7f85bf096700 1 --2- 172.21.15.160:0/570356043 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f85b8060220 0x7f85b80f9150 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:10:13.011 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:13.010+0000 7f85bf096700 1 -- 172.21.15.160:0/570356043 >> 172.21.15.160:0/570356043 conn(0x7f85b80f0b70 msgr2=0x7f85b80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:10:13.011 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:13.010+0000 7f85bf096700 1 -- 172.21.15.160:0/570356043 shutdown_connections 2023-12-25T20:10:13.011 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:13.010+0000 7f85bf096700 1 -- 172.21.15.160:0/570356043 wait complete. 2023-12-25T20:10:13.011 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:13.010+0000 7f85bf096700 1 Processor -- start 2023-12-25T20:10:13.011 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:13.010+0000 7f85bf096700 1 -- start start 2023-12-25T20:10:13.012 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:13.011+0000 7f85bf096700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f85b8060220 0x7f85b80f6bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:10:13.012 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:13.011+0000 7f85bf096700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85b80f5050 con 0x7f85b8060220 2023-12-25T20:10:13.012 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:13.011+0000 7f85bce32700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f85b8060220 0x7f85b80f6bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:10:13.012 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:13.011+0000 7f85bce32700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f85b8060220 0x7f85b80f6bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:45246/0 (socket says 172.21.15.160:45246) 2023-12-25T20:10:13.012 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:13.011+0000 7f85bce32700 1 -- 172.21.15.160:0/3895599854 learned_addr learned my addr 172.21.15.160:0/3895599854 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:10:13.013 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:13.011+0000 7f85bce32700 1 -- 172.21.15.160:0/3895599854 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85ac005040 con 0x7f85b8060220 2023-12-25T20:10:13.013 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:13.011+0000 7f85bce32700 1 --2- 172.21.15.160:0/3895599854 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f85b8060220 0x7f85b80f6bd0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f85ac013510 tx=0x7f85ac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:10:13.013 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:13.012+0000 7f85b57fa700 1 -- 172.21.15.160:0/3895599854 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f85ac009320 con 0x7f85b8060220 2023-12-25T20:10:13.013 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:13.012+0000 7f85bf096700 1 -- 172.21.15.160:0/3895599854 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85b80f51f0 con 0x7f85b8060220 2023-12-25T20:10:13.014 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:13.012+0000 7f85b57fa700 1 -- 172.21.15.160:0/3895599854 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f85ac013f00 con 0x7f85b8060220 2023-12-25T20:10:13.014 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:13.012+0000 7f85b57fa700 1 -- 172.21.15.160:0/3895599854 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f85ac014dc0 con 0x7f85b8060220 2023-12-25T20:10:13.014 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:13.012+0000 7f85bf096700 1 -- 172.21.15.160:0/3895599854 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85b80f5680 con 0x7f85b8060220 2023-12-25T20:10:13.014 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:13.012+0000 7f85b57fa700 1 -- 172.21.15.160:0/3895599854 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f85ac014420 con 0x7f85b8060220 2023-12-25T20:10:13.014 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:13.012+0000 7f85bf096700 1 -- 172.21.15.160:0/3895599854 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85b8059100 con 0x7f85b8060220 2023-12-25T20:10:13.014 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:13.013+0000 7f85b57fa700 1 --2- 172.21.15.160:0/3895599854 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f85a802ff60 0x7f85a8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:10:13.014 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:13.013+0000 7f85b57fa700 1 -- 172.21.15.160:0/3895599854 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f85ac041050 con 0x7f85b8060220 2023-12-25T20:10:13.014 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:13.013+0000 7f85b7fff700 1 --2- 172.21.15.160:0/3895599854 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f85a802ff60 0x7f85a8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:10:13.014 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:13.013+0000 7f85b7fff700 1 --2- 172.21.15.160:0/3895599854 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f85a802ff60 0x7f85a8032420 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f85a40096a0 tx=0x7f85a4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:10:13.017 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:13.017+0000 7f85b57fa700 1 -- 172.21.15.160:0/3895599854 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f85ac0095f0 con 0x7f85b8060220 2023-12-25T20:10:13.222 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:13.223+0000 7f85bf096700 1 -- 172.21.15.160:0/3895599854 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f85b80fbf50 con 0x7f85b8060220 2023-12-25T20:10:13.223 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:13.223+0000 7f85b57fa700 1 -- 172.21.15.160:0/3895599854 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f85ac01cb10 con 0x7f85b8060220 2023-12-25T20:10:13.223 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:10:13.223 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:10:13.225 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:13.225+0000 7f85bf096700 1 -- 172.21.15.160:0/3895599854 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f85a802ff60 msgr2=0x7f85a8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:10:13.225 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:13.225+0000 7f85bf096700 1 --2- 172.21.15.160:0/3895599854 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f85a802ff60 0x7f85a8032420 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f85a40096a0 tx=0x7f85a4006b40).stop 2023-12-25T20:10:13.225 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:13.225+0000 7f85bf096700 1 -- 172.21.15.160:0/3895599854 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f85b8060220 msgr2=0x7f85b80f6bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:10:13.225 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:13.225+0000 7f85bf096700 1 --2- 172.21.15.160:0/3895599854 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f85b8060220 0x7f85b80f6bd0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f85ac013510 tx=0x7f85ac00b040).stop 2023-12-25T20:10:13.225 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:13.225+0000 7f85bf096700 1 -- 172.21.15.160:0/3895599854 shutdown_connections 2023-12-25T20:10:13.225 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:13.225+0000 7f85bf096700 1 --2- 172.21.15.160:0/3895599854 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f85a802ff60 0x7f85a8032420 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:10:13.225 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:13.225+0000 7f85bf096700 1 --2- 172.21.15.160:0/3895599854 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f85b8060220 0x7f85b80f6bd0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:10:13.226 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:13.225+0000 7f85bf096700 1 -- 172.21.15.160:0/3895599854 >> 172.21.15.160:0/3895599854 conn(0x7f85b80f0b70 msgr2=0x7f85b80f16a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:10:13.226 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:13.225+0000 7f85bf096700 1 -- 172.21.15.160:0/3895599854 shutdown_connections 2023-12-25T20:10:13.226 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:13.225+0000 7f85bf096700 1 -- 172.21.15.160:0/3895599854 wait complete. 2023-12-25T20:10:13.226 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:10:14.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:10:13 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:10:12.649613+0000 mgr.smithi050.nbzhun (mgr.14162) 90 : cluster [DBG] pgmap v83: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:10:14.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:10:13 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:10:13.224107+0000 mon.smithi050 (mon.0) 173 : audit [DBG] from='client.? 172.21.15.160:0/3895599854' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:10:14.652 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:10:14.652 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:10:15.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:10:14 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:10:14.764007+0000 mon.smithi050 (mon.0) 174 : audit [INF] from='mgr.14162 172.21.15.50:0/3178367813' entity='mgr.smithi050.nbzhun' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi050.nbzhun/mirror_snapshot_schedule"}]: dispatch 2023-12-25T20:10:15.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:10:14 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:10:14.765351+0000 mon.smithi050 (mon.0) 175 : audit [INF] from='mgr.14162 172.21.15.50:0/3178367813' entity='mgr.smithi050.nbzhun' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi050.nbzhun/trash_purge_schedule"}]: dispatch 2023-12-25T20:10:15.859 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:15.859+0000 7f7a8a23a700 1 -- 172.21.15.160:0/2256425482 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7a840f6510 msgr2=0x7f7a840f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:10:15.860 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:15.859+0000 7f7a8a23a700 1 --2- 172.21.15.160:0/2256425482 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7a840f6510 0x7f7a840f6930 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f7a74009fb0 tx=0x7f7a74009b30).stop 2023-12-25T20:10:15.860 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:15.859+0000 7f7a8a23a700 1 -- 172.21.15.160:0/2256425482 shutdown_connections 2023-12-25T20:10:15.860 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:15.859+0000 7f7a8a23a700 1 --2- 172.21.15.160:0/2256425482 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7a840f6510 0x7f7a840f6930 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:10:15.860 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:15.859+0000 7f7a8a23a700 1 -- 172.21.15.160:0/2256425482 >> 172.21.15.160:0/2256425482 conn(0x7f7a840f1c70 msgr2=0x7f7a840f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:10:15.860 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:15.859+0000 7f7a8a23a700 1 -- 172.21.15.160:0/2256425482 shutdown_connections 2023-12-25T20:10:15.860 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:15.859+0000 7f7a8a23a700 1 -- 172.21.15.160:0/2256425482 wait complete. 2023-12-25T20:10:15.860 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:15.860+0000 7f7a8a23a700 1 Processor -- start 2023-12-25T20:10:15.860 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:15.860+0000 7f7a8a23a700 1 -- start start 2023-12-25T20:10:15.862 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:15.860+0000 7f7a8a23a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7a840f6510 0x7f7a84108930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:10:15.862 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:15.860+0000 7f7a8a23a700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a84108e20 con 0x7f7a840f6510 2023-12-25T20:10:15.862 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:15.860+0000 7f7a837fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7a840f6510 0x7f7a84108930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:10:15.862 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:15.860+0000 7f7a837fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7a840f6510 0x7f7a84108930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:45274/0 (socket says 172.21.15.160:45274) 2023-12-25T20:10:15.862 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:15.860+0000 7f7a837fe700 1 -- 172.21.15.160:0/3019421919 learned_addr learned my addr 172.21.15.160:0/3019421919 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:10:15.862 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:15.861+0000 7f7a837fe700 1 -- 172.21.15.160:0/3019421919 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a74005040 con 0x7f7a840f6510 2023-12-25T20:10:15.862 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:15.861+0000 7f7a837fe700 1 --2- 172.21.15.160:0/3019421919 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7a840f6510 0x7f7a84108930 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f7a74013510 tx=0x7f7a7400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:10:15.862 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:15.861+0000 7f7a6bfff700 1 -- 172.21.15.160:0/3019421919 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7a74009320 con 0x7f7a840f6510 2023-12-25T20:10:15.862 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:15.861+0000 7f7a8a23a700 1 -- 172.21.15.160:0/3019421919 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a84109020 con 0x7f7a840f6510 2023-12-25T20:10:15.862 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:15.861+0000 7f7a6bfff700 1 -- 172.21.15.160:0/3019421919 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7a74013f00 con 0x7f7a840f6510 2023-12-25T20:10:15.862 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:15.861+0000 7f7a6bfff700 1 -- 172.21.15.160:0/3019421919 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7a74014d90 con 0x7f7a840f6510 2023-12-25T20:10:15.862 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:15.861+0000 7f7a8a23a700 1 -- 172.21.15.160:0/3019421919 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a841094b0 con 0x7f7a840f6510 2023-12-25T20:10:15.863 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:15.862+0000 7f7a6bfff700 1 -- 172.21.15.160:0/3019421919 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7a740263a0 con 0x7f7a840f6510 2023-12-25T20:10:15.863 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:15.862+0000 7f7a6bfff700 1 --2- 172.21.15.160:0/3019421919 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f7a6c02ff60 0x7f7a6c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:10:15.863 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:15.862+0000 7f7a6bfff700 1 -- 172.21.15.160:0/3019421919 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7a74041100 con 0x7f7a840f6510 2023-12-25T20:10:15.863 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:15.862+0000 7f7a8a23a700 1 -- 172.21.15.160:0/3019421919 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a84045ad0 con 0x7f7a840f6510 2023-12-25T20:10:15.863 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:15.863+0000 7f7a82ffd700 1 --2- 172.21.15.160:0/3019421919 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f7a6c02ff60 0x7f7a6c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:10:15.864 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:15.864+0000 7f7a82ffd700 1 --2- 172.21.15.160:0/3019421919 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f7a6c02ff60 0x7f7a6c032420 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f7a780096a0 tx=0x7f7a78006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:10:15.866 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:15.867+0000 7f7a6bfff700 1 -- 172.21.15.160:0/3019421919 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7a74012920 con 0x7f7a840f6510 2023-12-25T20:10:16.069 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:16.070+0000 7f7a8a23a700 1 -- 172.21.15.160:0/3019421919 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7a840fc050 con 0x7f7a840f6510 2023-12-25T20:10:16.071 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:16.071+0000 7f7a6bfff700 1 -- 172.21.15.160:0/3019421919 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7a7402b020 con 0x7f7a840f6510 2023-12-25T20:10:16.071 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:10:16.071 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:10:16.072 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:16.072+0000 7f7a8a23a700 1 -- 172.21.15.160:0/3019421919 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f7a6c02ff60 msgr2=0x7f7a6c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:10:16.073 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:16.072+0000 7f7a8a23a700 1 --2- 172.21.15.160:0/3019421919 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f7a6c02ff60 0x7f7a6c032420 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f7a780096a0 tx=0x7f7a78006b40).stop 2023-12-25T20:10:16.073 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:16.072+0000 7f7a8a23a700 1 -- 172.21.15.160:0/3019421919 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7a840f6510 msgr2=0x7f7a84108930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:10:16.073 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:16.072+0000 7f7a8a23a700 1 --2- 172.21.15.160:0/3019421919 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7a840f6510 0x7f7a84108930 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f7a74013510 tx=0x7f7a7400b040).stop 2023-12-25T20:10:16.073 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:16.073+0000 7f7a8a23a700 1 -- 172.21.15.160:0/3019421919 shutdown_connections 2023-12-25T20:10:16.073 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:16.073+0000 7f7a8a23a700 1 --2- 172.21.15.160:0/3019421919 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f7a6c02ff60 0x7f7a6c032420 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:10:16.073 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:16.073+0000 7f7a8a23a700 1 --2- 172.21.15.160:0/3019421919 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7a840f6510 0x7f7a84108930 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:10:16.074 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:16.073+0000 7f7a8a23a700 1 -- 172.21.15.160:0/3019421919 >> 172.21.15.160:0/3019421919 conn(0x7f7a840f1c70 msgr2=0x7f7a840f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:10:16.074 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:16.073+0000 7f7a8a23a700 1 -- 172.21.15.160:0/3019421919 shutdown_connections 2023-12-25T20:10:16.074 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:16.073+0000 7f7a8a23a700 1 -- 172.21.15.160:0/3019421919 wait complete. 2023-12-25T20:10:16.074 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:10:16.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:10:15 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:10:14.650025+0000 mgr.smithi050.nbzhun (mgr.14162) 91 : cluster [DBG] pgmap v84: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:10:17.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:10:16 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:10:16.071289+0000 mon.smithi050 (mon.0) 176 : audit [DBG] from='client.? 172.21.15.160:0/3019421919' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:10:17.432 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:10:17.432 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:10:18.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:10:17 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:10:16.650374+0000 mgr.smithi050.nbzhun (mgr.14162) 92 : cluster [DBG] pgmap v85: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:10:18.634 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:18.631+0000 7f637c393700 1 -- 172.21.15.160:0/2192599654 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f63740f5410 msgr2=0x7f63740f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:10:18.634 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:18.631+0000 7f637c393700 1 --2- 172.21.15.160:0/2192599654 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f63740f5410 0x7f63740f5830 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7f6370009fb0 tx=0x7f6370009b30).stop 2023-12-25T20:10:18.634 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:18.631+0000 7f637c393700 1 -- 172.21.15.160:0/2192599654 shutdown_connections 2023-12-25T20:10:18.634 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:18.631+0000 7f637c393700 1 --2- 172.21.15.160:0/2192599654 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f63740f5410 0x7f63740f5830 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:10:18.634 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:18.631+0000 7f637c393700 1 -- 172.21.15.160:0/2192599654 >> 172.21.15.160:0/2192599654 conn(0x7f63740f0b70 msgr2=0x7f63740f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:10:18.634 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:18.631+0000 7f637c393700 1 -- 172.21.15.160:0/2192599654 shutdown_connections 2023-12-25T20:10:18.634 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:18.631+0000 7f637c393700 1 -- 172.21.15.160:0/2192599654 wait complete. 2023-12-25T20:10:18.635 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:18.632+0000 7f637c393700 1 Processor -- start 2023-12-25T20:10:18.635 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:18.632+0000 7f637c393700 1 -- start start 2023-12-25T20:10:18.635 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:18.632+0000 7f637c393700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f63740f5410 0x7f63740fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:10:18.635 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:18.632+0000 7f637c393700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63740fb820 con 0x7f63740f5410 2023-12-25T20:10:18.635 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:18.632+0000 7f637a12f700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f63740f5410 0x7f63740fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:10:18.635 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:18.632+0000 7f637a12f700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f63740f5410 0x7f63740fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:33340/0 (socket says 172.21.15.160:33340) 2023-12-25T20:10:18.635 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:18.632+0000 7f637a12f700 1 -- 172.21.15.160:0/964939902 learned_addr learned my addr 172.21.15.160:0/964939902 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:10:18.635 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:18.632+0000 7f637a12f700 1 -- 172.21.15.160:0/964939902 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6370005040 con 0x7f63740f5410 2023-12-25T20:10:18.635 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:18.633+0000 7f637a12f700 1 --2- 172.21.15.160:0/964939902 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f63740f5410 0x7f63740fd3a0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7f6370013510 tx=0x7f637000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:10:18.635 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:18.633+0000 7f636affd700 1 -- 172.21.15.160:0/964939902 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6370009320 con 0x7f63740f5410 2023-12-25T20:10:18.635 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:18.633+0000 7f637c393700 1 -- 172.21.15.160:0/964939902 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63740fb9c0 con 0x7f63740f5410 2023-12-25T20:10:18.635 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:18.633+0000 7f636affd700 1 -- 172.21.15.160:0/964939902 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6370013f00 con 0x7f63740f5410 2023-12-25T20:10:18.635 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:18.633+0000 7f636affd700 1 -- 172.21.15.160:0/964939902 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6370014d90 con 0x7f63740f5410 2023-12-25T20:10:18.635 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:18.633+0000 7f637c393700 1 -- 172.21.15.160:0/964939902 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63740fbe50 con 0x7f63740f5410 2023-12-25T20:10:18.636 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:18.634+0000 7f636affd700 1 -- 172.21.15.160:0/964939902 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f63700263a0 con 0x7f63740f5410 2023-12-25T20:10:18.636 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:18.634+0000 7f636affd700 1 --2- 172.21.15.160:0/964939902 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f636002ff60 0x7f6360032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:10:18.636 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:18.634+0000 7f637c393700 1 -- 172.21.15.160:0/964939902 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6374059100 con 0x7f63740f5410 2023-12-25T20:10:18.637 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:18.634+0000 7f636affd700 1 -- 172.21.15.160:0/964939902 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f63700413b0 con 0x7f63740f5410 2023-12-25T20:10:18.637 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:18.634+0000 7f637992e700 1 --2- 172.21.15.160:0/964939902 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f636002ff60 0x7f6360032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:10:18.637 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:18.634+0000 7f637992e700 1 --2- 172.21.15.160:0/964939902 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f636002ff60 0x7f6360032420 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f63640096a0 tx=0x7f6364006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:10:18.639 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:18.639+0000 7f636affd700 1 -- 172.21.15.160:0/964939902 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f63700126f0 con 0x7f63740f5410 2023-12-25T20:10:18.847 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:18.846+0000 7f637c393700 1 -- 172.21.15.160:0/964939902 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f63740fae60 con 0x7f63740f5410 2023-12-25T20:10:18.848 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:18.847+0000 7f636affd700 1 -- 172.21.15.160:0/964939902 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6370012330 con 0x7f63740f5410 2023-12-25T20:10:18.848 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:10:18.848 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:10:18.850 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:18.849+0000 7f637c393700 1 -- 172.21.15.160:0/964939902 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f636002ff60 msgr2=0x7f6360032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:10:18.850 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:18.849+0000 7f637c393700 1 --2- 172.21.15.160:0/964939902 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f636002ff60 0x7f6360032420 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f63640096a0 tx=0x7f6364006b40).stop 2023-12-25T20:10:18.850 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:18.849+0000 7f637c393700 1 -- 172.21.15.160:0/964939902 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f63740f5410 msgr2=0x7f63740fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:10:18.850 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:18.849+0000 7f637c393700 1 --2- 172.21.15.160:0/964939902 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f63740f5410 0x7f63740fd3a0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7f6370013510 tx=0x7f637000b040).stop 2023-12-25T20:10:18.850 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:18.849+0000 7f637c393700 1 -- 172.21.15.160:0/964939902 shutdown_connections 2023-12-25T20:10:18.850 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:18.849+0000 7f637c393700 1 --2- 172.21.15.160:0/964939902 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f636002ff60 0x7f6360032420 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:10:18.850 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:18.849+0000 7f637c393700 1 --2- 172.21.15.160:0/964939902 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f63740f5410 0x7f63740fd3a0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:10:18.850 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:18.849+0000 7f637c393700 1 -- 172.21.15.160:0/964939902 >> 172.21.15.160:0/964939902 conn(0x7f63740f0b70 msgr2=0x7f637405fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:10:18.850 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:18.849+0000 7f637c393700 1 -- 172.21.15.160:0/964939902 shutdown_connections 2023-12-25T20:10:18.851 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:18.849+0000 7f637c393700 1 -- 172.21.15.160:0/964939902 wait complete. 2023-12-25T20:10:18.851 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:10:20.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:10:19 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:10:18.650671+0000 mgr.smithi050.nbzhun (mgr.14162) 93 : cluster [DBG] pgmap v86: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:10:20.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:10:19 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:10:18.848843+0000 mon.smithi050 (mon.0) 177 : audit [DBG] from='client.? 172.21.15.160:0/964939902' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:10:20.237 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:10:20.237 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:10:21.473 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:21.470+0000 7f4c6f4c2700 1 -- 172.21.15.160:0/295541202 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4c68060280 msgr2=0x7f4c680f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:10:21.473 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:21.470+0000 7f4c6f4c2700 1 --2- 172.21.15.160:0/295541202 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4c68060280 0x7f4c680f90f0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7f4c58009fb0 tx=0x7f4c58009b30).stop 2023-12-25T20:10:21.473 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:21.470+0000 7f4c6f4c2700 1 -- 172.21.15.160:0/295541202 shutdown_connections 2023-12-25T20:10:21.473 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:21.470+0000 7f4c6f4c2700 1 --2- 172.21.15.160:0/295541202 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4c68060280 0x7f4c680f90f0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:10:21.473 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:21.470+0000 7f4c6f4c2700 1 -- 172.21.15.160:0/295541202 >> 172.21.15.160:0/295541202 conn(0x7f4c680f0b70 msgr2=0x7f4c680f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:10:21.473 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:21.470+0000 7f4c6f4c2700 1 -- 172.21.15.160:0/295541202 shutdown_connections 2023-12-25T20:10:21.473 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:21.471+0000 7f4c6f4c2700 1 -- 172.21.15.160:0/295541202 wait complete. 2023-12-25T20:10:21.473 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:21.471+0000 7f4c6f4c2700 1 Processor -- start 2023-12-25T20:10:21.473 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:21.471+0000 7f4c6f4c2700 1 -- start start 2023-12-25T20:10:21.474 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:21.471+0000 7f4c6f4c2700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4c68060280 0x7f4c681066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:10:21.474 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:21.471+0000 7f4c6f4c2700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c68106bd0 con 0x7f4c68060280 2023-12-25T20:10:21.474 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:21.471+0000 7f4c6d25e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4c68060280 0x7f4c681066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:10:21.474 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:21.471+0000 7f4c6d25e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4c68060280 0x7f4c681066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:33352/0 (socket says 172.21.15.160:33352) 2023-12-25T20:10:21.474 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:21.471+0000 7f4c6d25e700 1 -- 172.21.15.160:0/2869571182 learned_addr learned my addr 172.21.15.160:0/2869571182 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:10:21.474 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:21.472+0000 7f4c6d25e700 1 -- 172.21.15.160:0/2869571182 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c58005040 con 0x7f4c68060280 2023-12-25T20:10:21.474 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:21.472+0000 7f4c6d25e700 1 --2- 172.21.15.160:0/2869571182 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4c68060280 0x7f4c681066e0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7f4c58009830 tx=0x7f4c5800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:10:21.474 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:21.472+0000 7f4c5dffb700 1 -- 172.21.15.160:0/2869571182 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4c58009320 con 0x7f4c68060280 2023-12-25T20:10:21.474 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:21.472+0000 7f4c6f4c2700 1 -- 172.21.15.160:0/2869571182 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c68106dd0 con 0x7f4c68060280 2023-12-25T20:10:21.474 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:21.472+0000 7f4c5dffb700 1 -- 172.21.15.160:0/2869571182 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4c580136c0 con 0x7f4c68060280 2023-12-25T20:10:21.474 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:21.472+0000 7f4c5dffb700 1 -- 172.21.15.160:0/2869571182 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4c5801ee90 con 0x7f4c68060280 2023-12-25T20:10:21.474 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:21.472+0000 7f4c6f4c2700 1 -- 172.21.15.160:0/2869571182 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c68107260 con 0x7f4c68060280 2023-12-25T20:10:21.475 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:21.473+0000 7f4c5dffb700 1 -- 172.21.15.160:0/2869571182 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4c5801e420 con 0x7f4c68060280 2023-12-25T20:10:21.475 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:21.473+0000 7f4c6f4c2700 1 -- 172.21.15.160:0/2869571182 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c68045ad0 con 0x7f4c68060280 2023-12-25T20:10:21.475 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:21.473+0000 7f4c5dffb700 1 --2- 172.21.15.160:0/2869571182 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f4c5402ff10 0x7f4c540323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:10:21.475 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:21.473+0000 7f4c5dffb700 1 -- 172.21.15.160:0/2869571182 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4c58040f90 con 0x7f4c68060280 2023-12-25T20:10:21.475 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:21.473+0000 7f4c6ca5d700 1 --2- 172.21.15.160:0/2869571182 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f4c5402ff10 0x7f4c540323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:10:21.475 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:21.474+0000 7f4c6ca5d700 1 --2- 172.21.15.160:0/2869571182 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f4c5402ff10 0x7f4c540323d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f4c640096a0 tx=0x7f4c64006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:10:21.478 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:21.478+0000 7f4c5dffb700 1 -- 172.21.15.160:0/2869571182 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4c5801c790 con 0x7f4c68060280 2023-12-25T20:10:21.682 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:21.681+0000 7f4c6f4c2700 1 -- 172.21.15.160:0/2869571182 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4c681080b0 con 0x7f4c68060280 2023-12-25T20:10:21.682 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:21.682+0000 7f4c5dffb700 1 -- 172.21.15.160:0/2869571182 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4c5802c2d0 con 0x7f4c68060280 2023-12-25T20:10:21.683 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:10:21.683 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:10:21.684 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:21.684+0000 7f4c6f4c2700 1 -- 172.21.15.160:0/2869571182 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f4c5402ff10 msgr2=0x7f4c540323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:10:21.685 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:21.684+0000 7f4c6f4c2700 1 --2- 172.21.15.160:0/2869571182 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f4c5402ff10 0x7f4c540323d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f4c640096a0 tx=0x7f4c64006b40).stop 2023-12-25T20:10:21.685 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:21.684+0000 7f4c6f4c2700 1 -- 172.21.15.160:0/2869571182 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4c68060280 msgr2=0x7f4c681066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:10:21.685 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:21.684+0000 7f4c6f4c2700 1 --2- 172.21.15.160:0/2869571182 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4c68060280 0x7f4c681066e0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7f4c58009830 tx=0x7f4c5800b040).stop 2023-12-25T20:10:21.685 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:21.684+0000 7f4c6f4c2700 1 -- 172.21.15.160:0/2869571182 shutdown_connections 2023-12-25T20:10:21.685 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:21.684+0000 7f4c6f4c2700 1 --2- 172.21.15.160:0/2869571182 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f4c5402ff10 0x7f4c540323d0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:10:21.685 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:21.684+0000 7f4c6f4c2700 1 --2- 172.21.15.160:0/2869571182 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4c68060280 0x7f4c681066e0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:10:21.685 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:21.684+0000 7f4c6f4c2700 1 -- 172.21.15.160:0/2869571182 >> 172.21.15.160:0/2869571182 conn(0x7f4c680f0b70 msgr2=0x7f4c680f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:10:21.685 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:21.684+0000 7f4c6f4c2700 1 -- 172.21.15.160:0/2869571182 shutdown_connections 2023-12-25T20:10:21.685 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:21.684+0000 7f4c6f4c2700 1 -- 172.21.15.160:0/2869571182 wait complete. 2023-12-25T20:10:21.685 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:10:22.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:10:21 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:10:20.650973+0000 mgr.smithi050.nbzhun (mgr.14162) 94 : cluster [DBG] pgmap v87: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:10:22.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:10:21 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:10:21.683676+0000 mon.smithi050 (mon.0) 178 : audit [DBG] from='client.? 172.21.15.160:0/2869571182' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:10:23.075 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:10:23.075 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:10:24.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:10:23 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:10:22.651226+0000 mgr.smithi050.nbzhun (mgr.14162) 95 : cluster [DBG] pgmap v88: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:10:24.374 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:24.372+0000 7fd1e2eb3700 1 -- 172.21.15.160:0/2673080520 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd1dc0f5410 msgr2=0x7fd1dc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:10:24.374 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:24.372+0000 7fd1e2eb3700 1 --2- 172.21.15.160:0/2673080520 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd1dc0f5410 0x7fd1dc0f5830 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7fd1d0009fb0 tx=0x7fd1d0009b30).stop 2023-12-25T20:10:24.374 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:24.372+0000 7fd1e2eb3700 1 -- 172.21.15.160:0/2673080520 shutdown_connections 2023-12-25T20:10:24.374 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:24.372+0000 7fd1e2eb3700 1 --2- 172.21.15.160:0/2673080520 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd1dc0f5410 0x7fd1dc0f5830 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:10:24.374 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:24.372+0000 7fd1e2eb3700 1 -- 172.21.15.160:0/2673080520 >> 172.21.15.160:0/2673080520 conn(0x7fd1dc0f0b70 msgr2=0x7fd1dc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:10:24.374 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:24.372+0000 7fd1e2eb3700 1 -- 172.21.15.160:0/2673080520 shutdown_connections 2023-12-25T20:10:24.375 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:24.372+0000 7fd1e2eb3700 1 -- 172.21.15.160:0/2673080520 wait complete. 2023-12-25T20:10:24.375 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:24.373+0000 7fd1e2eb3700 1 Processor -- start 2023-12-25T20:10:24.375 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:24.373+0000 7fd1e2eb3700 1 -- start start 2023-12-25T20:10:24.375 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:24.373+0000 7fd1e2eb3700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd1dc0f5410 0x7fd1dc108920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:10:24.375 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:24.373+0000 7fd1e2eb3700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1dc108e10 con 0x7fd1dc0f5410 2023-12-25T20:10:24.375 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:24.373+0000 7fd1e0c4f700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd1dc0f5410 0x7fd1dc108920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:10:24.375 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:24.373+0000 7fd1e0c4f700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd1dc0f5410 0x7fd1dc108920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:33366/0 (socket says 172.21.15.160:33366) 2023-12-25T20:10:24.375 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:24.373+0000 7fd1e0c4f700 1 -- 172.21.15.160:0/3222520621 learned_addr learned my addr 172.21.15.160:0/3222520621 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:10:24.376 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:24.374+0000 7fd1e0c4f700 1 -- 172.21.15.160:0/3222520621 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1d0005040 con 0x7fd1dc0f5410 2023-12-25T20:10:24.376 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:24.374+0000 7fd1e0c4f700 1 --2- 172.21.15.160:0/3222520621 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd1dc0f5410 0x7fd1dc108920 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7fd1d0013510 tx=0x7fd1d000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:10:24.376 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:24.374+0000 7fd1d97fa700 1 -- 172.21.15.160:0/3222520621 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd1d0009320 con 0x7fd1dc0f5410 2023-12-25T20:10:24.377 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:24.374+0000 7fd1e2eb3700 1 -- 172.21.15.160:0/3222520621 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1dc109010 con 0x7fd1dc0f5410 2023-12-25T20:10:24.377 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:24.374+0000 7fd1d97fa700 1 -- 172.21.15.160:0/3222520621 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd1d0013f00 con 0x7fd1dc0f5410 2023-12-25T20:10:24.377 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:24.374+0000 7fd1d97fa700 1 -- 172.21.15.160:0/3222520621 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd1d0014d90 con 0x7fd1dc0f5410 2023-12-25T20:10:24.377 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:24.374+0000 7fd1e2eb3700 1 -- 172.21.15.160:0/3222520621 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1dc1094a0 con 0x7fd1dc0f5410 2023-12-25T20:10:24.377 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:24.375+0000 7fd1d97fa700 1 -- 172.21.15.160:0/3222520621 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd1d0014420 con 0x7fd1dc0f5410 2023-12-25T20:10:24.377 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:24.375+0000 7fd1e2eb3700 1 -- 172.21.15.160:0/3222520621 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1dc045ad0 con 0x7fd1dc0f5410 2023-12-25T20:10:24.377 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:24.375+0000 7fd1d97fa700 1 --2- 172.21.15.160:0/3222520621 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fd1cc02ff60 0x7fd1cc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:10:24.377 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:24.375+0000 7fd1d97fa700 1 -- 172.21.15.160:0/3222520621 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd1d00410c0 con 0x7fd1dc0f5410 2023-12-25T20:10:24.377 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:24.375+0000 7fd1dbfff700 1 --2- 172.21.15.160:0/3222520621 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fd1cc02ff60 0x7fd1cc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:10:24.377 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:24.376+0000 7fd1dbfff700 1 --2- 172.21.15.160:0/3222520621 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fd1cc02ff60 0x7fd1cc032420 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7fd1c80096a0 tx=0x7fd1c8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:10:24.380 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:24.380+0000 7fd1d97fa700 1 -- 172.21.15.160:0/3222520621 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd1d00095f0 con 0x7fd1dc0f5410 2023-12-25T20:10:24.587 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:24.586+0000 7fd1e2eb3700 1 -- 172.21.15.160:0/3222520621 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd1dc10a350 con 0x7fd1dc0f5410 2023-12-25T20:10:24.588 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:24.587+0000 7fd1d97fa700 1 -- 172.21.15.160:0/3222520621 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd1d001db70 con 0x7fd1dc0f5410 2023-12-25T20:10:24.588 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:10:24.588 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:10:24.590 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:24.589+0000 7fd1e2eb3700 1 -- 172.21.15.160:0/3222520621 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fd1cc02ff60 msgr2=0x7fd1cc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:10:24.590 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:24.589+0000 7fd1e2eb3700 1 --2- 172.21.15.160:0/3222520621 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fd1cc02ff60 0x7fd1cc032420 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7fd1c80096a0 tx=0x7fd1c8006b40).stop 2023-12-25T20:10:24.590 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:24.589+0000 7fd1e2eb3700 1 -- 172.21.15.160:0/3222520621 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd1dc0f5410 msgr2=0x7fd1dc108920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:10:24.590 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:24.589+0000 7fd1e2eb3700 1 --2- 172.21.15.160:0/3222520621 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd1dc0f5410 0x7fd1dc108920 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7fd1d0013510 tx=0x7fd1d000b040).stop 2023-12-25T20:10:24.590 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:24.589+0000 7fd1e2eb3700 1 -- 172.21.15.160:0/3222520621 shutdown_connections 2023-12-25T20:10:24.590 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:24.589+0000 7fd1e2eb3700 1 --2- 172.21.15.160:0/3222520621 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fd1cc02ff60 0x7fd1cc032420 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:10:24.590 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:24.589+0000 7fd1e2eb3700 1 --2- 172.21.15.160:0/3222520621 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd1dc0f5410 0x7fd1dc108920 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:10:24.590 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:24.589+0000 7fd1e2eb3700 1 -- 172.21.15.160:0/3222520621 >> 172.21.15.160:0/3222520621 conn(0x7fd1dc0f0b70 msgr2=0x7fd1dc05fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:10:24.590 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:24.589+0000 7fd1e2eb3700 1 -- 172.21.15.160:0/3222520621 shutdown_connections 2023-12-25T20:10:24.590 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:24.589+0000 7fd1e2eb3700 1 -- 172.21.15.160:0/3222520621 wait complete. 2023-12-25T20:10:24.591 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:10:25.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:10:24 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:10:24.588639+0000 mon.smithi050 (mon.0) 179 : audit [DBG] from='client.? 172.21.15.160:0/3222520621' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:10:25.996 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:10:25.997 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:10:26.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:10:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:10:24.651597+0000 mgr.smithi050.nbzhun (mgr.14162) 96 : cluster [DBG] pgmap v89: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:10:27.223 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:27.222+0000 7f8e3535a700 1 -- 172.21.15.160:0/2695435014 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8e300f6510 msgr2=0x7f8e300f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:10:27.223 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:27.222+0000 7f8e3535a700 1 --2- 172.21.15.160:0/2695435014 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8e300f6510 0x7f8e300f6930 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7f8e24009fb0 tx=0x7f8e24009b30).stop 2023-12-25T20:10:27.223 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:27.222+0000 7f8e3535a700 1 -- 172.21.15.160:0/2695435014 shutdown_connections 2023-12-25T20:10:27.223 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:27.222+0000 7f8e3535a700 1 --2- 172.21.15.160:0/2695435014 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8e300f6510 0x7f8e300f6930 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:10:27.223 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:27.222+0000 7f8e3535a700 1 -- 172.21.15.160:0/2695435014 >> 172.21.15.160:0/2695435014 conn(0x7f8e300f1c70 msgr2=0x7f8e300f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:10:27.223 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:27.222+0000 7f8e3535a700 1 -- 172.21.15.160:0/2695435014 shutdown_connections 2023-12-25T20:10:27.224 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:27.222+0000 7f8e3535a700 1 -- 172.21.15.160:0/2695435014 wait complete. 2023-12-25T20:10:27.225 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:27.223+0000 7f8e3535a700 1 Processor -- start 2023-12-25T20:10:27.226 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:27.223+0000 7f8e3535a700 1 -- start start 2023-12-25T20:10:27.226 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:27.223+0000 7f8e3535a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8e300f6510 0x7f8e30106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:10:27.226 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:27.223+0000 7f8e3535a700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e30106c20 con 0x7f8e300f6510 2023-12-25T20:10:27.226 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:27.223+0000 7f8e2effd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8e300f6510 0x7f8e30106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:10:27.226 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:27.223+0000 7f8e2effd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8e300f6510 0x7f8e30106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:33380/0 (socket says 172.21.15.160:33380) 2023-12-25T20:10:27.226 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:27.223+0000 7f8e2effd700 1 -- 172.21.15.160:0/1354566952 learned_addr learned my addr 172.21.15.160:0/1354566952 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:10:27.226 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:27.223+0000 7f8e2effd700 1 -- 172.21.15.160:0/1354566952 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e24005040 con 0x7f8e300f6510 2023-12-25T20:10:27.226 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:27.224+0000 7f8e2effd700 1 --2- 172.21.15.160:0/1354566952 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8e300f6510 0x7f8e30106730 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7f8e24013510 tx=0x7f8e2400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:10:27.226 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:27.224+0000 7f8e177fe700 1 -- 172.21.15.160:0/1354566952 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8e24009320 con 0x7f8e300f6510 2023-12-25T20:10:27.226 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:27.224+0000 7f8e3535a700 1 -- 172.21.15.160:0/1354566952 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e30106e20 con 0x7f8e300f6510 2023-12-25T20:10:27.226 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:27.224+0000 7f8e177fe700 1 -- 172.21.15.160:0/1354566952 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8e24013f00 con 0x7f8e300f6510 2023-12-25T20:10:27.227 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:27.224+0000 7f8e177fe700 1 -- 172.21.15.160:0/1354566952 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8e24014d90 con 0x7f8e300f6510 2023-12-25T20:10:27.227 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:27.224+0000 7f8e3535a700 1 -- 172.21.15.160:0/1354566952 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e301072b0 con 0x7f8e300f6510 2023-12-25T20:10:27.227 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:27.225+0000 7f8e177fe700 1 -- 172.21.15.160:0/1354566952 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8e240095f0 con 0x7f8e300f6510 2023-12-25T20:10:27.227 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:27.225+0000 7f8e3535a700 1 -- 172.21.15.160:0/1354566952 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e30045ad0 con 0x7f8e300f6510 2023-12-25T20:10:27.227 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:27.225+0000 7f8e177fe700 1 --2- 172.21.15.160:0/1354566952 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f8e1802ff60 0x7f8e18032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:10:27.227 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:27.225+0000 7f8e177fe700 1 -- 172.21.15.160:0/1354566952 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8e24040fe0 con 0x7f8e300f6510 2023-12-25T20:10:27.227 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:27.225+0000 7f8e2e7fc700 1 --2- 172.21.15.160:0/1354566952 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f8e1802ff60 0x7f8e18032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:10:27.227 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:27.225+0000 7f8e2e7fc700 1 --2- 172.21.15.160:0/1354566952 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f8e1802ff60 0x7f8e18032420 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f8e200096a0 tx=0x7f8e20006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:10:27.230 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:27.230+0000 7f8e177fe700 1 -- 172.21.15.160:0/1354566952 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8e2401d7f0 con 0x7f8e300f6510 2023-12-25T20:10:27.433 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:27.432+0000 7f8e3535a700 1 -- 172.21.15.160:0/1354566952 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8e300fbf60 con 0x7f8e300f6510 2023-12-25T20:10:27.434 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:27.433+0000 7f8e177fe700 1 -- 172.21.15.160:0/1354566952 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8e24010230 con 0x7f8e300f6510 2023-12-25T20:10:27.434 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:10:27.434 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:10:27.435 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:27.435+0000 7f8e3535a700 1 -- 172.21.15.160:0/1354566952 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f8e1802ff60 msgr2=0x7f8e18032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:10:27.436 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:27.435+0000 7f8e3535a700 1 --2- 172.21.15.160:0/1354566952 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f8e1802ff60 0x7f8e18032420 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f8e200096a0 tx=0x7f8e20006b40).stop 2023-12-25T20:10:27.436 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:27.435+0000 7f8e3535a700 1 -- 172.21.15.160:0/1354566952 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8e300f6510 msgr2=0x7f8e30106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:10:27.436 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:27.435+0000 7f8e3535a700 1 --2- 172.21.15.160:0/1354566952 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8e300f6510 0x7f8e30106730 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7f8e24013510 tx=0x7f8e2400b040).stop 2023-12-25T20:10:27.436 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:27.435+0000 7f8e3535a700 1 -- 172.21.15.160:0/1354566952 shutdown_connections 2023-12-25T20:10:27.436 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:27.435+0000 7f8e3535a700 1 --2- 172.21.15.160:0/1354566952 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f8e1802ff60 0x7f8e18032420 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:10:27.436 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:27.435+0000 7f8e3535a700 1 --2- 172.21.15.160:0/1354566952 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8e300f6510 0x7f8e30106730 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:10:27.436 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:27.435+0000 7f8e3535a700 1 -- 172.21.15.160:0/1354566952 >> 172.21.15.160:0/1354566952 conn(0x7f8e300f1c70 msgr2=0x7f8e300f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:10:27.436 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:27.435+0000 7f8e3535a700 1 -- 172.21.15.160:0/1354566952 shutdown_connections 2023-12-25T20:10:27.436 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:27.435+0000 7f8e3535a700 1 -- 172.21.15.160:0/1354566952 wait complete. 2023-12-25T20:10:27.436 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:10:28.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:10:27 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:10:26.651886+0000 mgr.smithi050.nbzhun (mgr.14162) 97 : cluster [DBG] pgmap v90: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:10:28.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:10:27 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:10:27.434796+0000 mon.smithi050 (mon.0) 180 : audit [DBG] from='client.? 172.21.15.160:0/1354566952' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:10:28.819 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:10:28.819 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:10:30.079 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:30.076+0000 7feae9485700 1 -- 172.21.15.160:0/2038860904 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feae40f51a0 msgr2=0x7feae40f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:10:30.079 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:30.076+0000 7feae9485700 1 --2- 172.21.15.160:0/2038860904 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feae40f51a0 0x7feae40f55c0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7fead4009fb0 tx=0x7fead4009b30).stop 2023-12-25T20:10:30.079 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:30.077+0000 7feae9485700 1 -- 172.21.15.160:0/2038860904 shutdown_connections 2023-12-25T20:10:30.079 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:30.077+0000 7feae9485700 1 --2- 172.21.15.160:0/2038860904 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feae40f51a0 0x7feae40f55c0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:10:30.080 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:30.077+0000 7feae9485700 1 -- 172.21.15.160:0/2038860904 >> 172.21.15.160:0/2038860904 conn(0x7feae40f09a0 msgr2=0x7feae40f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:10:30.080 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:30.077+0000 7feae9485700 1 -- 172.21.15.160:0/2038860904 shutdown_connections 2023-12-25T20:10:30.080 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:30.077+0000 7feae9485700 1 -- 172.21.15.160:0/2038860904 wait complete. 2023-12-25T20:10:30.080 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:30.077+0000 7feae9485700 1 Processor -- start 2023-12-25T20:10:30.080 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:30.077+0000 7feae9485700 1 -- start start 2023-12-25T20:10:30.080 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:30.077+0000 7feae9485700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feae40f51a0 0x7feae40fd120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:10:30.080 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:30.077+0000 7feae9485700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feae40fb5a0 con 0x7feae40f51a0 2023-12-25T20:10:30.080 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:30.078+0000 7feae3fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feae40f51a0 0x7feae40fd120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:10:30.080 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:30.078+0000 7feae3fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feae40f51a0 0x7feae40fd120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:39076/0 (socket says 172.21.15.160:39076) 2023-12-25T20:10:30.080 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:30.078+0000 7feae3fff700 1 -- 172.21.15.160:0/3697373043 learned_addr learned my addr 172.21.15.160:0/3697373043 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:10:30.080 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:30.078+0000 7feae3fff700 1 -- 172.21.15.160:0/3697373043 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fead4005040 con 0x7feae40f51a0 2023-12-25T20:10:30.080 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:30.078+0000 7feae3fff700 1 --2- 172.21.15.160:0/3697373043 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feae40f51a0 0x7feae40fd120 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7fead4013510 tx=0x7fead400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:10:30.080 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:30.078+0000 7feae0ff9700 1 -- 172.21.15.160:0/3697373043 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fead4009320 con 0x7feae40f51a0 2023-12-25T20:10:30.080 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:30.078+0000 7feae9485700 1 -- 172.21.15.160:0/3697373043 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feae40fb740 con 0x7feae40f51a0 2023-12-25T20:10:30.081 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:30.078+0000 7feae0ff9700 1 -- 172.21.15.160:0/3697373043 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fead4013f00 con 0x7feae40f51a0 2023-12-25T20:10:30.081 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:30.079+0000 7feae0ff9700 1 -- 172.21.15.160:0/3697373043 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fead4014dc0 con 0x7feae40f51a0 2023-12-25T20:10:30.081 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:30.079+0000 7feae9485700 1 -- 172.21.15.160:0/3697373043 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feae40fbbd0 con 0x7feae40f51a0 2023-12-25T20:10:30.081 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:30.079+0000 7feae0ff9700 1 -- 172.21.15.160:0/3697373043 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fead4014420 con 0x7feae40f51a0 2023-12-25T20:10:30.082 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:30.079+0000 7feae9485700 1 -- 172.21.15.160:0/3697373043 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feae4059100 con 0x7feae40f51a0 2023-12-25T20:10:30.082 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:30.079+0000 7feae0ff9700 1 --2- 172.21.15.160:0/3697373043 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7feacc02ff60 0x7feacc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:10:30.082 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:30.080+0000 7feae0ff9700 1 -- 172.21.15.160:0/3697373043 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fead4041030 con 0x7feae40f51a0 2023-12-25T20:10:30.082 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:30.080+0000 7feae37fe700 1 --2- 172.21.15.160:0/3697373043 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7feacc02ff60 0x7feacc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:10:30.082 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:30.080+0000 7feae37fe700 1 --2- 172.21.15.160:0/3697373043 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7feacc02ff60 0x7feacc032420 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7fead80096a0 tx=0x7fead8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:10:30.085 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:30.084+0000 7feae0ff9700 1 -- 172.21.15.160:0/3697373043 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fead40095f0 con 0x7feae40f51a0 2023-12-25T20:10:30.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:10:29 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:10:28.652218+0000 mgr.smithi050.nbzhun (mgr.14162) 98 : cluster [DBG] pgmap v91: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:10:30.288 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:30.287+0000 7feae9485700 1 -- 172.21.15.160:0/3697373043 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7feae40fabe0 con 0x7feae40f51a0 2023-12-25T20:10:30.288 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:30.288+0000 7feae0ff9700 1 -- 172.21.15.160:0/3697373043 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fead401cb10 con 0x7feae40f51a0 2023-12-25T20:10:30.289 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:10:30.289 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:10:30.290 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:30.290+0000 7feae9485700 1 -- 172.21.15.160:0/3697373043 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7feacc02ff60 msgr2=0x7feacc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:10:30.290 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:30.290+0000 7feae9485700 1 --2- 172.21.15.160:0/3697373043 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7feacc02ff60 0x7feacc032420 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7fead80096a0 tx=0x7fead8006b40).stop 2023-12-25T20:10:30.291 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:30.290+0000 7feae9485700 1 -- 172.21.15.160:0/3697373043 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feae40f51a0 msgr2=0x7feae40fd120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:10:30.291 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:30.290+0000 7feae9485700 1 --2- 172.21.15.160:0/3697373043 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feae40f51a0 0x7feae40fd120 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7fead4013510 tx=0x7fead400b040).stop 2023-12-25T20:10:30.291 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:30.290+0000 7feae9485700 1 -- 172.21.15.160:0/3697373043 shutdown_connections 2023-12-25T20:10:30.291 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:30.290+0000 7feae9485700 1 --2- 172.21.15.160:0/3697373043 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7feacc02ff60 0x7feacc032420 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:10:30.291 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:30.290+0000 7feae9485700 1 --2- 172.21.15.160:0/3697373043 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feae40f51a0 0x7feae40fd120 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:10:30.291 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:30.290+0000 7feae9485700 1 -- 172.21.15.160:0/3697373043 >> 172.21.15.160:0/3697373043 conn(0x7feae40f09a0 msgr2=0x7feae405fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:10:30.291 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:30.290+0000 7feae9485700 1 -- 172.21.15.160:0/3697373043 shutdown_connections 2023-12-25T20:10:30.291 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:30.290+0000 7feae9485700 1 -- 172.21.15.160:0/3697373043 wait complete. 2023-12-25T20:10:30.291 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:10:31.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:10:30 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:10:30.289515+0000 mon.smithi050 (mon.0) 181 : audit [DBG] from='client.? 172.21.15.160:0/3697373043' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:10:31.703 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:10:31.703 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:10:32.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:10:31 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:10:30.652531+0000 mgr.smithi050.nbzhun (mgr.14162) 99 : cluster [DBG] pgmap v92: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:10:32.972 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:32.970+0000 7fe8cdb1c700 1 -- 172.21.15.160:0/3135317077 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe8c8060280 msgr2=0x7fe8c80f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:10:32.972 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:32.970+0000 7fe8cdb1c700 1 --2- 172.21.15.160:0/3135317077 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe8c8060280 0x7fe8c80f90f0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7fe8b0009fb0 tx=0x7fe8b0009b30).stop 2023-12-25T20:10:32.972 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:32.970+0000 7fe8cdb1c700 1 -- 172.21.15.160:0/3135317077 shutdown_connections 2023-12-25T20:10:32.972 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:32.970+0000 7fe8cdb1c700 1 --2- 172.21.15.160:0/3135317077 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe8c8060280 0x7fe8c80f90f0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:10:32.972 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:32.970+0000 7fe8cdb1c700 1 -- 172.21.15.160:0/3135317077 >> 172.21.15.160:0/3135317077 conn(0x7fe8c80f0b70 msgr2=0x7fe8c80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:10:32.973 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:32.970+0000 7fe8cdb1c700 1 -- 172.21.15.160:0/3135317077 shutdown_connections 2023-12-25T20:10:32.973 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:32.970+0000 7fe8cdb1c700 1 -- 172.21.15.160:0/3135317077 wait complete. 2023-12-25T20:10:32.973 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:32.971+0000 7fe8cdb1c700 1 Processor -- start 2023-12-25T20:10:32.973 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:32.971+0000 7fe8cdb1c700 1 -- start start 2023-12-25T20:10:32.973 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:32.971+0000 7fe8cdb1c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe8c8060280 0x7fe8c81066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:10:32.973 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:32.971+0000 7fe8cdb1c700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8c8106bd0 con 0x7fe8c8060280 2023-12-25T20:10:32.973 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:32.971+0000 7fe8c77fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe8c8060280 0x7fe8c81066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:10:32.973 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:32.971+0000 7fe8c77fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe8c8060280 0x7fe8c81066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:39090/0 (socket says 172.21.15.160:39090) 2023-12-25T20:10:32.973 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:32.971+0000 7fe8c77fe700 1 -- 172.21.15.160:0/274348586 learned_addr learned my addr 172.21.15.160:0/274348586 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:10:32.973 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:32.971+0000 7fe8c77fe700 1 -- 172.21.15.160:0/274348586 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8b0005040 con 0x7fe8c8060280 2023-12-25T20:10:32.973 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:32.972+0000 7fe8c77fe700 1 --2- 172.21.15.160:0/274348586 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe8c8060280 0x7fe8c81066e0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7fe8b0013510 tx=0x7fe8b000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:10:32.974 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:32.972+0000 7fe8bffff700 1 -- 172.21.15.160:0/274348586 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe8b0009320 con 0x7fe8c8060280 2023-12-25T20:10:32.974 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:32.972+0000 7fe8cdb1c700 1 -- 172.21.15.160:0/274348586 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8c8106dd0 con 0x7fe8c8060280 2023-12-25T20:10:32.975 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:32.972+0000 7fe8bffff700 1 -- 172.21.15.160:0/274348586 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe8b0013f00 con 0x7fe8c8060280 2023-12-25T20:10:32.975 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:32.972+0000 7fe8bffff700 1 -- 172.21.15.160:0/274348586 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe8b0014dc0 con 0x7fe8c8060280 2023-12-25T20:10:32.975 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:32.972+0000 7fe8cdb1c700 1 -- 172.21.15.160:0/274348586 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8c8107260 con 0x7fe8c8060280 2023-12-25T20:10:32.975 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:32.973+0000 7fe8bffff700 1 -- 172.21.15.160:0/274348586 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe8b00095f0 con 0x7fe8c8060280 2023-12-25T20:10:32.975 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:32.973+0000 7fe8cdb1c700 1 -- 172.21.15.160:0/274348586 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8c8045ad0 con 0x7fe8c8060280 2023-12-25T20:10:32.975 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:32.973+0000 7fe8bffff700 1 --2- 172.21.15.160:0/274348586 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fe8b402ffb0 0x7fe8b4032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:10:32.975 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:32.973+0000 7fe8bffff700 1 -- 172.21.15.160:0/274348586 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe8b00410a0 con 0x7fe8c8060280 2023-12-25T20:10:32.975 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:32.973+0000 7fe8c6ffd700 1 --2- 172.21.15.160:0/274348586 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fe8b402ffb0 0x7fe8b4032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:10:32.975 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:32.973+0000 7fe8c6ffd700 1 --2- 172.21.15.160:0/274348586 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fe8b402ffb0 0x7fe8b4032470 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7fe8b80096a0 tx=0x7fe8b8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:10:32.978 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:32.977+0000 7fe8bffff700 1 -- 172.21.15.160:0/274348586 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe8b006d2b0 con 0x7fe8c8060280 2023-12-25T20:10:33.183 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:33.182+0000 7fe8cdb1c700 1 -- 172.21.15.160:0/274348586 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe8c8108110 con 0x7fe8c8060280 2023-12-25T20:10:33.183 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:33.183+0000 7fe8bffff700 1 -- 172.21.15.160:0/274348586 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe8b0014420 con 0x7fe8c8060280 2023-12-25T20:10:33.184 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:10:33.184 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:10:33.185 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:33.184+0000 7fe8cdb1c700 1 -- 172.21.15.160:0/274348586 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fe8b402ffb0 msgr2=0x7fe8b4032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:10:33.185 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:33.184+0000 7fe8cdb1c700 1 --2- 172.21.15.160:0/274348586 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fe8b402ffb0 0x7fe8b4032470 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7fe8b80096a0 tx=0x7fe8b8006b40).stop 2023-12-25T20:10:33.185 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:33.184+0000 7fe8cdb1c700 1 -- 172.21.15.160:0/274348586 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe8c8060280 msgr2=0x7fe8c81066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:10:33.185 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:33.184+0000 7fe8cdb1c700 1 --2- 172.21.15.160:0/274348586 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe8c8060280 0x7fe8c81066e0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7fe8b0013510 tx=0x7fe8b000b040).stop 2023-12-25T20:10:33.186 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:33.184+0000 7fe8cdb1c700 1 -- 172.21.15.160:0/274348586 shutdown_connections 2023-12-25T20:10:33.186 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:33.184+0000 7fe8cdb1c700 1 --2- 172.21.15.160:0/274348586 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fe8b402ffb0 0x7fe8b4032470 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:10:33.186 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:33.184+0000 7fe8cdb1c700 1 --2- 172.21.15.160:0/274348586 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe8c8060280 0x7fe8c81066e0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:10:33.186 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:33.184+0000 7fe8cdb1c700 1 -- 172.21.15.160:0/274348586 >> 172.21.15.160:0/274348586 conn(0x7fe8c80f0b70 msgr2=0x7fe8c80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:10:33.186 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:33.184+0000 7fe8cdb1c700 1 -- 172.21.15.160:0/274348586 shutdown_connections 2023-12-25T20:10:33.186 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:33.184+0000 7fe8cdb1c700 1 -- 172.21.15.160:0/274348586 wait complete. 2023-12-25T20:10:33.186 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:10:34.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:10:33 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:10:32.652859+0000 mgr.smithi050.nbzhun (mgr.14162) 100 : cluster [DBG] pgmap v93: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:10:34.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:10:33 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:10:33.184524+0000 mon.smithi050 (mon.0) 182 : audit [DBG] from='client.? 172.21.15.160:0/274348586' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:10:34.604 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:10:34.604 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:10:35.847 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:35.843+0000 7f58420f9700 1 -- 172.21.15.160:0/747614195 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f583c060220 msgr2=0x7f583c0f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:10:35.847 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:35.843+0000 7f58420f9700 1 --2- 172.21.15.160:0/747614195 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f583c060220 0x7f583c0f9150 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7f582c009960 tx=0x7f582c004d10).stop 2023-12-25T20:10:35.848 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:35.843+0000 7f58420f9700 1 -- 172.21.15.160:0/747614195 shutdown_connections 2023-12-25T20:10:35.848 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:35.843+0000 7f58420f9700 1 --2- 172.21.15.160:0/747614195 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f583c060220 0x7f583c0f9150 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:10:35.848 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:35.843+0000 7f58420f9700 1 -- 172.21.15.160:0/747614195 >> 172.21.15.160:0/747614195 conn(0x7f583c0f0b70 msgr2=0x7f583c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:10:35.848 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:35.843+0000 7f58420f9700 1 -- 172.21.15.160:0/747614195 shutdown_connections 2023-12-25T20:10:35.848 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:35.843+0000 7f58420f9700 1 -- 172.21.15.160:0/747614195 wait complete. 2023-12-25T20:10:35.848 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:35.844+0000 7f58420f9700 1 Processor -- start 2023-12-25T20:10:35.848 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:35.844+0000 7f58420f9700 1 -- start start 2023-12-25T20:10:35.848 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:35.844+0000 7f58420f9700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f583c060220 0x7f583c0f6bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:10:35.848 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:35.844+0000 7f58420f9700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f583c0f5050 con 0x7f583c060220 2023-12-25T20:10:35.848 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:35.844+0000 7f583b7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f583c060220 0x7f583c0f6bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:10:35.848 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:35.844+0000 7f583b7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f583c060220 0x7f583c0f6bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:39104/0 (socket says 172.21.15.160:39104) 2023-12-25T20:10:35.848 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:35.844+0000 7f583b7fe700 1 -- 172.21.15.160:0/1863341954 learned_addr learned my addr 172.21.15.160:0/1863341954 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:10:35.848 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:35.845+0000 7f583b7fe700 1 -- 172.21.15.160:0/1863341954 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f582c009640 con 0x7f583c060220 2023-12-25T20:10:35.848 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:35.845+0000 7f583b7fe700 1 --2- 172.21.15.160:0/1863341954 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f583c060220 0x7f583c0f6bd0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7f582c007bd0 tx=0x7f582c004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:10:35.849 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:35.845+0000 7f5833fff700 1 -- 172.21.15.160:0/1863341954 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f582c018070 con 0x7f583c060220 2023-12-25T20:10:35.849 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:35.845+0000 7f5833fff700 1 -- 172.21.15.160:0/1863341954 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f582c00b870 con 0x7f583c060220 2023-12-25T20:10:35.849 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:35.845+0000 7f58420f9700 1 -- 172.21.15.160:0/1863341954 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f583c0f51f0 con 0x7f583c060220 2023-12-25T20:10:35.849 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:35.845+0000 7f5833fff700 1 -- 172.21.15.160:0/1863341954 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f582c01d420 con 0x7f583c060220 2023-12-25T20:10:35.849 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:35.845+0000 7f58420f9700 1 -- 172.21.15.160:0/1863341954 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f583c0f5680 con 0x7f583c060220 2023-12-25T20:10:35.849 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:35.846+0000 7f5833fff700 1 -- 172.21.15.160:0/1863341954 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f582c024060 con 0x7f583c060220 2023-12-25T20:10:35.849 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:35.846+0000 7f58420f9700 1 -- 172.21.15.160:0/1863341954 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f583c059100 con 0x7f583c060220 2023-12-25T20:10:35.849 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:35.846+0000 7f5833fff700 1 --2- 172.21.15.160:0/1863341954 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f582802ff60 0x7f5828032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:10:35.849 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:35.846+0000 7f5833fff700 1 -- 172.21.15.160:0/1863341954 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f582c041230 con 0x7f583c060220 2023-12-25T20:10:35.849 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:35.846+0000 7f583affd700 1 --2- 172.21.15.160:0/1863341954 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f582802ff60 0x7f5828032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:10:35.849 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:35.847+0000 7f583affd700 1 --2- 172.21.15.160:0/1863341954 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f582802ff60 0x7f5828032420 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f58240096a0 tx=0x7f5824006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:10:35.852 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:35.851+0000 7f5833fff700 1 -- 172.21.15.160:0/1863341954 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f582c025520 con 0x7f583c060220 2023-12-25T20:10:36.059 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:36.058+0000 7f58420f9700 1 -- 172.21.15.160:0/1863341954 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f583c045ad0 con 0x7f583c060220 2023-12-25T20:10:36.059 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:36.059+0000 7f5833fff700 1 -- 172.21.15.160:0/1863341954 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f582c025340 con 0x7f583c060220 2023-12-25T20:10:36.060 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:10:36.060 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:10:36.061 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:36.060+0000 7f58420f9700 1 -- 172.21.15.160:0/1863341954 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f582802ff60 msgr2=0x7f5828032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:10:36.061 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:36.060+0000 7f58420f9700 1 --2- 172.21.15.160:0/1863341954 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f582802ff60 0x7f5828032420 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f58240096a0 tx=0x7f5824006b40).stop 2023-12-25T20:10:36.061 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:36.060+0000 7f58420f9700 1 -- 172.21.15.160:0/1863341954 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f583c060220 msgr2=0x7f583c0f6bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:10:36.061 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:36.060+0000 7f58420f9700 1 --2- 172.21.15.160:0/1863341954 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f583c060220 0x7f583c0f6bd0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7f582c007bd0 tx=0x7f582c004d10).stop 2023-12-25T20:10:36.061 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:36.060+0000 7f58420f9700 1 -- 172.21.15.160:0/1863341954 shutdown_connections 2023-12-25T20:10:36.061 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:36.060+0000 7f58420f9700 1 --2- 172.21.15.160:0/1863341954 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f582802ff60 0x7f5828032420 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:10:36.062 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:36.060+0000 7f58420f9700 1 --2- 172.21.15.160:0/1863341954 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f583c060220 0x7f583c0f6bd0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:10:36.062 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:36.060+0000 7f58420f9700 1 -- 172.21.15.160:0/1863341954 >> 172.21.15.160:0/1863341954 conn(0x7f583c0f0b70 msgr2=0x7f583c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:10:36.062 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:36.060+0000 7f58420f9700 1 -- 172.21.15.160:0/1863341954 shutdown_connections 2023-12-25T20:10:36.062 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:36.060+0000 7f58420f9700 1 -- 172.21.15.160:0/1863341954 wait complete. 2023-12-25T20:10:36.062 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:10:36.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:10:35 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:10:34.653192+0000 mgr.smithi050.nbzhun (mgr.14162) 101 : cluster [DBG] pgmap v94: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:10:37.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:10:36 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:10:36.060606+0000 mon.smithi050 (mon.0) 183 : audit [DBG] from='client.? 172.21.15.160:0/1863341954' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:10:37.489 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:10:37.489 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:10:38.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:10:37 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:10:36.653488+0000 mgr.smithi050.nbzhun (mgr.14162) 102 : cluster [DBG] pgmap v95: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:10:38.701 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:38.700+0000 7efd888ed700 1 -- 172.21.15.160:0/1711353991 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efd840f51a0 msgr2=0x7efd840f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:10:38.701 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:38.700+0000 7efd888ed700 1 --2- 172.21.15.160:0/1711353991 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efd840f51a0 0x7efd840f55c0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7efd74009fb0 tx=0x7efd74009b30).stop 2023-12-25T20:10:38.701 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:38.700+0000 7efd888ed700 1 -- 172.21.15.160:0/1711353991 shutdown_connections 2023-12-25T20:10:38.702 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:38.700+0000 7efd888ed700 1 --2- 172.21.15.160:0/1711353991 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efd840f51a0 0x7efd840f55c0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:10:38.702 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:38.700+0000 7efd888ed700 1 -- 172.21.15.160:0/1711353991 >> 172.21.15.160:0/1711353991 conn(0x7efd840f09a0 msgr2=0x7efd840f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:10:38.702 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:38.700+0000 7efd888ed700 1 -- 172.21.15.160:0/1711353991 shutdown_connections 2023-12-25T20:10:38.702 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:38.700+0000 7efd888ed700 1 -- 172.21.15.160:0/1711353991 wait complete. 2023-12-25T20:10:38.702 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:38.700+0000 7efd888ed700 1 Processor -- start 2023-12-25T20:10:38.702 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:38.701+0000 7efd888ed700 1 -- start start 2023-12-25T20:10:38.702 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:38.701+0000 7efd888ed700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efd840fd0e0 0x7efd840fb5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:10:38.702 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:38.701+0000 7efd888ed700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd840fd500 con 0x7efd840fd0e0 2023-12-25T20:10:38.704 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:38.701+0000 7efd837fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efd840fd0e0 0x7efd840fb5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:10:38.704 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:38.701+0000 7efd837fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efd840fd0e0 0x7efd840fb5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:49490/0 (socket says 172.21.15.160:49490) 2023-12-25T20:10:38.704 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:38.701+0000 7efd837fe700 1 -- 172.21.15.160:0/2583976399 learned_addr learned my addr 172.21.15.160:0/2583976399 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:10:38.704 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:38.701+0000 7efd837fe700 1 -- 172.21.15.160:0/2583976399 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd74005040 con 0x7efd840fd0e0 2023-12-25T20:10:38.704 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:38.701+0000 7efd837fe700 1 --2- 172.21.15.160:0/2583976399 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efd840fd0e0 0x7efd840fb5a0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7efd74009f80 tx=0x7efd74009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:10:38.704 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:38.702+0000 7efd6bfff700 1 -- 172.21.15.160:0/2583976399 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efd740136c0 con 0x7efd840fd0e0 2023-12-25T20:10:38.704 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:38.702+0000 7efd6bfff700 1 -- 172.21.15.160:0/2583976399 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efd74013e70 con 0x7efd840fd0e0 2023-12-25T20:10:38.704 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:38.702+0000 7efd888ed700 1 -- 172.21.15.160:0/2583976399 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd840fba90 con 0x7efd840fd0e0 2023-12-25T20:10:38.704 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:38.702+0000 7efd6bfff700 1 -- 172.21.15.160:0/2583976399 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efd74014420 con 0x7efd840fd0e0 2023-12-25T20:10:38.704 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:38.702+0000 7efd888ed700 1 -- 172.21.15.160:0/2583976399 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd840fbf80 con 0x7efd840fd0e0 2023-12-25T20:10:38.704 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:38.703+0000 7efd6bfff700 1 -- 172.21.15.160:0/2583976399 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7efd74025060 con 0x7efd840fd0e0 2023-12-25T20:10:38.705 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:38.703+0000 7efd888ed700 1 -- 172.21.15.160:0/2583976399 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd84045ad0 con 0x7efd840fd0e0 2023-12-25T20:10:38.705 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:38.703+0000 7efd6bfff700 1 --2- 172.21.15.160:0/2583976399 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7efd6c02ffb0 0x7efd6c032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:10:38.705 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:38.703+0000 7efd6bfff700 1 -- 172.21.15.160:0/2583976399 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7efd74040f00 con 0x7efd840fd0e0 2023-12-25T20:10:38.705 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:38.703+0000 7efd82ffd700 1 --2- 172.21.15.160:0/2583976399 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7efd6c02ffb0 0x7efd6c032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:10:38.705 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:38.703+0000 7efd82ffd700 1 --2- 172.21.15.160:0/2583976399 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7efd6c02ffb0 0x7efd6c032470 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7efd780096a0 tx=0x7efd78006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:10:38.708 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:38.708+0000 7efd6bfff700 1 -- 172.21.15.160:0/2583976399 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7efd7401cb60 con 0x7efd840fd0e0 2023-12-25T20:10:38.914 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:38.913+0000 7efd888ed700 1 -- 172.21.15.160:0/2583976399 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efd8410c800 con 0x7efd840fd0e0 2023-12-25T20:10:38.914 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:38.914+0000 7efd6bfff700 1 -- 172.21.15.160:0/2583976399 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7efd7402d2d0 con 0x7efd840fd0e0 2023-12-25T20:10:38.915 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:10:38.915 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:10:38.916 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:38.915+0000 7efd888ed700 1 -- 172.21.15.160:0/2583976399 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7efd6c02ffb0 msgr2=0x7efd6c032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:10:38.917 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:38.915+0000 7efd888ed700 1 --2- 172.21.15.160:0/2583976399 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7efd6c02ffb0 0x7efd6c032470 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7efd780096a0 tx=0x7efd78006b40).stop 2023-12-25T20:10:38.917 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:38.915+0000 7efd888ed700 1 -- 172.21.15.160:0/2583976399 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efd840fd0e0 msgr2=0x7efd840fb5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:10:38.917 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:38.915+0000 7efd888ed700 1 --2- 172.21.15.160:0/2583976399 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efd840fd0e0 0x7efd840fb5a0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7efd74009f80 tx=0x7efd74009b30).stop 2023-12-25T20:10:38.917 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:38.915+0000 7efd888ed700 1 -- 172.21.15.160:0/2583976399 shutdown_connections 2023-12-25T20:10:38.917 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:38.915+0000 7efd888ed700 1 --2- 172.21.15.160:0/2583976399 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7efd6c02ffb0 0x7efd6c032470 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:10:38.917 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:38.915+0000 7efd888ed700 1 --2- 172.21.15.160:0/2583976399 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efd840fd0e0 0x7efd840fb5a0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:10:38.917 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:38.915+0000 7efd888ed700 1 -- 172.21.15.160:0/2583976399 >> 172.21.15.160:0/2583976399 conn(0x7efd840f09a0 msgr2=0x7efd8405fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:10:38.917 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:38.915+0000 7efd888ed700 1 -- 172.21.15.160:0/2583976399 shutdown_connections 2023-12-25T20:10:38.917 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:38.915+0000 7efd888ed700 1 -- 172.21.15.160:0/2583976399 wait complete. 2023-12-25T20:10:38.918 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:10:40.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:10:39 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:10:38.653811+0000 mgr.smithi050.nbzhun (mgr.14162) 103 : cluster [DBG] pgmap v96: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:10:40.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:10:39 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:10:38.915268+0000 mon.smithi050 (mon.0) 184 : audit [DBG] from='client.? 172.21.15.160:0/2583976399' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:10:40.327 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:10:40.327 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:10:41.525 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:41.522+0000 7f6074ec0700 1 -- 172.21.15.160:0/546501181 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f60700f5410 msgr2=0x7f60700f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:10:41.525 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:41.522+0000 7f6074ec0700 1 --2- 172.21.15.160:0/546501181 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f60700f5410 0x7f60700f5830 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7f6064009fb0 tx=0x7f6064009b30).stop 2023-12-25T20:10:41.525 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:41.522+0000 7f6074ec0700 1 -- 172.21.15.160:0/546501181 shutdown_connections 2023-12-25T20:10:41.525 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:41.522+0000 7f6074ec0700 1 --2- 172.21.15.160:0/546501181 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f60700f5410 0x7f60700f5830 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:10:41.525 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:41.522+0000 7f6074ec0700 1 -- 172.21.15.160:0/546501181 >> 172.21.15.160:0/546501181 conn(0x7f60700f0b70 msgr2=0x7f60700f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:10:41.526 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:41.522+0000 7f6074ec0700 1 -- 172.21.15.160:0/546501181 shutdown_connections 2023-12-25T20:10:41.526 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:41.522+0000 7f6074ec0700 1 -- 172.21.15.160:0/546501181 wait complete. 2023-12-25T20:10:41.526 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:41.523+0000 7f6074ec0700 1 Processor -- start 2023-12-25T20:10:41.526 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:41.523+0000 7f6074ec0700 1 -- start start 2023-12-25T20:10:41.526 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:41.523+0000 7f6074ec0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f60700f5410 0x7f6070109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:10:41.526 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:41.523+0000 7f6074ec0700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6070060240 con 0x7f60700f5410 2023-12-25T20:10:41.526 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:41.523+0000 7f606e59c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f60700f5410 0x7f6070109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:10:41.526 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:41.523+0000 7f606e59c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f60700f5410 0x7f6070109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:49502/0 (socket says 172.21.15.160:49502) 2023-12-25T20:10:41.526 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:41.523+0000 7f606e59c700 1 -- 172.21.15.160:0/1417841543 learned_addr learned my addr 172.21.15.160:0/1417841543 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:10:41.526 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:41.524+0000 7f606e59c700 1 -- 172.21.15.160:0/1417841543 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6064005040 con 0x7f60700f5410 2023-12-25T20:10:41.526 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:41.524+0000 7f606e59c700 1 --2- 172.21.15.160:0/1417841543 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f60700f5410 0x7f6070109430 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7f6064014000 tx=0x7f606400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:10:41.526 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:41.524+0000 7f605effd700 1 -- 172.21.15.160:0/1417841543 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f60640095f0 con 0x7f60700f5410 2023-12-25T20:10:41.526 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:41.524+0000 7f6074ec0700 1 -- 172.21.15.160:0/1417841543 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6070060440 con 0x7f60700f5410 2023-12-25T20:10:41.526 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:41.524+0000 7f605effd700 1 -- 172.21.15.160:0/1417841543 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6064007e70 con 0x7f60700f5410 2023-12-25T20:10:41.527 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:41.524+0000 7f605effd700 1 -- 172.21.15.160:0/1417841543 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6064014ea0 con 0x7f60700f5410 2023-12-25T20:10:41.527 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:41.524+0000 7f6074ec0700 1 -- 172.21.15.160:0/1417841543 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6070109ae0 con 0x7f60700f5410 2023-12-25T20:10:41.528 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:41.525+0000 7f605effd700 1 -- 172.21.15.160:0/1417841543 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6064025070 con 0x7f60700f5410 2023-12-25T20:10:41.528 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:41.525+0000 7f6074ec0700 1 -- 172.21.15.160:0/1417841543 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6070045ad0 con 0x7f60700f5410 2023-12-25T20:10:41.528 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:41.525+0000 7f605effd700 1 --2- 172.21.15.160:0/1417841543 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f605802ff60 0x7f6058032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:10:41.528 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:41.525+0000 7f605effd700 1 -- 172.21.15.160:0/1417841543 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6064040a30 con 0x7f60700f5410 2023-12-25T20:10:41.528 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:41.525+0000 7f606dd9b700 1 --2- 172.21.15.160:0/1417841543 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f605802ff60 0x7f6058032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:10:41.528 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:41.526+0000 7f606dd9b700 1 --2- 172.21.15.160:0/1417841543 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f605802ff60 0x7f6058032420 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f60600096a0 tx=0x7f6060006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:10:41.530 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:41.530+0000 7f605effd700 1 -- 172.21.15.160:0/1417841543 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f60640079c0 con 0x7f60700f5410 2023-12-25T20:10:41.736 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:41.735+0000 7f6074ec0700 1 -- 172.21.15.160:0/1417841543 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f60700fae60 con 0x7f60700f5410 2023-12-25T20:10:41.736 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:41.736+0000 7f605effd700 1 -- 172.21.15.160:0/1417841543 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f606400f300 con 0x7f60700f5410 2023-12-25T20:10:41.736 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:10:41.737 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:10:41.738 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:41.737+0000 7f6074ec0700 1 -- 172.21.15.160:0/1417841543 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f605802ff60 msgr2=0x7f6058032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:10:41.738 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:41.737+0000 7f6074ec0700 1 --2- 172.21.15.160:0/1417841543 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f605802ff60 0x7f6058032420 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f60600096a0 tx=0x7f6060006b40).stop 2023-12-25T20:10:41.738 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:41.737+0000 7f6074ec0700 1 -- 172.21.15.160:0/1417841543 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f60700f5410 msgr2=0x7f6070109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:10:41.738 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:41.737+0000 7f6074ec0700 1 --2- 172.21.15.160:0/1417841543 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f60700f5410 0x7f6070109430 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7f6064014000 tx=0x7f606400b040).stop 2023-12-25T20:10:41.738 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:41.737+0000 7f6074ec0700 1 -- 172.21.15.160:0/1417841543 shutdown_connections 2023-12-25T20:10:41.738 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:41.737+0000 7f6074ec0700 1 --2- 172.21.15.160:0/1417841543 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f605802ff60 0x7f6058032420 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:10:41.738 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:41.737+0000 7f6074ec0700 1 --2- 172.21.15.160:0/1417841543 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f60700f5410 0x7f6070109430 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:10:41.738 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:41.737+0000 7f6074ec0700 1 -- 172.21.15.160:0/1417841543 >> 172.21.15.160:0/1417841543 conn(0x7f60700f0b70 msgr2=0x7f60700f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:10:41.739 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:41.737+0000 7f6074ec0700 1 -- 172.21.15.160:0/1417841543 shutdown_connections 2023-12-25T20:10:41.739 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:41.737+0000 7f6074ec0700 1 -- 172.21.15.160:0/1417841543 wait complete. 2023-12-25T20:10:41.739 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:10:42.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:10:41 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:10:40.654130+0000 mgr.smithi050.nbzhun (mgr.14162) 104 : cluster [DBG] pgmap v97: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:10:42.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:10:41 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:10:41.737340+0000 mon.smithi050 (mon.0) 185 : audit [DBG] from='client.? 172.21.15.160:0/1417841543' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:10:43.156 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:10:43.156 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:10:44.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:10:43 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:10:42.654423+0000 mgr.smithi050.nbzhun (mgr.14162) 105 : cluster [DBG] pgmap v98: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:10:44.403 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:44.399+0000 7f3c2474b700 1 -- 172.21.15.160:0/700559085 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3c1c0f6510 msgr2=0x7f3c1c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:10:44.404 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:44.399+0000 7f3c2474b700 1 --2- 172.21.15.160:0/700559085 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3c1c0f6510 0x7f3c1c0f6930 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7f3c0c009fb0 tx=0x7f3c0c009b30).stop 2023-12-25T20:10:44.404 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:44.399+0000 7f3c2474b700 1 -- 172.21.15.160:0/700559085 shutdown_connections 2023-12-25T20:10:44.404 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:44.399+0000 7f3c2474b700 1 --2- 172.21.15.160:0/700559085 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3c1c0f6510 0x7f3c1c0f6930 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:10:44.404 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:44.399+0000 7f3c2474b700 1 -- 172.21.15.160:0/700559085 >> 172.21.15.160:0/700559085 conn(0x7f3c1c0f1c70 msgr2=0x7f3c1c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:10:44.404 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:44.399+0000 7f3c2474b700 1 -- 172.21.15.160:0/700559085 shutdown_connections 2023-12-25T20:10:44.405 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:44.399+0000 7f3c2474b700 1 -- 172.21.15.160:0/700559085 wait complete. 2023-12-25T20:10:44.405 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:44.400+0000 7f3c2474b700 1 Processor -- start 2023-12-25T20:10:44.405 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:44.400+0000 7f3c2474b700 1 -- start start 2023-12-25T20:10:44.405 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:44.400+0000 7f3c2474b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3c1c0f6510 0x7f3c1c106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:10:44.405 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:44.400+0000 7f3c2474b700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c1c106c20 con 0x7f3c1c0f6510 2023-12-25T20:10:44.405 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:44.400+0000 7f3c224e7700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3c1c0f6510 0x7f3c1c106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:10:44.406 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:44.400+0000 7f3c224e7700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3c1c0f6510 0x7f3c1c106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:49508/0 (socket says 172.21.15.160:49508) 2023-12-25T20:10:44.406 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:44.400+0000 7f3c224e7700 1 -- 172.21.15.160:0/2497187657 learned_addr learned my addr 172.21.15.160:0/2497187657 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:10:44.406 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:44.400+0000 7f3c224e7700 1 -- 172.21.15.160:0/2497187657 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c0c005040 con 0x7f3c1c0f6510 2023-12-25T20:10:44.406 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:44.400+0000 7f3c224e7700 1 --2- 172.21.15.160:0/2497187657 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3c1c0f6510 0x7f3c1c106730 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7f3c0c013510 tx=0x7f3c0c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:10:44.406 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:44.401+0000 7f3c12ffd700 1 -- 172.21.15.160:0/2497187657 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3c0c009320 con 0x7f3c1c0f6510 2023-12-25T20:10:44.406 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:44.401+0000 7f3c2474b700 1 -- 172.21.15.160:0/2497187657 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c1c106e20 con 0x7f3c1c0f6510 2023-12-25T20:10:44.407 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:44.401+0000 7f3c12ffd700 1 -- 172.21.15.160:0/2497187657 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3c0c013f00 con 0x7f3c1c0f6510 2023-12-25T20:10:44.407 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:44.401+0000 7f3c12ffd700 1 -- 172.21.15.160:0/2497187657 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3c0c014d90 con 0x7f3c1c0f6510 2023-12-25T20:10:44.407 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:44.401+0000 7f3c2474b700 1 -- 172.21.15.160:0/2497187657 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c1c1072b0 con 0x7f3c1c0f6510 2023-12-25T20:10:44.408 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:44.401+0000 7f3c12ffd700 1 -- 172.21.15.160:0/2497187657 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3c0c0263a0 con 0x7f3c1c0f6510 2023-12-25T20:10:44.408 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:44.402+0000 7f3c12ffd700 1 --2- 172.21.15.160:0/2497187657 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f3c0802ff60 0x7f3c08032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:10:44.408 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:44.402+0000 7f3c2474b700 1 -- 172.21.15.160:0/2497187657 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c1c045ad0 con 0x7f3c1c0f6510 2023-12-25T20:10:44.408 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:44.402+0000 7f3c12ffd700 1 -- 172.21.15.160:0/2497187657 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3c0c0410f0 con 0x7f3c1c0f6510 2023-12-25T20:10:44.408 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:44.402+0000 7f3c21ce6700 1 --2- 172.21.15.160:0/2497187657 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f3c0802ff60 0x7f3c08032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:10:44.409 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:44.402+0000 7f3c21ce6700 1 --2- 172.21.15.160:0/2497187657 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f3c0802ff60 0x7f3c08032420 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f3c180096a0 tx=0x7f3c18006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:10:44.410 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:44.407+0000 7f3c12ffd700 1 -- 172.21.15.160:0/2497187657 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3c0c012920 con 0x7f3c1c0f6510 2023-12-25T20:10:44.613 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:44.612+0000 7f3c2474b700 1 -- 172.21.15.160:0/2497187657 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3c1c0fc050 con 0x7f3c1c0f6510 2023-12-25T20:10:44.614 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:44.613+0000 7f3c12ffd700 1 -- 172.21.15.160:0/2497187657 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3c0c02b020 con 0x7f3c1c0f6510 2023-12-25T20:10:44.614 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:10:44.614 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:10:44.615 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:44.614+0000 7f3c2474b700 1 -- 172.21.15.160:0/2497187657 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f3c0802ff60 msgr2=0x7f3c08032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:10:44.615 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:44.614+0000 7f3c2474b700 1 --2- 172.21.15.160:0/2497187657 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f3c0802ff60 0x7f3c08032420 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f3c180096a0 tx=0x7f3c18006b40).stop 2023-12-25T20:10:44.615 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:44.614+0000 7f3c2474b700 1 -- 172.21.15.160:0/2497187657 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3c1c0f6510 msgr2=0x7f3c1c106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:10:44.616 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:44.614+0000 7f3c2474b700 1 --2- 172.21.15.160:0/2497187657 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3c1c0f6510 0x7f3c1c106730 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7f3c0c013510 tx=0x7f3c0c00b040).stop 2023-12-25T20:10:44.616 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:44.614+0000 7f3c2474b700 1 -- 172.21.15.160:0/2497187657 shutdown_connections 2023-12-25T20:10:44.616 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:44.614+0000 7f3c2474b700 1 --2- 172.21.15.160:0/2497187657 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f3c0802ff60 0x7f3c08032420 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:10:44.616 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:44.614+0000 7f3c2474b700 1 --2- 172.21.15.160:0/2497187657 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3c1c0f6510 0x7f3c1c106730 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:10:44.616 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:44.614+0000 7f3c2474b700 1 -- 172.21.15.160:0/2497187657 >> 172.21.15.160:0/2497187657 conn(0x7f3c1c0f1c70 msgr2=0x7f3c1c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:10:44.616 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:44.615+0000 7f3c2474b700 1 -- 172.21.15.160:0/2497187657 shutdown_connections 2023-12-25T20:10:44.616 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:44.615+0000 7f3c2474b700 1 -- 172.21.15.160:0/2497187657 wait complete. 2023-12-25T20:10:44.616 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:10:45.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:10:44 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:10:44.614618+0000 mon.smithi050 (mon.0) 186 : audit [DBG] from='client.? 172.21.15.160:0/2497187657' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:10:46.035 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:10:46.035 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:10:46.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:10:45 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:10:44.654695+0000 mgr.smithi050.nbzhun (mgr.14162) 106 : cluster [DBG] pgmap v99: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:10:47.271 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:47.270+0000 7fc5a2fec700 1 -- 172.21.15.160:0/180085710 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc59c0f5410 msgr2=0x7fc59c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:10:47.271 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:47.270+0000 7fc5a2fec700 1 --2- 172.21.15.160:0/180085710 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc59c0f5410 0x7fc59c0f5830 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0x7fc590009fb0 tx=0x7fc590009b30).stop 2023-12-25T20:10:47.271 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:47.270+0000 7fc5a2fec700 1 -- 172.21.15.160:0/180085710 shutdown_connections 2023-12-25T20:10:47.271 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:47.270+0000 7fc5a2fec700 1 --2- 172.21.15.160:0/180085710 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc59c0f5410 0x7fc59c0f5830 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:10:47.271 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:47.270+0000 7fc5a2fec700 1 -- 172.21.15.160:0/180085710 >> 172.21.15.160:0/180085710 conn(0x7fc59c0f0b70 msgr2=0x7fc59c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:10:47.272 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:47.270+0000 7fc5a2fec700 1 -- 172.21.15.160:0/180085710 shutdown_connections 2023-12-25T20:10:47.272 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:47.270+0000 7fc5a2fec700 1 -- 172.21.15.160:0/180085710 wait complete. 2023-12-25T20:10:47.274 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:47.271+0000 7fc5a2fec700 1 Processor -- start 2023-12-25T20:10:47.274 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:47.271+0000 7fc5a2fec700 1 -- start start 2023-12-25T20:10:47.274 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:47.271+0000 7fc5a2fec700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc59c0f5410 0x7fc59c105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:10:47.274 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:47.271+0000 7fc5a2fec700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc59c105b20 con 0x7fc59c0f5410 2023-12-25T20:10:47.274 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:47.271+0000 7fc5a0d88700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc59c0f5410 0x7fc59c105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:10:47.274 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:47.271+0000 7fc5a0d88700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc59c0f5410 0x7fc59c105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:49512/0 (socket says 172.21.15.160:49512) 2023-12-25T20:10:47.274 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:47.271+0000 7fc5a0d88700 1 -- 172.21.15.160:0/893483494 learned_addr learned my addr 172.21.15.160:0/893483494 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:10:47.274 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:47.272+0000 7fc5a0d88700 1 -- 172.21.15.160:0/893483494 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc590005040 con 0x7fc59c0f5410 2023-12-25T20:10:47.275 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:47.272+0000 7fc5a0d88700 1 --2- 172.21.15.160:0/893483494 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc59c0f5410 0x7fc59c105630 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7fc590013510 tx=0x7fc59000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:10:47.275 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:47.272+0000 7fc5997fa700 1 -- 172.21.15.160:0/893483494 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc590009320 con 0x7fc59c0f5410 2023-12-25T20:10:47.275 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:47.272+0000 7fc5a2fec700 1 -- 172.21.15.160:0/893483494 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc59c105d20 con 0x7fc59c0f5410 2023-12-25T20:10:47.275 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:47.272+0000 7fc5997fa700 1 -- 172.21.15.160:0/893483494 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc590013f00 con 0x7fc59c0f5410 2023-12-25T20:10:47.275 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:47.272+0000 7fc5997fa700 1 -- 172.21.15.160:0/893483494 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc590014d90 con 0x7fc59c0f5410 2023-12-25T20:10:47.275 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:47.272+0000 7fc5a2fec700 1 -- 172.21.15.160:0/893483494 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc59c1061b0 con 0x7fc59c0f5410 2023-12-25T20:10:47.275 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:47.273+0000 7fc5997fa700 1 -- 172.21.15.160:0/893483494 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc5900095f0 con 0x7fc59c0f5410 2023-12-25T20:10:47.275 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:47.273+0000 7fc5a2fec700 1 -- 172.21.15.160:0/893483494 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc59c045ad0 con 0x7fc59c0f5410 2023-12-25T20:10:47.275 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:47.273+0000 7fc5997fa700 1 --2- 172.21.15.160:0/893483494 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fc58c02ff60 0x7fc58c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:10:47.275 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:47.273+0000 7fc5997fa700 1 -- 172.21.15.160:0/893483494 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc590040fe0 con 0x7fc59c0f5410 2023-12-25T20:10:47.275 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:47.273+0000 7fc59bfff700 1 --2- 172.21.15.160:0/893483494 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fc58c02ff60 0x7fc58c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:10:47.276 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:47.274+0000 7fc59bfff700 1 --2- 172.21.15.160:0/893483494 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fc58c02ff60 0x7fc58c032420 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7fc5880096a0 tx=0x7fc588006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:10:47.278 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:47.278+0000 7fc5997fa700 1 -- 172.21.15.160:0/893483494 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc59001d7f0 con 0x7fc59c0f5410 2023-12-25T20:10:47.481 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:47.481+0000 7fc5a2fec700 1 -- 172.21.15.160:0/893483494 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc59c0fae60 con 0x7fc59c0f5410 2023-12-25T20:10:47.482 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:47.481+0000 7fc5997fa700 1 -- 172.21.15.160:0/893483494 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc590010230 con 0x7fc59c0f5410 2023-12-25T20:10:47.482 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:10:47.482 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:10:47.484 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:47.483+0000 7fc5a2fec700 1 -- 172.21.15.160:0/893483494 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fc58c02ff60 msgr2=0x7fc58c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:10:47.484 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:47.483+0000 7fc5a2fec700 1 --2- 172.21.15.160:0/893483494 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fc58c02ff60 0x7fc58c032420 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7fc5880096a0 tx=0x7fc588006b40).stop 2023-12-25T20:10:47.484 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:47.483+0000 7fc5a2fec700 1 -- 172.21.15.160:0/893483494 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc59c0f5410 msgr2=0x7fc59c105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:10:47.484 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:47.483+0000 7fc5a2fec700 1 --2- 172.21.15.160:0/893483494 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc59c0f5410 0x7fc59c105630 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7fc590013510 tx=0x7fc59000b040).stop 2023-12-25T20:10:47.484 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:47.483+0000 7fc5a2fec700 1 -- 172.21.15.160:0/893483494 shutdown_connections 2023-12-25T20:10:47.484 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:47.483+0000 7fc5a2fec700 1 --2- 172.21.15.160:0/893483494 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fc58c02ff60 0x7fc58c032420 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:10:47.484 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:47.483+0000 7fc5a2fec700 1 --2- 172.21.15.160:0/893483494 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc59c0f5410 0x7fc59c105630 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:10:47.485 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:47.483+0000 7fc5a2fec700 1 -- 172.21.15.160:0/893483494 >> 172.21.15.160:0/893483494 conn(0x7fc59c0f0b70 msgr2=0x7fc59c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:10:47.485 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:47.483+0000 7fc5a2fec700 1 -- 172.21.15.160:0/893483494 shutdown_connections 2023-12-25T20:10:47.485 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:47.483+0000 7fc5a2fec700 1 -- 172.21.15.160:0/893483494 wait complete. 2023-12-25T20:10:47.485 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:10:48.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:10:47 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:10:46.655030+0000 mgr.smithi050.nbzhun (mgr.14162) 107 : cluster [DBG] pgmap v100: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:10:48.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:10:47 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:10:47.483119+0000 mon.smithi050 (mon.0) 187 : audit [DBG] from='client.? 172.21.15.160:0/893483494' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:10:48.930 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:10:48.931 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:10:50.142 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:50.140+0000 7ff5e107d700 1 -- 172.21.15.160:0/1557629223 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff5dc0f5410 msgr2=0x7ff5dc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:10:50.142 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:50.140+0000 7ff5e107d700 1 --2- 172.21.15.160:0/1557629223 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff5dc0f5410 0x7ff5dc0f5830 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7ff5d0009fb0 tx=0x7ff5d0009b30).stop 2023-12-25T20:10:50.142 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:50.140+0000 7ff5e107d700 1 -- 172.21.15.160:0/1557629223 shutdown_connections 2023-12-25T20:10:50.142 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:50.140+0000 7ff5e107d700 1 --2- 172.21.15.160:0/1557629223 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff5dc0f5410 0x7ff5dc0f5830 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:10:50.142 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:50.140+0000 7ff5e107d700 1 -- 172.21.15.160:0/1557629223 >> 172.21.15.160:0/1557629223 conn(0x7ff5dc0f0b70 msgr2=0x7ff5dc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:10:50.142 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:50.140+0000 7ff5e107d700 1 -- 172.21.15.160:0/1557629223 shutdown_connections 2023-12-25T20:10:50.142 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:50.140+0000 7ff5e107d700 1 -- 172.21.15.160:0/1557629223 wait complete. 2023-12-25T20:10:50.143 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:50.141+0000 7ff5e107d700 1 Processor -- start 2023-12-25T20:10:50.143 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:50.141+0000 7ff5e107d700 1 -- start start 2023-12-25T20:10:50.143 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:50.141+0000 7ff5e107d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff5dc0f5410 0x7ff5dc109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:10:50.143 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:50.141+0000 7ff5e107d700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5dc060220 con 0x7ff5dc0f5410 2023-12-25T20:10:50.143 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:50.141+0000 7ff5dad9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff5dc0f5410 0x7ff5dc109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:10:50.143 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:50.141+0000 7ff5dad9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff5dc0f5410 0x7ff5dc109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:38628/0 (socket says 172.21.15.160:38628) 2023-12-25T20:10:50.143 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:50.141+0000 7ff5dad9d700 1 -- 172.21.15.160:0/2114496581 learned_addr learned my addr 172.21.15.160:0/2114496581 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:10:50.144 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:50.142+0000 7ff5dad9d700 1 -- 172.21.15.160:0/2114496581 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5d0005040 con 0x7ff5dc0f5410 2023-12-25T20:10:50.144 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:50.142+0000 7ff5dad9d700 1 --2- 172.21.15.160:0/2114496581 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff5dc0f5410 0x7ff5dc109450 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7ff5d0013510 tx=0x7ff5d000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:10:50.145 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:50.142+0000 7ff5c37fe700 1 -- 172.21.15.160:0/2114496581 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff5d0009320 con 0x7ff5dc0f5410 2023-12-25T20:10:50.145 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:50.142+0000 7ff5e107d700 1 -- 172.21.15.160:0/2114496581 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5dc060420 con 0x7ff5dc0f5410 2023-12-25T20:10:50.145 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:50.142+0000 7ff5c37fe700 1 -- 172.21.15.160:0/2114496581 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff5d0013f00 con 0x7ff5dc0f5410 2023-12-25T20:10:50.145 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:50.142+0000 7ff5c37fe700 1 -- 172.21.15.160:0/2114496581 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff5d0014d90 con 0x7ff5dc0f5410 2023-12-25T20:10:50.145 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:50.142+0000 7ff5e107d700 1 -- 172.21.15.160:0/2114496581 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5dc109b00 con 0x7ff5dc0f5410 2023-12-25T20:10:50.145 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:50.143+0000 7ff5c37fe700 1 -- 172.21.15.160:0/2114496581 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff5d00095f0 con 0x7ff5dc0f5410 2023-12-25T20:10:50.145 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:50.143+0000 7ff5e107d700 1 -- 172.21.15.160:0/2114496581 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5dc059100 con 0x7ff5dc0f5410 2023-12-25T20:10:50.145 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:50.143+0000 7ff5c37fe700 1 --2- 172.21.15.160:0/2114496581 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7ff5c402ff60 0x7ff5c4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:10:50.145 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:50.143+0000 7ff5c37fe700 1 -- 172.21.15.160:0/2114496581 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff5d0041080 con 0x7ff5dc0f5410 2023-12-25T20:10:50.145 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:50.143+0000 7ff5da59c700 1 --2- 172.21.15.160:0/2114496581 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7ff5c402ff60 0x7ff5c4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:10:50.145 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:50.144+0000 7ff5da59c700 1 --2- 172.21.15.160:0/2114496581 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7ff5c402ff60 0x7ff5c4032420 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7ff5cc0096a0 tx=0x7ff5cc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:10:50.149 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:50.148+0000 7ff5c37fe700 1 -- 172.21.15.160:0/2114496581 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff5d0014420 con 0x7ff5dc0f5410 2023-12-25T20:10:50.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:10:49 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:10:48.655386+0000 mgr.smithi050.nbzhun (mgr.14162) 108 : cluster [DBG] pgmap v101: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:10:50.351 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:50.351+0000 7ff5e107d700 1 -- 172.21.15.160:0/2114496581 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff5dc045ad0 con 0x7ff5dc0f5410 2023-12-25T20:10:50.352 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:50.351+0000 7ff5c37fe700 1 -- 172.21.15.160:0/2114496581 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff5d001d7f0 con 0x7ff5dc0f5410 2023-12-25T20:10:50.352 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:10:50.352 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:10:50.354 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:50.353+0000 7ff5e107d700 1 -- 172.21.15.160:0/2114496581 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7ff5c402ff60 msgr2=0x7ff5c4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:10:50.354 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:50.353+0000 7ff5e107d700 1 --2- 172.21.15.160:0/2114496581 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7ff5c402ff60 0x7ff5c4032420 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7ff5cc0096a0 tx=0x7ff5cc006b40).stop 2023-12-25T20:10:50.355 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:50.353+0000 7ff5e107d700 1 -- 172.21.15.160:0/2114496581 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff5dc0f5410 msgr2=0x7ff5dc109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:10:50.355 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:50.353+0000 7ff5e107d700 1 --2- 172.21.15.160:0/2114496581 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff5dc0f5410 0x7ff5dc109450 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7ff5d0013510 tx=0x7ff5d000b040).stop 2023-12-25T20:10:50.355 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:50.353+0000 7ff5e107d700 1 -- 172.21.15.160:0/2114496581 shutdown_connections 2023-12-25T20:10:50.355 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:50.353+0000 7ff5e107d700 1 --2- 172.21.15.160:0/2114496581 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7ff5c402ff60 0x7ff5c4032420 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:10:50.355 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:50.353+0000 7ff5e107d700 1 --2- 172.21.15.160:0/2114496581 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff5dc0f5410 0x7ff5dc109450 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:10:50.355 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:50.353+0000 7ff5e107d700 1 -- 172.21.15.160:0/2114496581 >> 172.21.15.160:0/2114496581 conn(0x7ff5dc0f0b70 msgr2=0x7ff5dc0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:10:50.355 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:50.353+0000 7ff5e107d700 1 -- 172.21.15.160:0/2114496581 shutdown_connections 2023-12-25T20:10:50.355 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:50.353+0000 7ff5e107d700 1 -- 172.21.15.160:0/2114496581 wait complete. 2023-12-25T20:10:50.355 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:10:51.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:10:50 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:10:50.353253+0000 mon.smithi050 (mon.0) 188 : audit [DBG] from='client.? 172.21.15.160:0/2114496581' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:10:51.787 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:10:51.787 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:10:52.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:10:51 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:10:50.655723+0000 mgr.smithi050.nbzhun (mgr.14162) 109 : cluster [DBG] pgmap v102: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:10:53.047 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:53.047+0000 7f3b1910a700 1 -- 172.21.15.160:0/3460165846 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3b140f62a0 msgr2=0x7f3b140f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:10:53.048 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:53.047+0000 7f3b1910a700 1 --2- 172.21.15.160:0/3460165846 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3b140f62a0 0x7f3b140f66c0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7f3b04009fb0 tx=0x7f3b04009b30).stop 2023-12-25T20:10:53.048 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:53.047+0000 7f3b1910a700 1 -- 172.21.15.160:0/3460165846 shutdown_connections 2023-12-25T20:10:53.048 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:53.047+0000 7f3b1910a700 1 --2- 172.21.15.160:0/3460165846 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3b140f62a0 0x7f3b140f66c0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:10:53.048 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:53.047+0000 7f3b1910a700 1 -- 172.21.15.160:0/3460165846 >> 172.21.15.160:0/3460165846 conn(0x7f3b140f1aa0 msgr2=0x7f3b140f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:10:53.048 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:53.047+0000 7f3b1910a700 1 -- 172.21.15.160:0/3460165846 shutdown_connections 2023-12-25T20:10:53.048 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:53.047+0000 7f3b1910a700 1 -- 172.21.15.160:0/3460165846 wait complete. 2023-12-25T20:10:53.049 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:53.047+0000 7f3b1910a700 1 Processor -- start 2023-12-25T20:10:53.049 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:53.047+0000 7f3b1910a700 1 -- start start 2023-12-25T20:10:53.050 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:53.048+0000 7f3b1910a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3b140f62a0 0x7f3b141064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:10:53.050 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:53.048+0000 7f3b1910a700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b141069a0 con 0x7f3b140f62a0 2023-12-25T20:10:53.050 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:53.048+0000 7f3b13fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3b140f62a0 0x7f3b141064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:10:53.050 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:53.048+0000 7f3b13fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3b140f62a0 0x7f3b141064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:38650/0 (socket says 172.21.15.160:38650) 2023-12-25T20:10:53.050 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:53.048+0000 7f3b13fff700 1 -- 172.21.15.160:0/1343986468 learned_addr learned my addr 172.21.15.160:0/1343986468 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:10:53.050 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:53.048+0000 7f3b13fff700 1 -- 172.21.15.160:0/1343986468 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b04005040 con 0x7f3b140f62a0 2023-12-25T20:10:53.050 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:53.048+0000 7f3b13fff700 1 --2- 172.21.15.160:0/1343986468 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3b140f62a0 0x7f3b141064b0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7f3b04013510 tx=0x7f3b0400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:10:53.050 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:53.048+0000 7f3b10ff9700 1 -- 172.21.15.160:0/1343986468 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3b04009320 con 0x7f3b140f62a0 2023-12-25T20:10:53.050 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:53.049+0000 7f3b1910a700 1 -- 172.21.15.160:0/1343986468 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b14106ba0 con 0x7f3b140f62a0 2023-12-25T20:10:53.050 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:53.049+0000 7f3b10ff9700 1 -- 172.21.15.160:0/1343986468 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3b04013f00 con 0x7f3b140f62a0 2023-12-25T20:10:53.050 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:53.049+0000 7f3b10ff9700 1 -- 172.21.15.160:0/1343986468 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3b04014dc0 con 0x7f3b140f62a0 2023-12-25T20:10:53.050 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:53.049+0000 7f3b1910a700 1 -- 172.21.15.160:0/1343986468 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b14107030 con 0x7f3b140f62a0 2023-12-25T20:10:53.051 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:53.049+0000 7f3b10ff9700 1 -- 172.21.15.160:0/1343986468 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3b04014420 con 0x7f3b140f62a0 2023-12-25T20:10:53.051 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:53.049+0000 7f3b1910a700 1 -- 172.21.15.160:0/1343986468 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b14045ad0 con 0x7f3b140f62a0 2023-12-25T20:10:53.051 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:53.050+0000 7f3b10ff9700 1 --2- 172.21.15.160:0/1343986468 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f3afc02ff60 0x7f3afc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:10:53.051 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:53.050+0000 7f3b10ff9700 1 -- 172.21.15.160:0/1343986468 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3b04041020 con 0x7f3b140f62a0 2023-12-25T20:10:53.051 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:53.050+0000 7f3b137fe700 1 --2- 172.21.15.160:0/1343986468 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f3afc02ff60 0x7f3afc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:10:53.051 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:53.050+0000 7f3b137fe700 1 --2- 172.21.15.160:0/1343986468 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f3afc02ff60 0x7f3afc032420 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f3b080096a0 tx=0x7f3b08006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:10:53.055 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:53.055+0000 7f3b10ff9700 1 -- 172.21.15.160:0/1343986468 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3b040095f0 con 0x7f3b140f62a0 2023-12-25T20:10:53.258 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:53.257+0000 7f3b1910a700 1 -- 172.21.15.160:0/1343986468 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3b14107e80 con 0x7f3b140f62a0 2023-12-25T20:10:53.259 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:53.259+0000 7f3b10ff9700 1 -- 172.21.15.160:0/1343986468 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3b0401cb10 con 0x7f3b140f62a0 2023-12-25T20:10:53.259 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:10:53.259 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:10:53.261 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:53.260+0000 7f3b1910a700 1 -- 172.21.15.160:0/1343986468 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f3afc02ff60 msgr2=0x7f3afc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:10:53.261 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:53.260+0000 7f3b1910a700 1 --2- 172.21.15.160:0/1343986468 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f3afc02ff60 0x7f3afc032420 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f3b080096a0 tx=0x7f3b08006b40).stop 2023-12-25T20:10:53.261 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:53.260+0000 7f3b1910a700 1 -- 172.21.15.160:0/1343986468 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3b140f62a0 msgr2=0x7f3b141064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:10:53.261 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:53.260+0000 7f3b1910a700 1 --2- 172.21.15.160:0/1343986468 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3b140f62a0 0x7f3b141064b0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7f3b04013510 tx=0x7f3b0400b040).stop 2023-12-25T20:10:53.261 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:53.260+0000 7f3b1910a700 1 -- 172.21.15.160:0/1343986468 shutdown_connections 2023-12-25T20:10:53.261 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:53.260+0000 7f3b1910a700 1 --2- 172.21.15.160:0/1343986468 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f3afc02ff60 0x7f3afc032420 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:10:53.261 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:53.260+0000 7f3b1910a700 1 --2- 172.21.15.160:0/1343986468 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3b140f62a0 0x7f3b141064b0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:10:53.261 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:53.260+0000 7f3b1910a700 1 -- 172.21.15.160:0/1343986468 >> 172.21.15.160:0/1343986468 conn(0x7f3b140f1aa0 msgr2=0x7f3b140f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:10:53.261 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:53.260+0000 7f3b1910a700 1 -- 172.21.15.160:0/1343986468 shutdown_connections 2023-12-25T20:10:53.262 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:53.260+0000 7f3b1910a700 1 -- 172.21.15.160:0/1343986468 wait complete. 2023-12-25T20:10:53.262 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:10:54.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:10:53 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:10:52.655955+0000 mgr.smithi050.nbzhun (mgr.14162) 110 : cluster [DBG] pgmap v103: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:10:54.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:10:53 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:10:53.259492+0000 mon.smithi050 (mon.0) 189 : audit [DBG] from='client.? 172.21.15.160:0/1343986468' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:10:54.670 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:10:54.670 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:10:55.841 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:55.840+0000 7f8c19b60700 1 -- 172.21.15.160:0/1309225952 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8c140f6510 msgr2=0x7f8c140f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:10:55.841 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:55.840+0000 7f8c19b60700 1 --2- 172.21.15.160:0/1309225952 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8c140f6510 0x7f8c140f6930 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7f8c0c009fb0 tx=0x7f8c0c009b30).stop 2023-12-25T20:10:55.841 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:55.840+0000 7f8c19b60700 1 -- 172.21.15.160:0/1309225952 shutdown_connections 2023-12-25T20:10:55.841 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:55.840+0000 7f8c19b60700 1 --2- 172.21.15.160:0/1309225952 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8c140f6510 0x7f8c140f6930 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:10:55.841 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:55.840+0000 7f8c19b60700 1 -- 172.21.15.160:0/1309225952 >> 172.21.15.160:0/1309225952 conn(0x7f8c140f1c70 msgr2=0x7f8c140f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:10:55.841 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:55.840+0000 7f8c19b60700 1 -- 172.21.15.160:0/1309225952 shutdown_connections 2023-12-25T20:10:55.841 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:55.840+0000 7f8c19b60700 1 -- 172.21.15.160:0/1309225952 wait complete. 2023-12-25T20:10:55.841 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:55.840+0000 7f8c19b60700 1 Processor -- start 2023-12-25T20:10:55.841 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:55.841+0000 7f8c19b60700 1 -- start start 2023-12-25T20:10:55.841 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:55.841+0000 7f8c19b60700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8c140f6510 0x7f8c14106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:10:55.842 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:55.841+0000 7f8c19b60700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c14106c20 con 0x7f8c140f6510 2023-12-25T20:10:55.842 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:55.841+0000 7f8c137fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8c140f6510 0x7f8c14106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:10:55.842 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:55.841+0000 7f8c137fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8c140f6510 0x7f8c14106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:38670/0 (socket says 172.21.15.160:38670) 2023-12-25T20:10:55.842 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:55.841+0000 7f8c137fe700 1 -- 172.21.15.160:0/2190955470 learned_addr learned my addr 172.21.15.160:0/2190955470 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:10:55.843 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:55.841+0000 7f8c137fe700 1 -- 172.21.15.160:0/2190955470 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c0c005040 con 0x7f8c140f6510 2023-12-25T20:10:55.843 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:55.841+0000 7f8c137fe700 1 --2- 172.21.15.160:0/2190955470 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8c140f6510 0x7f8c14106730 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7f8c0c013510 tx=0x7f8c0c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:10:55.843 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:55.842+0000 7f8c0bfff700 1 -- 172.21.15.160:0/2190955470 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8c0c009320 con 0x7f8c140f6510 2023-12-25T20:10:55.843 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:55.842+0000 7f8c19b60700 1 -- 172.21.15.160:0/2190955470 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c14106e20 con 0x7f8c140f6510 2023-12-25T20:10:55.844 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:55.842+0000 7f8c0bfff700 1 -- 172.21.15.160:0/2190955470 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8c0c013f00 con 0x7f8c140f6510 2023-12-25T20:10:55.844 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:55.842+0000 7f8c0bfff700 1 -- 172.21.15.160:0/2190955470 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8c0c014d90 con 0x7f8c140f6510 2023-12-25T20:10:55.844 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:55.842+0000 7f8c19b60700 1 -- 172.21.15.160:0/2190955470 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c141072b0 con 0x7f8c140f6510 2023-12-25T20:10:55.844 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:55.842+0000 7f8c0bfff700 1 -- 172.21.15.160:0/2190955470 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8c0c0095f0 con 0x7f8c140f6510 2023-12-25T20:10:55.844 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:55.843+0000 7f8c19b60700 1 -- 172.21.15.160:0/2190955470 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c14045ad0 con 0x7f8c140f6510 2023-12-25T20:10:55.844 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:55.843+0000 7f8c0bfff700 1 --2- 172.21.15.160:0/2190955470 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f8c0002ff60 0x7f8c00032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:10:55.844 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:55.843+0000 7f8c0bfff700 1 -- 172.21.15.160:0/2190955470 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8c0c0410c0 con 0x7f8c140f6510 2023-12-25T20:10:55.844 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:55.843+0000 7f8c12ffd700 1 --2- 172.21.15.160:0/2190955470 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f8c0002ff60 0x7f8c00032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:10:55.844 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:55.843+0000 7f8c12ffd700 1 --2- 172.21.15.160:0/2190955470 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f8c0002ff60 0x7f8c00032420 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f8bfc0096a0 tx=0x7f8bfc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:10:55.847 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:55.847+0000 7f8c0bfff700 1 -- 172.21.15.160:0/2190955470 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8c0c014420 con 0x7f8c140f6510 2023-12-25T20:10:56.055 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:56.055+0000 7f8c19b60700 1 -- 172.21.15.160:0/2190955470 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8c140fbf60 con 0x7f8c140f6510 2023-12-25T20:10:56.055 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:56.056+0000 7f8c0bfff700 1 -- 172.21.15.160:0/2190955470 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8c0c010230 con 0x7f8c140f6510 2023-12-25T20:10:56.056 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:10:56.056 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:10:56.057 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:56.058+0000 7f8c19b60700 1 -- 172.21.15.160:0/2190955470 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f8c0002ff60 msgr2=0x7f8c00032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:10:56.057 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:56.058+0000 7f8c19b60700 1 --2- 172.21.15.160:0/2190955470 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f8c0002ff60 0x7f8c00032420 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f8bfc0096a0 tx=0x7f8bfc006b40).stop 2023-12-25T20:10:56.058 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:56.058+0000 7f8c19b60700 1 -- 172.21.15.160:0/2190955470 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8c140f6510 msgr2=0x7f8c14106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:10:56.058 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:56.058+0000 7f8c19b60700 1 --2- 172.21.15.160:0/2190955470 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8c140f6510 0x7f8c14106730 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7f8c0c013510 tx=0x7f8c0c00b040).stop 2023-12-25T20:10:56.058 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:56.058+0000 7f8c19b60700 1 -- 172.21.15.160:0/2190955470 shutdown_connections 2023-12-25T20:10:56.058 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:56.058+0000 7f8c19b60700 1 --2- 172.21.15.160:0/2190955470 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f8c0002ff60 0x7f8c00032420 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:10:56.058 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:56.058+0000 7f8c19b60700 1 --2- 172.21.15.160:0/2190955470 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8c140f6510 0x7f8c14106730 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:10:56.058 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:56.058+0000 7f8c19b60700 1 -- 172.21.15.160:0/2190955470 >> 172.21.15.160:0/2190955470 conn(0x7f8c140f1c70 msgr2=0x7f8c140f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:10:56.058 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:56.058+0000 7f8c19b60700 1 -- 172.21.15.160:0/2190955470 shutdown_connections 2023-12-25T20:10:56.058 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:56.058+0000 7f8c19b60700 1 -- 172.21.15.160:0/2190955470 wait complete. 2023-12-25T20:10:56.058 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:10:56.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:10:55 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:10:54.656246+0000 mgr.smithi050.nbzhun (mgr.14162) 111 : cluster [DBG] pgmap v104: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:10:57.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:10:56 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:10:56.056517+0000 mon.smithi050 (mon.0) 190 : audit [DBG] from='client.? 172.21.15.160:0/2190955470' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:10:57.453 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:10:57.453 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:10:58.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:10:57 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:10:56.656478+0000 mgr.smithi050.nbzhun (mgr.14162) 112 : cluster [DBG] pgmap v105: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:10:58.663 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:58.663+0000 7fc82ab32700 1 -- 172.21.15.160:0/25760833 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc824060280 msgr2=0x7fc8240f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:10:58.664 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:58.663+0000 7fc82ab32700 1 --2- 172.21.15.160:0/25760833 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc824060280 0x7fc8240f90f0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7fc814009fb0 tx=0x7fc814009b30).stop 2023-12-25T20:10:58.664 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:58.663+0000 7fc82ab32700 1 -- 172.21.15.160:0/25760833 shutdown_connections 2023-12-25T20:10:58.664 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:58.663+0000 7fc82ab32700 1 --2- 172.21.15.160:0/25760833 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc824060280 0x7fc8240f90f0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:10:58.664 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:58.663+0000 7fc82ab32700 1 -- 172.21.15.160:0/25760833 >> 172.21.15.160:0/25760833 conn(0x7fc8240f0b70 msgr2=0x7fc8240f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:10:58.664 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:58.663+0000 7fc82ab32700 1 -- 172.21.15.160:0/25760833 shutdown_connections 2023-12-25T20:10:58.664 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:58.663+0000 7fc82ab32700 1 -- 172.21.15.160:0/25760833 wait complete. 2023-12-25T20:10:58.664 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:58.663+0000 7fc82ab32700 1 Processor -- start 2023-12-25T20:10:58.664 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:58.663+0000 7fc82ab32700 1 -- start start 2023-12-25T20:10:58.664 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:58.664+0000 7fc82ab32700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc824060280 0x7fc8240f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:10:58.664 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:58.664+0000 7fc82ab32700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8240f4ff0 con 0x7fc824060280 2023-12-25T20:10:58.667 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:58.664+0000 7fc8288ce700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc824060280 0x7fc8240f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:10:58.667 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:58.664+0000 7fc8288ce700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc824060280 0x7fc8240f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:58938/0 (socket says 172.21.15.160:58938) 2023-12-25T20:10:58.668 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:58.664+0000 7fc8288ce700 1 -- 172.21.15.160:0/2506135019 learned_addr learned my addr 172.21.15.160:0/2506135019 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:10:58.668 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:58.665+0000 7fc8288ce700 1 -- 172.21.15.160:0/2506135019 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc814005040 con 0x7fc824060280 2023-12-25T20:10:58.668 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:58.665+0000 7fc8288ce700 1 --2- 172.21.15.160:0/2506135019 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc824060280 0x7fc8240f6b70 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7fc814013510 tx=0x7fc81400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:10:58.668 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:58.665+0000 7fc8217fa700 1 -- 172.21.15.160:0/2506135019 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc814009320 con 0x7fc824060280 2023-12-25T20:10:58.669 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:58.665+0000 7fc8217fa700 1 -- 172.21.15.160:0/2506135019 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc814013f00 con 0x7fc824060280 2023-12-25T20:10:58.669 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:58.665+0000 7fc8217fa700 1 -- 172.21.15.160:0/2506135019 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc814014dc0 con 0x7fc824060280 2023-12-25T20:10:58.669 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:58.665+0000 7fc82ab32700 1 -- 172.21.15.160:0/2506135019 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8240f5190 con 0x7fc824060280 2023-12-25T20:10:58.669 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:58.665+0000 7fc82ab32700 1 -- 172.21.15.160:0/2506135019 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8240f5620 con 0x7fc824060280 2023-12-25T20:10:58.669 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:58.666+0000 7fc8217fa700 1 -- 172.21.15.160:0/2506135019 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc8140095f0 con 0x7fc824060280 2023-12-25T20:10:58.669 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:58.666+0000 7fc82ab32700 1 -- 172.21.15.160:0/2506135019 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc824059100 con 0x7fc824060280 2023-12-25T20:10:58.670 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:58.666+0000 7fc8217fa700 1 --2- 172.21.15.160:0/2506135019 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fc80c02ff60 0x7fc80c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:10:58.670 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:58.666+0000 7fc8217fa700 1 -- 172.21.15.160:0/2506135019 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc814041100 con 0x7fc824060280 2023-12-25T20:10:58.670 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:58.666+0000 7fc823fff700 1 --2- 172.21.15.160:0/2506135019 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fc80c02ff60 0x7fc80c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:10:58.670 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:58.666+0000 7fc823fff700 1 --2- 172.21.15.160:0/2506135019 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fc80c02ff60 0x7fc80c032420 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7fc8180096a0 tx=0x7fc818006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:10:58.671 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:58.671+0000 7fc8217fa700 1 -- 172.21.15.160:0/2506135019 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc814014420 con 0x7fc824060280 2023-12-25T20:10:58.874 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:58.875+0000 7fc82ab32700 1 -- 172.21.15.160:0/2506135019 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc8240fbf50 con 0x7fc824060280 2023-12-25T20:10:58.876 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:58.877+0000 7fc8217fa700 1 -- 172.21.15.160:0/2506135019 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc814010230 con 0x7fc824060280 2023-12-25T20:10:58.877 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:10:58.877 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:10:58.878 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:58.878+0000 7fc82ab32700 1 -- 172.21.15.160:0/2506135019 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fc80c02ff60 msgr2=0x7fc80c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:10:58.878 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:58.878+0000 7fc82ab32700 1 --2- 172.21.15.160:0/2506135019 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fc80c02ff60 0x7fc80c032420 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7fc8180096a0 tx=0x7fc818006b40).stop 2023-12-25T20:10:58.878 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:58.878+0000 7fc82ab32700 1 -- 172.21.15.160:0/2506135019 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc824060280 msgr2=0x7fc8240f6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:10:58.878 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:58.878+0000 7fc82ab32700 1 --2- 172.21.15.160:0/2506135019 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc824060280 0x7fc8240f6b70 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7fc814013510 tx=0x7fc81400b040).stop 2023-12-25T20:10:58.879 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:58.878+0000 7fc82ab32700 1 -- 172.21.15.160:0/2506135019 shutdown_connections 2023-12-25T20:10:58.879 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:58.878+0000 7fc82ab32700 1 --2- 172.21.15.160:0/2506135019 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fc80c02ff60 0x7fc80c032420 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:10:58.879 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:58.878+0000 7fc82ab32700 1 --2- 172.21.15.160:0/2506135019 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc824060280 0x7fc8240f6b70 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:10:58.879 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:58.878+0000 7fc82ab32700 1 -- 172.21.15.160:0/2506135019 >> 172.21.15.160:0/2506135019 conn(0x7fc8240f0b70 msgr2=0x7fc8240f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:10:58.879 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:58.878+0000 7fc82ab32700 1 -- 172.21.15.160:0/2506135019 shutdown_connections 2023-12-25T20:10:58.879 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:10:58.878+0000 7fc82ab32700 1 -- 172.21.15.160:0/2506135019 wait complete. 2023-12-25T20:10:58.880 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:11:00.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:10:59 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:10:58.656794+0000 mgr.smithi050.nbzhun (mgr.14162) 113 : cluster [DBG] pgmap v106: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:11:00.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:10:59 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:10:58.877421+0000 mon.smithi050 (mon.0) 191 : audit [DBG] from='client.? 172.21.15.160:0/2506135019' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:11:00.310 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:11:00.310 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:11:01.579 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:01.577+0000 7f5f33e70700 1 -- 172.21.15.160:0/521009085 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5f2c0f5410 msgr2=0x7f5f2c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:11:01.579 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:01.577+0000 7f5f33e70700 1 --2- 172.21.15.160:0/521009085 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5f2c0f5410 0x7f5f2c0f5830 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7f5f1c009fb0 tx=0x7f5f1c009b30).stop 2023-12-25T20:11:01.579 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:01.577+0000 7f5f33e70700 1 -- 172.21.15.160:0/521009085 shutdown_connections 2023-12-25T20:11:01.579 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:01.577+0000 7f5f33e70700 1 --2- 172.21.15.160:0/521009085 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5f2c0f5410 0x7f5f2c0f5830 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:11:01.579 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:01.577+0000 7f5f33e70700 1 -- 172.21.15.160:0/521009085 >> 172.21.15.160:0/521009085 conn(0x7f5f2c0f0b70 msgr2=0x7f5f2c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:11:01.579 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:01.578+0000 7f5f33e70700 1 -- 172.21.15.160:0/521009085 shutdown_connections 2023-12-25T20:11:01.579 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:01.578+0000 7f5f33e70700 1 -- 172.21.15.160:0/521009085 wait complete. 2023-12-25T20:11:01.579 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:01.578+0000 7f5f33e70700 1 Processor -- start 2023-12-25T20:11:01.579 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:01.578+0000 7f5f33e70700 1 -- start start 2023-12-25T20:11:01.580 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:01.578+0000 7f5f33e70700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5f2c0f5410 0x7f5f2c109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:11:01.580 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:01.578+0000 7f5f33e70700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f2c060220 con 0x7f5f2c0f5410 2023-12-25T20:11:01.580 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:01.578+0000 7f5f31c0c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5f2c0f5410 0x7f5f2c109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:11:01.580 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:01.578+0000 7f5f31c0c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5f2c0f5410 0x7f5f2c109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:58946/0 (socket says 172.21.15.160:58946) 2023-12-25T20:11:01.580 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:01.578+0000 7f5f31c0c700 1 -- 172.21.15.160:0/57453377 learned_addr learned my addr 172.21.15.160:0/57453377 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:11:01.580 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:01.579+0000 7f5f31c0c700 1 -- 172.21.15.160:0/57453377 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f1c005040 con 0x7f5f2c0f5410 2023-12-25T20:11:01.580 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:01.579+0000 7f5f31c0c700 1 --2- 172.21.15.160:0/57453377 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5f2c0f5410 0x7f5f2c109450 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7f5f1c013510 tx=0x7f5f1c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:11:01.580 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:01.579+0000 7f5f227fc700 1 -- 172.21.15.160:0/57453377 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5f1c009320 con 0x7f5f2c0f5410 2023-12-25T20:11:01.580 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:01.579+0000 7f5f33e70700 1 -- 172.21.15.160:0/57453377 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f2c060420 con 0x7f5f2c0f5410 2023-12-25T20:11:01.580 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:01.579+0000 7f5f227fc700 1 -- 172.21.15.160:0/57453377 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5f1c013f00 con 0x7f5f2c0f5410 2023-12-25T20:11:01.580 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:01.579+0000 7f5f227fc700 1 -- 172.21.15.160:0/57453377 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5f1c014d90 con 0x7f5f2c0f5410 2023-12-25T20:11:01.581 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:01.579+0000 7f5f33e70700 1 -- 172.21.15.160:0/57453377 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f2c109b00 con 0x7f5f2c0f5410 2023-12-25T20:11:01.581 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:01.580+0000 7f5f227fc700 1 -- 172.21.15.160:0/57453377 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5f1c0095f0 con 0x7f5f2c0f5410 2023-12-25T20:11:01.582 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:01.580+0000 7f5f227fc700 1 --2- 172.21.15.160:0/57453377 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f5f1802ff60 0x7f5f18032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:11:01.582 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:01.580+0000 7f5f227fc700 1 -- 172.21.15.160:0/57453377 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5f1c040740 con 0x7f5f2c0f5410 2023-12-25T20:11:01.582 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:01.580+0000 7f5f3140b700 1 --2- 172.21.15.160:0/57453377 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f5f1802ff60 0x7f5f18032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:11:01.582 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:01.580+0000 7f5f33e70700 1 -- 172.21.15.160:0/57453377 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f2c059100 con 0x7f5f2c0f5410 2023-12-25T20:11:01.582 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:01.581+0000 7f5f3140b700 1 --2- 172.21.15.160:0/57453377 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f5f1802ff60 0x7f5f18032420 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f5f280096a0 tx=0x7f5f28006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:11:01.585 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:01.585+0000 7f5f227fc700 1 -- 172.21.15.160:0/57453377 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5f1c014420 con 0x7f5f2c0f5410 2023-12-25T20:11:01.792 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:01.793+0000 7f5f33e70700 1 -- 172.21.15.160:0/57453377 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5f2c045ad0 con 0x7f5f2c0f5410 2023-12-25T20:11:01.793 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:01.793+0000 7f5f227fc700 1 -- 172.21.15.160:0/57453377 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5f1c01b740 con 0x7f5f2c0f5410 2023-12-25T20:11:01.793 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:11:01.793 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:11:01.795 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:01.795+0000 7f5f33e70700 1 -- 172.21.15.160:0/57453377 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f5f1802ff60 msgr2=0x7f5f18032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:11:01.795 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:01.795+0000 7f5f33e70700 1 --2- 172.21.15.160:0/57453377 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f5f1802ff60 0x7f5f18032420 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f5f280096a0 tx=0x7f5f28006b40).stop 2023-12-25T20:11:01.795 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:01.795+0000 7f5f33e70700 1 -- 172.21.15.160:0/57453377 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5f2c0f5410 msgr2=0x7f5f2c109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:11:01.795 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:01.795+0000 7f5f33e70700 1 --2- 172.21.15.160:0/57453377 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5f2c0f5410 0x7f5f2c109450 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7f5f1c013510 tx=0x7f5f1c00b040).stop 2023-12-25T20:11:01.795 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:01.795+0000 7f5f33e70700 1 -- 172.21.15.160:0/57453377 shutdown_connections 2023-12-25T20:11:01.795 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:01.795+0000 7f5f33e70700 1 --2- 172.21.15.160:0/57453377 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f5f1802ff60 0x7f5f18032420 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:11:01.795 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:01.795+0000 7f5f33e70700 1 --2- 172.21.15.160:0/57453377 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5f2c0f5410 0x7f5f2c109450 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:11:01.795 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:01.795+0000 7f5f33e70700 1 -- 172.21.15.160:0/57453377 >> 172.21.15.160:0/57453377 conn(0x7f5f2c0f0b70 msgr2=0x7f5f2c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:11:01.795 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:01.795+0000 7f5f33e70700 1 -- 172.21.15.160:0/57453377 shutdown_connections 2023-12-25T20:11:01.795 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:01.795+0000 7f5f33e70700 1 -- 172.21.15.160:0/57453377 wait complete. 2023-12-25T20:11:01.796 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:11:02.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:11:01 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:11:00.657179+0000 mgr.smithi050.nbzhun (mgr.14162) 114 : cluster [DBG] pgmap v107: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:11:02.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:11:01 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:11:01.793911+0000 mon.smithi050 (mon.0) 192 : audit [DBG] from='client.? 172.21.15.160:0/57453377' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:11:03.211 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:11:03.212 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:11:04.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:11:03 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:11:02.657520+0000 mgr.smithi050.nbzhun (mgr.14162) 115 : cluster [DBG] pgmap v108: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:11:04.444 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:04.443+0000 7f0cb0336700 1 -- 172.21.15.160:0/2349333077 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0ca80f5410 msgr2=0x7f0ca80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:11:04.444 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:04.443+0000 7f0cb0336700 1 --2- 172.21.15.160:0/2349333077 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0ca80f5410 0x7f0ca80f5830 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7f0c98009fb0 tx=0x7f0c98009b30).stop 2023-12-25T20:11:04.444 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:04.443+0000 7f0cb0336700 1 -- 172.21.15.160:0/2349333077 shutdown_connections 2023-12-25T20:11:04.444 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:04.443+0000 7f0cb0336700 1 --2- 172.21.15.160:0/2349333077 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0ca80f5410 0x7f0ca80f5830 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:11:04.444 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:04.443+0000 7f0cb0336700 1 -- 172.21.15.160:0/2349333077 >> 172.21.15.160:0/2349333077 conn(0x7f0ca80f0b70 msgr2=0x7f0ca80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:11:04.444 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:04.443+0000 7f0cb0336700 1 -- 172.21.15.160:0/2349333077 shutdown_connections 2023-12-25T20:11:04.445 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:04.443+0000 7f0cb0336700 1 -- 172.21.15.160:0/2349333077 wait complete. 2023-12-25T20:11:04.445 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:04.444+0000 7f0cb0336700 1 Processor -- start 2023-12-25T20:11:04.445 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:04.444+0000 7f0cb0336700 1 -- start start 2023-12-25T20:11:04.445 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:04.444+0000 7f0cb0336700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0ca80f5410 0x7f0ca8109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:11:04.445 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:04.444+0000 7f0cb0336700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ca8060240 con 0x7f0ca80f5410 2023-12-25T20:11:04.445 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:04.444+0000 7f0cae0d2700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0ca80f5410 0x7f0ca8109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:11:04.445 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:04.444+0000 7f0cae0d2700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0ca80f5410 0x7f0ca8109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:58966/0 (socket says 172.21.15.160:58966) 2023-12-25T20:11:04.445 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:04.444+0000 7f0cae0d2700 1 -- 172.21.15.160:0/998473319 learned_addr learned my addr 172.21.15.160:0/998473319 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:11:04.445 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:04.444+0000 7f0cae0d2700 1 -- 172.21.15.160:0/998473319 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c98005040 con 0x7f0ca80f5410 2023-12-25T20:11:04.445 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:04.445+0000 7f0cae0d2700 1 --2- 172.21.15.160:0/998473319 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0ca80f5410 0x7f0ca8109430 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7f0c98013510 tx=0x7f0c9800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:11:04.446 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:04.445+0000 7f0c9effd700 1 -- 172.21.15.160:0/998473319 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0c98009320 con 0x7f0ca80f5410 2023-12-25T20:11:04.446 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:04.445+0000 7f0cb0336700 1 -- 172.21.15.160:0/998473319 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ca8060440 con 0x7f0ca80f5410 2023-12-25T20:11:04.446 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:04.445+0000 7f0c9effd700 1 -- 172.21.15.160:0/998473319 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0c98013f00 con 0x7f0ca80f5410 2023-12-25T20:11:04.446 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:04.445+0000 7f0c9effd700 1 -- 172.21.15.160:0/998473319 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0c98014d90 con 0x7f0ca80f5410 2023-12-25T20:11:04.447 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:04.445+0000 7f0cb0336700 1 -- 172.21.15.160:0/998473319 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ca8109ae0 con 0x7f0ca80f5410 2023-12-25T20:11:04.447 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:04.446+0000 7f0cb0336700 1 -- 172.21.15.160:0/998473319 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ca8045ad0 con 0x7f0ca80f5410 2023-12-25T20:11:04.447 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:04.446+0000 7f0c9effd700 1 -- 172.21.15.160:0/998473319 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0c980095f0 con 0x7f0ca80f5410 2023-12-25T20:11:04.447 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:04.446+0000 7f0c9effd700 1 --2- 172.21.15.160:0/998473319 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f0c9402ff60 0x7f0c94032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:11:04.447 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:04.446+0000 7f0c9effd700 1 -- 172.21.15.160:0/998473319 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0c98040fe0 con 0x7f0ca80f5410 2023-12-25T20:11:04.447 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:04.446+0000 7f0cad8d1700 1 --2- 172.21.15.160:0/998473319 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f0c9402ff60 0x7f0c94032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:11:04.447 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:04.446+0000 7f0cad8d1700 1 --2- 172.21.15.160:0/998473319 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f0c9402ff60 0x7f0c94032420 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f0ca40096a0 tx=0x7f0ca4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:11:04.450 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:04.451+0000 7f0c9effd700 1 -- 172.21.15.160:0/998473319 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0c9801d7f0 con 0x7f0ca80f5410 2023-12-25T20:11:04.657 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:04.657+0000 7f0cb0336700 1 -- 172.21.15.160:0/998473319 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0ca80fae60 con 0x7f0ca80f5410 2023-12-25T20:11:04.657 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:04.657+0000 7f0c9effd700 1 -- 172.21.15.160:0/998473319 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0c98010230 con 0x7f0ca80f5410 2023-12-25T20:11:04.657 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:11:04.657 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:11:04.659 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:04.659+0000 7f0cb0336700 1 -- 172.21.15.160:0/998473319 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f0c9402ff60 msgr2=0x7f0c94032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:11:04.659 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:04.659+0000 7f0cb0336700 1 --2- 172.21.15.160:0/998473319 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f0c9402ff60 0x7f0c94032420 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f0ca40096a0 tx=0x7f0ca4006b40).stop 2023-12-25T20:11:04.659 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:04.659+0000 7f0cb0336700 1 -- 172.21.15.160:0/998473319 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0ca80f5410 msgr2=0x7f0ca8109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:11:04.659 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:04.659+0000 7f0cb0336700 1 --2- 172.21.15.160:0/998473319 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0ca80f5410 0x7f0ca8109430 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7f0c98013510 tx=0x7f0c9800b040).stop 2023-12-25T20:11:04.659 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:04.659+0000 7f0cb0336700 1 -- 172.21.15.160:0/998473319 shutdown_connections 2023-12-25T20:11:04.659 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:04.659+0000 7f0cb0336700 1 --2- 172.21.15.160:0/998473319 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f0c9402ff60 0x7f0c94032420 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:11:04.660 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:04.659+0000 7f0cb0336700 1 --2- 172.21.15.160:0/998473319 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0ca80f5410 0x7f0ca8109430 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:11:04.660 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:04.659+0000 7f0cb0336700 1 -- 172.21.15.160:0/998473319 >> 172.21.15.160:0/998473319 conn(0x7f0ca80f0b70 msgr2=0x7f0ca80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:11:04.660 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:04.659+0000 7f0cb0336700 1 -- 172.21.15.160:0/998473319 shutdown_connections 2023-12-25T20:11:04.660 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:04.659+0000 7f0cb0336700 1 -- 172.21.15.160:0/998473319 wait complete. 2023-12-25T20:11:04.660 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:11:05.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:11:04 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:11:04.658260+0000 mon.smithi050 (mon.0) 193 : audit [DBG] from='client.? 172.21.15.160:0/998473319' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:11:06.078 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:11:06.078 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:11:06.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:11:05 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:11:04.657862+0000 mgr.smithi050.nbzhun (mgr.14162) 116 : cluster [DBG] pgmap v109: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:11:07.258 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:07.257+0000 7f7c9a668700 1 -- 172.21.15.160:0/3453103381 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7c940f78e0 msgr2=0x7f7c940f7d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:11:07.258 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:07.257+0000 7f7c9a668700 1 --2- 172.21.15.160:0/3453103381 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7c940f78e0 0x7f7c940f7d00 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7f7c88009fb0 tx=0x7f7c88009b30).stop 2023-12-25T20:11:07.258 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:07.257+0000 7f7c9a668700 1 -- 172.21.15.160:0/3453103381 shutdown_connections 2023-12-25T20:11:07.258 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:07.257+0000 7f7c9a668700 1 --2- 172.21.15.160:0/3453103381 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7c940f78e0 0x7f7c940f7d00 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:11:07.258 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:07.257+0000 7f7c9a668700 1 -- 172.21.15.160:0/3453103381 >> 172.21.15.160:0/3453103381 conn(0x7f7c940f0120 msgr2=0x7f7c940f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:11:07.258 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:07.257+0000 7f7c9a668700 1 -- 172.21.15.160:0/3453103381 shutdown_connections 2023-12-25T20:11:07.258 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:07.257+0000 7f7c9a668700 1 -- 172.21.15.160:0/3453103381 wait complete. 2023-12-25T20:11:07.258 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:07.257+0000 7f7c9a668700 1 Processor -- start 2023-12-25T20:11:07.258 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:07.257+0000 7f7c9a668700 1 -- start start 2023-12-25T20:11:07.258 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:07.257+0000 7f7c9a668700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7c940f78e0 0x7f7c941064e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:11:07.258 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:07.257+0000 7f7c9a668700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c941069d0 con 0x7f7c940f78e0 2023-12-25T20:11:07.258 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:07.258+0000 7f7c99666700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7c940f78e0 0x7f7c941064e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:11:07.259 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:07.258+0000 7f7c99666700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7c940f78e0 0x7f7c941064e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:58978/0 (socket says 172.21.15.160:58978) 2023-12-25T20:11:07.259 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:07.258+0000 7f7c99666700 1 -- 172.21.15.160:0/3235071618 learned_addr learned my addr 172.21.15.160:0/3235071618 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:11:07.259 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:07.258+0000 7f7c99666700 1 -- 172.21.15.160:0/3235071618 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c88005040 con 0x7f7c940f78e0 2023-12-25T20:11:07.259 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:07.258+0000 7f7c99666700 1 --2- 172.21.15.160:0/3235071618 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7c940f78e0 0x7f7c941064e0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7f7c88013510 tx=0x7f7c8800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:11:07.260 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:07.258+0000 7f7c85ffb700 1 -- 172.21.15.160:0/3235071618 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7c88009320 con 0x7f7c940f78e0 2023-12-25T20:11:07.260 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:07.258+0000 7f7c9a668700 1 -- 172.21.15.160:0/3235071618 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c94106bd0 con 0x7f7c940f78e0 2023-12-25T20:11:07.260 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:07.258+0000 7f7c85ffb700 1 -- 172.21.15.160:0/3235071618 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7c88013f00 con 0x7f7c940f78e0 2023-12-25T20:11:07.260 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:07.258+0000 7f7c85ffb700 1 -- 172.21.15.160:0/3235071618 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7c88014dc0 con 0x7f7c940f78e0 2023-12-25T20:11:07.260 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:07.259+0000 7f7c9a668700 1 -- 172.21.15.160:0/3235071618 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c94107060 con 0x7f7c940f78e0 2023-12-25T20:11:07.260 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:07.259+0000 7f7c9a668700 1 -- 172.21.15.160:0/3235071618 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c94045ad0 con 0x7f7c940f78e0 2023-12-25T20:11:07.260 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:07.259+0000 7f7c85ffb700 1 -- 172.21.15.160:0/3235071618 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7c880095f0 con 0x7f7c940f78e0 2023-12-25T20:11:07.260 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:07.260+0000 7f7c85ffb700 1 --2- 172.21.15.160:0/3235071618 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f7c8002ff60 0x7f7c80032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:11:07.260 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:07.260+0000 7f7c85ffb700 1 -- 172.21.15.160:0/3235071618 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7c88041020 con 0x7f7c940f78e0 2023-12-25T20:11:07.261 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:07.260+0000 7f7c98e65700 1 --2- 172.21.15.160:0/3235071618 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f7c8002ff60 0x7f7c80032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:11:07.261 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:07.260+0000 7f7c98e65700 1 --2- 172.21.15.160:0/3235071618 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f7c8002ff60 0x7f7c80032420 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f7c900096a0 tx=0x7f7c90006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:11:07.264 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:07.265+0000 7f7c85ffb700 1 -- 172.21.15.160:0/3235071618 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7c88014420 con 0x7f7c940f78e0 2023-12-25T20:11:07.469 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:07.469+0000 7f7c9a668700 1 -- 172.21.15.160:0/3235071618 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7c94107eb0 con 0x7f7c940f78e0 2023-12-25T20:11:07.470 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:07.470+0000 7f7c85ffb700 1 -- 172.21.15.160:0/3235071618 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7c8801c920 con 0x7f7c940f78e0 2023-12-25T20:11:07.470 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:11:07.470 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:11:07.472 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:07.472+0000 7f7c9a668700 1 -- 172.21.15.160:0/3235071618 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f7c8002ff60 msgr2=0x7f7c80032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:11:07.472 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:07.472+0000 7f7c9a668700 1 --2- 172.21.15.160:0/3235071618 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f7c8002ff60 0x7f7c80032420 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f7c900096a0 tx=0x7f7c90006b40).stop 2023-12-25T20:11:07.472 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:07.472+0000 7f7c9a668700 1 -- 172.21.15.160:0/3235071618 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7c940f78e0 msgr2=0x7f7c941064e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:11:07.472 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:07.472+0000 7f7c9a668700 1 --2- 172.21.15.160:0/3235071618 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7c940f78e0 0x7f7c941064e0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7f7c88013510 tx=0x7f7c8800b040).stop 2023-12-25T20:11:07.472 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:07.472+0000 7f7c9a668700 1 -- 172.21.15.160:0/3235071618 shutdown_connections 2023-12-25T20:11:07.472 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:07.472+0000 7f7c9a668700 1 --2- 172.21.15.160:0/3235071618 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f7c8002ff60 0x7f7c80032420 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:11:07.472 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:07.472+0000 7f7c9a668700 1 --2- 172.21.15.160:0/3235071618 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7c940f78e0 0x7f7c941064e0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:11:07.472 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:07.472+0000 7f7c9a668700 1 -- 172.21.15.160:0/3235071618 >> 172.21.15.160:0/3235071618 conn(0x7f7c940f0120 msgr2=0x7f7c940f0bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:11:07.472 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:07.472+0000 7f7c9a668700 1 -- 172.21.15.160:0/3235071618 shutdown_connections 2023-12-25T20:11:07.472 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:07.472+0000 7f7c9a668700 1 -- 172.21.15.160:0/3235071618 wait complete. 2023-12-25T20:11:07.473 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:11:08.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:11:07 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:11:06.658267+0000 mgr.smithi050.nbzhun (mgr.14162) 117 : cluster [DBG] pgmap v110: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:11:08.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:11:07 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:11:07.470838+0000 mon.smithi050 (mon.0) 194 : audit [DBG] from='client.? 172.21.15.160:0/3235071618' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:11:08.894 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:11:08.894 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:11:10.147 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:10.146+0000 7f4a53d90700 1 -- 172.21.15.160:0/3571716128 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4a4c0f5410 msgr2=0x7f4a4c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:11:10.147 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:10.146+0000 7f4a53d90700 1 --2- 172.21.15.160:0/3571716128 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4a4c0f5410 0x7f4a4c0f5830 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0x7f4a3c009fb0 tx=0x7f4a3c009b30).stop 2023-12-25T20:11:10.147 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:10.146+0000 7f4a53d90700 1 -- 172.21.15.160:0/3571716128 shutdown_connections 2023-12-25T20:11:10.147 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:10.146+0000 7f4a53d90700 1 --2- 172.21.15.160:0/3571716128 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4a4c0f5410 0x7f4a4c0f5830 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:11:10.147 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:10.146+0000 7f4a53d90700 1 -- 172.21.15.160:0/3571716128 >> 172.21.15.160:0/3571716128 conn(0x7f4a4c0f0b70 msgr2=0x7f4a4c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:11:10.147 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:10.146+0000 7f4a53d90700 1 -- 172.21.15.160:0/3571716128 shutdown_connections 2023-12-25T20:11:10.147 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:10.146+0000 7f4a53d90700 1 -- 172.21.15.160:0/3571716128 wait complete. 2023-12-25T20:11:10.148 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:10.147+0000 7f4a53d90700 1 Processor -- start 2023-12-25T20:11:10.148 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:10.147+0000 7f4a53d90700 1 -- start start 2023-12-25T20:11:10.148 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:10.147+0000 7f4a53d90700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4a4c0f5410 0x7f4a4c106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:11:10.148 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:10.147+0000 7f4a53d90700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a4c106c20 con 0x7f4a4c0f5410 2023-12-25T20:11:10.148 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:10.147+0000 7f4a51b2c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4a4c0f5410 0x7f4a4c106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:11:10.148 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:10.147+0000 7f4a51b2c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4a4c0f5410 0x7f4a4c106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:49550/0 (socket says 172.21.15.160:49550) 2023-12-25T20:11:10.148 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:10.147+0000 7f4a51b2c700 1 -- 172.21.15.160:0/3429426985 learned_addr learned my addr 172.21.15.160:0/3429426985 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:11:10.149 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:10.148+0000 7f4a51b2c700 1 -- 172.21.15.160:0/3429426985 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a3c005040 con 0x7f4a4c0f5410 2023-12-25T20:11:10.149 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:10.148+0000 7f4a51b2c700 1 --2- 172.21.15.160:0/3429426985 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4a4c0f5410 0x7f4a4c106730 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7f4a3c014000 tx=0x7f4a3c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:11:10.149 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:10.148+0000 7f4a427fc700 1 -- 172.21.15.160:0/3429426985 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4a3c0095f0 con 0x7f4a4c0f5410 2023-12-25T20:11:10.150 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:10.148+0000 7f4a53d90700 1 -- 172.21.15.160:0/3429426985 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a4c106e20 con 0x7f4a4c0f5410 2023-12-25T20:11:10.150 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:10.148+0000 7f4a427fc700 1 -- 172.21.15.160:0/3429426985 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4a3c007e70 con 0x7f4a4c0f5410 2023-12-25T20:11:10.150 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:10.148+0000 7f4a427fc700 1 -- 172.21.15.160:0/3429426985 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4a3c014ea0 con 0x7f4a4c0f5410 2023-12-25T20:11:10.150 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:10.148+0000 7f4a53d90700 1 -- 172.21.15.160:0/3429426985 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a4c1072b0 con 0x7f4a4c0f5410 2023-12-25T20:11:10.150 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:10.149+0000 7f4a427fc700 1 -- 172.21.15.160:0/3429426985 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4a3c025070 con 0x7f4a4c0f5410 2023-12-25T20:11:10.150 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:10.149+0000 7f4a53d90700 1 -- 172.21.15.160:0/3429426985 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a4c045ad0 con 0x7f4a4c0f5410 2023-12-25T20:11:10.150 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:10.149+0000 7f4a427fc700 1 --2- 172.21.15.160:0/3429426985 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f4a3802ff60 0x7f4a38032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:11:10.150 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:10.149+0000 7f4a427fc700 1 -- 172.21.15.160:0/3429426985 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4a3c040970 con 0x7f4a4c0f5410 2023-12-25T20:11:10.150 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:10.149+0000 7f4a5132b700 1 --2- 172.21.15.160:0/3429426985 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f4a3802ff60 0x7f4a38032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:11:10.150 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:10.150+0000 7f4a5132b700 1 --2- 172.21.15.160:0/3429426985 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f4a3802ff60 0x7f4a38032420 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f4a480096a0 tx=0x7f4a48006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:11:10.153 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:10.154+0000 7f4a427fc700 1 -- 172.21.15.160:0/3429426985 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4a3c0079c0 con 0x7f4a4c0f5410 2023-12-25T20:11:10.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:11:09 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:11:08.658618+0000 mgr.smithi050.nbzhun (mgr.14162) 118 : cluster [DBG] pgmap v111: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:11:10.357 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:10.357+0000 7f4a53d90700 1 -- 172.21.15.160:0/3429426985 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4a4c0fae60 con 0x7f4a4c0f5410 2023-12-25T20:11:10.357 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:10.358+0000 7f4a427fc700 1 -- 172.21.15.160:0/3429426985 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4a3c00f300 con 0x7f4a4c0f5410 2023-12-25T20:11:10.358 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:11:10.358 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:11:10.359 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:10.359+0000 7f4a53d90700 1 -- 172.21.15.160:0/3429426985 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f4a3802ff60 msgr2=0x7f4a38032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:11:10.359 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:10.359+0000 7f4a53d90700 1 --2- 172.21.15.160:0/3429426985 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f4a3802ff60 0x7f4a38032420 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f4a480096a0 tx=0x7f4a48006b40).stop 2023-12-25T20:11:10.360 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:10.359+0000 7f4a53d90700 1 -- 172.21.15.160:0/3429426985 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4a4c0f5410 msgr2=0x7f4a4c106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:11:10.360 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:10.359+0000 7f4a53d90700 1 --2- 172.21.15.160:0/3429426985 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4a4c0f5410 0x7f4a4c106730 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7f4a3c014000 tx=0x7f4a3c00b040).stop 2023-12-25T20:11:10.360 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:10.359+0000 7f4a53d90700 1 -- 172.21.15.160:0/3429426985 shutdown_connections 2023-12-25T20:11:10.360 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:10.360+0000 7f4a53d90700 1 --2- 172.21.15.160:0/3429426985 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f4a3802ff60 0x7f4a38032420 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:11:10.360 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:10.360+0000 7f4a53d90700 1 --2- 172.21.15.160:0/3429426985 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4a4c0f5410 0x7f4a4c106730 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:11:10.360 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:10.360+0000 7f4a53d90700 1 -- 172.21.15.160:0/3429426985 >> 172.21.15.160:0/3429426985 conn(0x7f4a4c0f0b70 msgr2=0x7f4a4c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:11:10.360 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:10.360+0000 7f4a53d90700 1 -- 172.21.15.160:0/3429426985 shutdown_connections 2023-12-25T20:11:10.360 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:10.360+0000 7f4a53d90700 1 -- 172.21.15.160:0/3429426985 wait complete. 2023-12-25T20:11:10.360 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:11:11.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:11:10 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:11:10.358620+0000 mon.smithi050 (mon.0) 195 : audit [DBG] from='client.? 172.21.15.160:0/3429426985' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:11:11.796 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:11:11.797 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:11:12.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:11:11 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:11:10.658985+0000 mgr.smithi050.nbzhun (mgr.14162) 119 : cluster [DBG] pgmap v112: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:11:12.998 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:12.996+0000 7ff577d92700 1 -- 172.21.15.160:0/2881915447 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff5700f6510 msgr2=0x7ff5700f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:11:12.999 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:12.996+0000 7ff577d92700 1 --2- 172.21.15.160:0/2881915447 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff5700f6510 0x7ff5700f6930 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7ff560009fb0 tx=0x7ff560009b30).stop 2023-12-25T20:11:12.999 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:12.996+0000 7ff577d92700 1 -- 172.21.15.160:0/2881915447 shutdown_connections 2023-12-25T20:11:12.999 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:12.996+0000 7ff577d92700 1 --2- 172.21.15.160:0/2881915447 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff5700f6510 0x7ff5700f6930 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:11:12.999 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:12.996+0000 7ff577d92700 1 -- 172.21.15.160:0/2881915447 >> 172.21.15.160:0/2881915447 conn(0x7ff5700f1c70 msgr2=0x7ff5700f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:11:12.999 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:12.996+0000 7ff577d92700 1 -- 172.21.15.160:0/2881915447 shutdown_connections 2023-12-25T20:11:12.999 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:12.996+0000 7ff577d92700 1 -- 172.21.15.160:0/2881915447 wait complete. 2023-12-25T20:11:12.999 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:12.997+0000 7ff577d92700 1 Processor -- start 2023-12-25T20:11:12.999 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:12.997+0000 7ff577d92700 1 -- start start 2023-12-25T20:11:12.999 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:12.997+0000 7ff577d92700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff5700f6510 0x7ff57006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:11:12.999 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:12.997+0000 7ff577d92700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff570069e20 con 0x7ff5700f6510 2023-12-25T20:11:12.999 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:12.997+0000 7ff575b2e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff5700f6510 0x7ff57006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:11:12.999 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:12.997+0000 7ff575b2e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff5700f6510 0x7ff57006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:49574/0 (socket says 172.21.15.160:49574) 2023-12-25T20:11:13.000 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:12.997+0000 7ff575b2e700 1 -- 172.21.15.160:0/3393112110 learned_addr learned my addr 172.21.15.160:0/3393112110 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:11:13.000 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:12.997+0000 7ff575b2e700 1 -- 172.21.15.160:0/3393112110 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff560005040 con 0x7ff5700f6510 2023-12-25T20:11:13.000 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:12.998+0000 7ff575b2e700 1 --2- 172.21.15.160:0/3393112110 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff5700f6510 0x7ff57006b9a0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7ff560013510 tx=0x7ff56000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:11:13.000 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:12.998+0000 7ff5667fc700 1 -- 172.21.15.160:0/3393112110 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff560009320 con 0x7ff5700f6510 2023-12-25T20:11:13.000 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:12.998+0000 7ff577d92700 1 -- 172.21.15.160:0/3393112110 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff570069fc0 con 0x7ff5700f6510 2023-12-25T20:11:13.000 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:12.998+0000 7ff5667fc700 1 -- 172.21.15.160:0/3393112110 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff560013f00 con 0x7ff5700f6510 2023-12-25T20:11:13.000 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:12.998+0000 7ff5667fc700 1 -- 172.21.15.160:0/3393112110 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff560014d90 con 0x7ff5700f6510 2023-12-25T20:11:13.000 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:12.998+0000 7ff577d92700 1 -- 172.21.15.160:0/3393112110 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff57006a450 con 0x7ff5700f6510 2023-12-25T20:11:13.000 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:12.999+0000 7ff5667fc700 1 -- 172.21.15.160:0/3393112110 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff560014420 con 0x7ff5700f6510 2023-12-25T20:11:13.000 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:12.999+0000 7ff577d92700 1 -- 172.21.15.160:0/3393112110 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff570059100 con 0x7ff5700f6510 2023-12-25T20:11:13.000 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:12.999+0000 7ff5667fc700 1 --2- 172.21.15.160:0/3393112110 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7ff55c02ffb0 0x7ff55c032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:11:13.000 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:12.999+0000 7ff5667fc700 1 -- 172.21.15.160:0/3393112110 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff560040ff0 con 0x7ff5700f6510 2023-12-25T20:11:13.000 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:12.999+0000 7ff57532d700 1 --2- 172.21.15.160:0/3393112110 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7ff55c02ffb0 0x7ff55c032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:11:13.001 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:12.999+0000 7ff57532d700 1 --2- 172.21.15.160:0/3393112110 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7ff55c02ffb0 0x7ff55c032470 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7ff56c0096a0 tx=0x7ff56c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:11:13.003 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:13.004+0000 7ff5667fc700 1 -- 172.21.15.160:0/3393112110 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff56006d2b0 con 0x7ff5700f6510 2023-12-25T20:11:13.209 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:13.209+0000 7ff577d92700 1 -- 172.21.15.160:0/3393112110 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff5700fbf60 con 0x7ff5700f6510 2023-12-25T20:11:13.210 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:13.210+0000 7ff5667fc700 1 -- 172.21.15.160:0/3393112110 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff5600095f0 con 0x7ff5700f6510 2023-12-25T20:11:13.210 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:11:13.210 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:11:13.211 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:13.212+0000 7ff577d92700 1 -- 172.21.15.160:0/3393112110 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7ff55c02ffb0 msgr2=0x7ff55c032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:11:13.212 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:13.212+0000 7ff577d92700 1 --2- 172.21.15.160:0/3393112110 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7ff55c02ffb0 0x7ff55c032470 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7ff56c0096a0 tx=0x7ff56c006b40).stop 2023-12-25T20:11:13.212 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:13.212+0000 7ff577d92700 1 -- 172.21.15.160:0/3393112110 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff5700f6510 msgr2=0x7ff57006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:11:13.212 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:13.212+0000 7ff577d92700 1 --2- 172.21.15.160:0/3393112110 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff5700f6510 0x7ff57006b9a0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7ff560013510 tx=0x7ff56000b040).stop 2023-12-25T20:11:13.212 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:13.212+0000 7ff577d92700 1 -- 172.21.15.160:0/3393112110 shutdown_connections 2023-12-25T20:11:13.212 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:13.212+0000 7ff577d92700 1 --2- 172.21.15.160:0/3393112110 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7ff55c02ffb0 0x7ff55c032470 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:11:13.212 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:13.212+0000 7ff577d92700 1 --2- 172.21.15.160:0/3393112110 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff5700f6510 0x7ff57006b9a0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:11:13.212 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:13.212+0000 7ff577d92700 1 -- 172.21.15.160:0/3393112110 >> 172.21.15.160:0/3393112110 conn(0x7ff5700f1c70 msgr2=0x7ff5700f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:11:13.212 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:13.212+0000 7ff577d92700 1 -- 172.21.15.160:0/3393112110 shutdown_connections 2023-12-25T20:11:13.212 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:13.212+0000 7ff577d92700 1 -- 172.21.15.160:0/3393112110 wait complete. 2023-12-25T20:11:13.213 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:11:14.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:11:13 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:11:12.659301+0000 mgr.smithi050.nbzhun (mgr.14162) 120 : cluster [DBG] pgmap v113: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:11:14.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:11:13 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:11:13.210753+0000 mon.smithi050 (mon.0) 196 : audit [DBG] from='client.? 172.21.15.160:0/3393112110' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:11:14.642 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:11:14.642 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:11:15.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:11:14 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:11:14.765208+0000 mon.smithi050 (mon.0) 197 : audit [INF] from='mgr.14162 172.21.15.50:0/3178367813' entity='mgr.smithi050.nbzhun' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi050.nbzhun/mirror_snapshot_schedule"}]: dispatch 2023-12-25T20:11:15.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:11:14 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:11:14.766220+0000 mon.smithi050 (mon.0) 198 : audit [INF] from='mgr.14162 172.21.15.50:0/3178367813' entity='mgr.smithi050.nbzhun' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi050.nbzhun/trash_purge_schedule"}]: dispatch 2023-12-25T20:11:15.838 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:15.836+0000 7f15180aa700 1 -- 172.21.15.160:0/3975898903 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f15100f6510 msgr2=0x7f15100f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:11:15.839 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:15.836+0000 7f15180aa700 1 --2- 172.21.15.160:0/3975898903 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f15100f6510 0x7f15100f6930 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7f1500009fb0 tx=0x7f1500009b30).stop 2023-12-25T20:11:15.839 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:15.836+0000 7f15180aa700 1 -- 172.21.15.160:0/3975898903 shutdown_connections 2023-12-25T20:11:15.839 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:15.836+0000 7f15180aa700 1 --2- 172.21.15.160:0/3975898903 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f15100f6510 0x7f15100f6930 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:11:15.839 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:15.836+0000 7f15180aa700 1 -- 172.21.15.160:0/3975898903 >> 172.21.15.160:0/3975898903 conn(0x7f15100f1c70 msgr2=0x7f15100f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:11:15.839 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:15.837+0000 7f15180aa700 1 -- 172.21.15.160:0/3975898903 shutdown_connections 2023-12-25T20:11:15.839 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:15.837+0000 7f15180aa700 1 -- 172.21.15.160:0/3975898903 wait complete. 2023-12-25T20:11:15.839 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:15.837+0000 7f15180aa700 1 Processor -- start 2023-12-25T20:11:15.839 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:15.837+0000 7f15180aa700 1 -- start start 2023-12-25T20:11:15.839 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:15.837+0000 7f15180aa700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f15100f6510 0x7f1510106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:11:15.839 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:15.837+0000 7f15180aa700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1510106c20 con 0x7f15100f6510 2023-12-25T20:11:15.840 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:15.838+0000 7f1515e46700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f15100f6510 0x7f1510106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:11:15.840 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:15.838+0000 7f1515e46700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f15100f6510 0x7f1510106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:49594/0 (socket says 172.21.15.160:49594) 2023-12-25T20:11:15.840 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:15.838+0000 7f1515e46700 1 -- 172.21.15.160:0/1085850805 learned_addr learned my addr 172.21.15.160:0/1085850805 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:11:15.840 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:15.838+0000 7f1515e46700 1 -- 172.21.15.160:0/1085850805 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1500005040 con 0x7f15100f6510 2023-12-25T20:11:15.840 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:15.838+0000 7f1515e46700 1 --2- 172.21.15.160:0/1085850805 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f15100f6510 0x7f1510106730 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7f1500013510 tx=0x7f150000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:11:15.840 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:15.838+0000 7f15067fc700 1 -- 172.21.15.160:0/1085850805 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1500009320 con 0x7f15100f6510 2023-12-25T20:11:15.840 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:15.838+0000 7f15180aa700 1 -- 172.21.15.160:0/1085850805 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1510106e20 con 0x7f15100f6510 2023-12-25T20:11:15.840 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:15.838+0000 7f15067fc700 1 -- 172.21.15.160:0/1085850805 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1500013f00 con 0x7f15100f6510 2023-12-25T20:11:15.840 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:15.838+0000 7f15067fc700 1 -- 172.21.15.160:0/1085850805 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1500014d90 con 0x7f15100f6510 2023-12-25T20:11:15.840 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:15.838+0000 7f15180aa700 1 -- 172.21.15.160:0/1085850805 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15101072b0 con 0x7f15100f6510 2023-12-25T20:11:15.841 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:15.839+0000 7f15067fc700 1 -- 172.21.15.160:0/1085850805 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f15000095f0 con 0x7f15100f6510 2023-12-25T20:11:15.841 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:15.839+0000 7f15180aa700 1 -- 172.21.15.160:0/1085850805 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1510045ad0 con 0x7f15100f6510 2023-12-25T20:11:15.841 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:15.839+0000 7f15067fc700 1 --2- 172.21.15.160:0/1085850805 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f14fc02ff60 0x7f14fc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:11:15.841 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:15.839+0000 7f15067fc700 1 -- 172.21.15.160:0/1085850805 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1500040ff0 con 0x7f15100f6510 2023-12-25T20:11:15.842 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:15.839+0000 7f1515645700 1 --2- 172.21.15.160:0/1085850805 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f14fc02ff60 0x7f14fc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:11:15.842 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:15.840+0000 7f1515645700 1 --2- 172.21.15.160:0/1085850805 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f14fc02ff60 0x7f14fc032420 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f150c0096a0 tx=0x7f150c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:11:15.843 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:15.844+0000 7f15067fc700 1 -- 172.21.15.160:0/1085850805 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1500014420 con 0x7f15100f6510 2023-12-25T20:11:16.049 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:16.049+0000 7f15180aa700 1 -- 172.21.15.160:0/1085850805 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f15100fbf60 con 0x7f15100f6510 2023-12-25T20:11:16.050 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:16.050+0000 7f15067fc700 1 -- 172.21.15.160:0/1085850805 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f150001d980 con 0x7f15100f6510 2023-12-25T20:11:16.050 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:11:16.050 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:11:16.051 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:16.051+0000 7f15180aa700 1 -- 172.21.15.160:0/1085850805 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f14fc02ff60 msgr2=0x7f14fc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:11:16.051 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:16.051+0000 7f15180aa700 1 --2- 172.21.15.160:0/1085850805 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f14fc02ff60 0x7f14fc032420 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f150c0096a0 tx=0x7f150c006b40).stop 2023-12-25T20:11:16.051 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:16.051+0000 7f15180aa700 1 -- 172.21.15.160:0/1085850805 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f15100f6510 msgr2=0x7f1510106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:11:16.052 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:16.051+0000 7f15180aa700 1 --2- 172.21.15.160:0/1085850805 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f15100f6510 0x7f1510106730 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7f1500013510 tx=0x7f150000b040).stop 2023-12-25T20:11:16.052 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:16.051+0000 7f15180aa700 1 -- 172.21.15.160:0/1085850805 shutdown_connections 2023-12-25T20:11:16.052 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:16.051+0000 7f15180aa700 1 --2- 172.21.15.160:0/1085850805 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f14fc02ff60 0x7f14fc032420 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:11:16.052 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:16.051+0000 7f15180aa700 1 --2- 172.21.15.160:0/1085850805 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f15100f6510 0x7f1510106730 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:11:16.052 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:16.051+0000 7f15180aa700 1 -- 172.21.15.160:0/1085850805 >> 172.21.15.160:0/1085850805 conn(0x7f15100f1c70 msgr2=0x7f15100f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:11:16.052 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:16.052+0000 7f15180aa700 1 -- 172.21.15.160:0/1085850805 shutdown_connections 2023-12-25T20:11:16.052 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:16.052+0000 7f15180aa700 1 -- 172.21.15.160:0/1085850805 wait complete. 2023-12-25T20:11:16.052 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:11:16.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:11:15 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:11:14.659638+0000 mgr.smithi050.nbzhun (mgr.14162) 121 : cluster [DBG] pgmap v114: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:11:17.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:11:16 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:11:16.050618+0000 mon.smithi050 (mon.0) 199 : audit [DBG] from='client.? 172.21.15.160:0/1085850805' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:11:17.430 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:11:17.430 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:11:18.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:11:17 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:11:16.659963+0000 mgr.smithi050.nbzhun (mgr.14162) 122 : cluster [DBG] pgmap v115: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:11:18.655 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:18.654+0000 7f6f290ff700 1 -- 172.21.15.160:0/3708738132 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6f240f6510 msgr2=0x7f6f240f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:11:18.655 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:18.654+0000 7f6f290ff700 1 --2- 172.21.15.160:0/3708738132 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6f240f6510 0x7f6f240f6930 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0x7f6f0c009fb0 tx=0x7f6f0c009b30).stop 2023-12-25T20:11:18.655 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:18.654+0000 7f6f290ff700 1 -- 172.21.15.160:0/3708738132 shutdown_connections 2023-12-25T20:11:18.655 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:18.654+0000 7f6f290ff700 1 --2- 172.21.15.160:0/3708738132 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6f240f6510 0x7f6f240f6930 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:11:18.655 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:18.654+0000 7f6f290ff700 1 -- 172.21.15.160:0/3708738132 >> 172.21.15.160:0/3708738132 conn(0x7f6f240f1c70 msgr2=0x7f6f240f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:11:18.655 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:18.654+0000 7f6f290ff700 1 -- 172.21.15.160:0/3708738132 shutdown_connections 2023-12-25T20:11:18.655 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:18.654+0000 7f6f290ff700 1 -- 172.21.15.160:0/3708738132 wait complete. 2023-12-25T20:11:18.656 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:18.655+0000 7f6f290ff700 1 Processor -- start 2023-12-25T20:11:18.656 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:18.655+0000 7f6f290ff700 1 -- start start 2023-12-25T20:11:18.656 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:18.655+0000 7f6f290ff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6f240f6510 0x7f6f24106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:11:18.656 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:18.655+0000 7f6f290ff700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f24106c20 con 0x7f6f240f6510 2023-12-25T20:11:18.656 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:18.655+0000 7f6f22d9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6f240f6510 0x7f6f24106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:11:18.656 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:18.655+0000 7f6f22d9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6f240f6510 0x7f6f24106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:38672/0 (socket says 172.21.15.160:38672) 2023-12-25T20:11:18.656 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:18.655+0000 7f6f22d9d700 1 -- 172.21.15.160:0/585204927 learned_addr learned my addr 172.21.15.160:0/585204927 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:11:18.657 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:18.656+0000 7f6f22d9d700 1 -- 172.21.15.160:0/585204927 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f0c005040 con 0x7f6f240f6510 2023-12-25T20:11:18.658 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:18.656+0000 7f6f22d9d700 1 --2- 172.21.15.160:0/585204927 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6f240f6510 0x7f6f24106730 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7f6f0c013510 tx=0x7f6f0c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:11:18.658 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:18.656+0000 7f6f1b7fe700 1 -- 172.21.15.160:0/585204927 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6f0c009320 con 0x7f6f240f6510 2023-12-25T20:11:18.658 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:18.656+0000 7f6f290ff700 1 -- 172.21.15.160:0/585204927 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f24106e20 con 0x7f6f240f6510 2023-12-25T20:11:18.658 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:18.656+0000 7f6f1b7fe700 1 -- 172.21.15.160:0/585204927 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6f0c013f00 con 0x7f6f240f6510 2023-12-25T20:11:18.658 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:18.656+0000 7f6f290ff700 1 -- 172.21.15.160:0/585204927 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f241072b0 con 0x7f6f240f6510 2023-12-25T20:11:18.658 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:18.656+0000 7f6f1b7fe700 1 -- 172.21.15.160:0/585204927 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6f0c014d90 con 0x7f6f240f6510 2023-12-25T20:11:18.658 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:18.657+0000 7f6f1b7fe700 1 -- 172.21.15.160:0/585204927 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6f0c0095f0 con 0x7f6f240f6510 2023-12-25T20:11:18.658 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:18.657+0000 7f6f290ff700 1 -- 172.21.15.160:0/585204927 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f24045ad0 con 0x7f6f240f6510 2023-12-25T20:11:18.658 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:18.657+0000 7f6f1b7fe700 1 --2- 172.21.15.160:0/585204927 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f6f1002ff60 0x7f6f10032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:11:18.658 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:18.657+0000 7f6f1b7fe700 1 -- 172.21.15.160:0/585204927 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6f0c040ff0 con 0x7f6f240f6510 2023-12-25T20:11:18.658 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:18.657+0000 7f6f2259c700 1 --2- 172.21.15.160:0/585204927 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f6f1002ff60 0x7f6f10032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:11:18.658 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:18.658+0000 7f6f2259c700 1 --2- 172.21.15.160:0/585204927 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f6f1002ff60 0x7f6f10032420 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f6f140096a0 tx=0x7f6f14006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:11:18.661 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:18.662+0000 7f6f1b7fe700 1 -- 172.21.15.160:0/585204927 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6f0c014420 con 0x7f6f240f6510 2023-12-25T20:11:18.867 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:18.867+0000 7f6f290ff700 1 -- 172.21.15.160:0/585204927 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6f240fbf60 con 0x7f6f240f6510 2023-12-25T20:11:18.867 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:18.868+0000 7f6f1b7fe700 1 -- 172.21.15.160:0/585204927 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6f0c010230 con 0x7f6f240f6510 2023-12-25T20:11:18.867 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:11:18.868 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:11:18.869 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:18.869+0000 7f6f290ff700 1 -- 172.21.15.160:0/585204927 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f6f1002ff60 msgr2=0x7f6f10032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:11:18.869 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:18.869+0000 7f6f290ff700 1 --2- 172.21.15.160:0/585204927 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f6f1002ff60 0x7f6f10032420 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f6f140096a0 tx=0x7f6f14006b40).stop 2023-12-25T20:11:18.869 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:18.869+0000 7f6f290ff700 1 -- 172.21.15.160:0/585204927 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6f240f6510 msgr2=0x7f6f24106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:11:18.869 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:18.869+0000 7f6f290ff700 1 --2- 172.21.15.160:0/585204927 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6f240f6510 0x7f6f24106730 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7f6f0c013510 tx=0x7f6f0c00b040).stop 2023-12-25T20:11:18.869 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:18.869+0000 7f6f290ff700 1 -- 172.21.15.160:0/585204927 shutdown_connections 2023-12-25T20:11:18.870 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:18.869+0000 7f6f290ff700 1 --2- 172.21.15.160:0/585204927 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f6f1002ff60 0x7f6f10032420 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:11:18.870 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:18.869+0000 7f6f290ff700 1 --2- 172.21.15.160:0/585204927 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6f240f6510 0x7f6f24106730 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:11:18.870 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:18.869+0000 7f6f290ff700 1 -- 172.21.15.160:0/585204927 >> 172.21.15.160:0/585204927 conn(0x7f6f240f1c70 msgr2=0x7f6f240f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:11:18.870 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:18.870+0000 7f6f290ff700 1 -- 172.21.15.160:0/585204927 shutdown_connections 2023-12-25T20:11:18.870 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:18.870+0000 7f6f290ff700 1 -- 172.21.15.160:0/585204927 wait complete. 2023-12-25T20:11:18.870 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:11:19.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:11:18 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:11:18.868425+0000 mon.smithi050 (mon.0) 200 : audit [DBG] from='client.? 172.21.15.160:0/585204927' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:11:20.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:11:19 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:11:18.660244+0000 mgr.smithi050.nbzhun (mgr.14162) 123 : cluster [DBG] pgmap v116: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:11:20.259 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:11:20.259 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:11:21.481 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:21.476+0000 7fd120213700 1 -- 172.21.15.160:0/1373455326 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd1180f5410 msgr2=0x7fd1180f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:11:21.481 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:21.476+0000 7fd120213700 1 --2- 172.21.15.160:0/1373455326 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd1180f5410 0x7fd1180f5830 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7fd114009fb0 tx=0x7fd114009b30).stop 2023-12-25T20:11:21.481 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:21.476+0000 7fd120213700 1 -- 172.21.15.160:0/1373455326 shutdown_connections 2023-12-25T20:11:21.481 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:21.476+0000 7fd120213700 1 --2- 172.21.15.160:0/1373455326 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd1180f5410 0x7fd1180f5830 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:11:21.481 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:21.476+0000 7fd120213700 1 -- 172.21.15.160:0/1373455326 >> 172.21.15.160:0/1373455326 conn(0x7fd1180f0b70 msgr2=0x7fd1180f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:11:21.482 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:21.477+0000 7fd120213700 1 -- 172.21.15.160:0/1373455326 shutdown_connections 2023-12-25T20:11:21.482 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:21.477+0000 7fd120213700 1 -- 172.21.15.160:0/1373455326 wait complete. 2023-12-25T20:11:21.482 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:21.477+0000 7fd120213700 1 Processor -- start 2023-12-25T20:11:21.482 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:21.477+0000 7fd120213700 1 -- start start 2023-12-25T20:11:21.482 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:21.477+0000 7fd120213700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd1180f5410 0x7fd118105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:11:21.482 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:21.477+0000 7fd120213700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd118105b20 con 0x7fd1180f5410 2023-12-25T20:11:21.483 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:21.478+0000 7fd11dfaf700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd1180f5410 0x7fd118105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:11:21.483 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:21.478+0000 7fd11dfaf700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd1180f5410 0x7fd118105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:38692/0 (socket says 172.21.15.160:38692) 2023-12-25T20:11:21.483 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:21.478+0000 7fd11dfaf700 1 -- 172.21.15.160:0/1741729187 learned_addr learned my addr 172.21.15.160:0/1741729187 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:11:21.483 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:21.478+0000 7fd11dfaf700 1 -- 172.21.15.160:0/1741729187 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd114005040 con 0x7fd1180f5410 2023-12-25T20:11:21.484 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:21.478+0000 7fd11dfaf700 1 --2- 172.21.15.160:0/1741729187 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd1180f5410 0x7fd118105630 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7fd114013510 tx=0x7fd11400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:11:21.484 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:21.478+0000 7fd10e7fc700 1 -- 172.21.15.160:0/1741729187 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd114009320 con 0x7fd1180f5410 2023-12-25T20:11:21.484 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:21.478+0000 7fd120213700 1 -- 172.21.15.160:0/1741729187 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd118105d20 con 0x7fd1180f5410 2023-12-25T20:11:21.484 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:21.478+0000 7fd10e7fc700 1 -- 172.21.15.160:0/1741729187 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd114013f00 con 0x7fd1180f5410 2023-12-25T20:11:21.484 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:21.478+0000 7fd10e7fc700 1 -- 172.21.15.160:0/1741729187 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd114014d90 con 0x7fd1180f5410 2023-12-25T20:11:21.484 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:21.478+0000 7fd120213700 1 -- 172.21.15.160:0/1741729187 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1181061b0 con 0x7fd1180f5410 2023-12-25T20:11:21.484 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:21.479+0000 7fd10e7fc700 1 -- 172.21.15.160:0/1741729187 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd1140095f0 con 0x7fd1180f5410 2023-12-25T20:11:21.485 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:21.479+0000 7fd120213700 1 -- 172.21.15.160:0/1741729187 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd118045ad0 con 0x7fd1180f5410 2023-12-25T20:11:21.485 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:21.479+0000 7fd10e7fc700 1 --2- 172.21.15.160:0/1741729187 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fd10402ff60 0x7fd104032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:11:21.485 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:21.479+0000 7fd10e7fc700 1 -- 172.21.15.160:0/1741729187 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd114041000 con 0x7fd1180f5410 2023-12-25T20:11:21.485 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:21.479+0000 7fd11d7ae700 1 --2- 172.21.15.160:0/1741729187 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fd10402ff60 0x7fd104032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:11:21.485 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:21.480+0000 7fd11d7ae700 1 --2- 172.21.15.160:0/1741729187 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fd10402ff60 0x7fd104032420 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7fd1080096a0 tx=0x7fd108006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:11:21.486 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:21.484+0000 7fd10e7fc700 1 -- 172.21.15.160:0/1741729187 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd11401d7f0 con 0x7fd1180f5410 2023-12-25T20:11:21.689 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:21.689+0000 7fd120213700 1 -- 172.21.15.160:0/1741729187 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd1180fae60 con 0x7fd1180f5410 2023-12-25T20:11:21.689 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:21.690+0000 7fd10e7fc700 1 -- 172.21.15.160:0/1741729187 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd114010230 con 0x7fd1180f5410 2023-12-25T20:11:21.690 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:11:21.690 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:11:21.691 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:21.692+0000 7fd120213700 1 -- 172.21.15.160:0/1741729187 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fd10402ff60 msgr2=0x7fd104032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:11:21.691 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:21.692+0000 7fd120213700 1 --2- 172.21.15.160:0/1741729187 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fd10402ff60 0x7fd104032420 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7fd1080096a0 tx=0x7fd108006b40).stop 2023-12-25T20:11:21.692 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:21.692+0000 7fd120213700 1 -- 172.21.15.160:0/1741729187 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd1180f5410 msgr2=0x7fd118105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:11:21.692 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:21.692+0000 7fd120213700 1 --2- 172.21.15.160:0/1741729187 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd1180f5410 0x7fd118105630 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7fd114013510 tx=0x7fd11400b040).stop 2023-12-25T20:11:21.692 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:21.692+0000 7fd120213700 1 -- 172.21.15.160:0/1741729187 shutdown_connections 2023-12-25T20:11:21.692 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:21.692+0000 7fd120213700 1 --2- 172.21.15.160:0/1741729187 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fd10402ff60 0x7fd104032420 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:11:21.692 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:21.692+0000 7fd120213700 1 --2- 172.21.15.160:0/1741729187 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd1180f5410 0x7fd118105630 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:11:21.692 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:21.692+0000 7fd120213700 1 -- 172.21.15.160:0/1741729187 >> 172.21.15.160:0/1741729187 conn(0x7fd1180f0b70 msgr2=0x7fd1180f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:11:21.692 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:21.692+0000 7fd120213700 1 -- 172.21.15.160:0/1741729187 shutdown_connections 2023-12-25T20:11:21.692 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:21.692+0000 7fd120213700 1 -- 172.21.15.160:0/1741729187 wait complete. 2023-12-25T20:11:21.692 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:11:22.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:11:21 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:11:20.660563+0000 mgr.smithi050.nbzhun (mgr.14162) 124 : cluster [DBG] pgmap v117: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:11:22.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:11:21 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:11:21.690560+0000 mon.smithi050 (mon.0) 201 : audit [DBG] from='client.? 172.21.15.160:0/1741729187' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:11:23.089 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:11:23.089 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:11:24.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:11:23 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:11:22.660943+0000 mgr.smithi050.nbzhun (mgr.14162) 125 : cluster [DBG] pgmap v118: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:11:24.373 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:24.373+0000 7fd73104f700 1 -- 172.21.15.160:0/2657180021 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd72c060280 msgr2=0x7fd72c0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:11:24.374 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:24.373+0000 7fd73104f700 1 --2- 172.21.15.160:0/2657180021 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd72c060280 0x7fd72c0f90f0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7fd720009fb0 tx=0x7fd720009b30).stop 2023-12-25T20:11:24.374 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:24.373+0000 7fd73104f700 1 -- 172.21.15.160:0/2657180021 shutdown_connections 2023-12-25T20:11:24.374 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:24.373+0000 7fd73104f700 1 --2- 172.21.15.160:0/2657180021 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd72c060280 0x7fd72c0f90f0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:11:24.374 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:24.373+0000 7fd73104f700 1 -- 172.21.15.160:0/2657180021 >> 172.21.15.160:0/2657180021 conn(0x7fd72c0f0b70 msgr2=0x7fd72c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:11:24.374 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:24.373+0000 7fd73104f700 1 -- 172.21.15.160:0/2657180021 shutdown_connections 2023-12-25T20:11:24.374 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:24.373+0000 7fd73104f700 1 -- 172.21.15.160:0/2657180021 wait complete. 2023-12-25T20:11:24.374 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:24.374+0000 7fd73104f700 1 Processor -- start 2023-12-25T20:11:24.374 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:24.374+0000 7fd73104f700 1 -- start start 2023-12-25T20:11:24.374 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:24.374+0000 7fd73104f700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd72c060280 0x7fd72c106690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:11:24.374 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:24.374+0000 7fd73104f700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd72c106b80 con 0x7fd72c060280 2023-12-25T20:11:24.376 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:24.374+0000 7fd72ad9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd72c060280 0x7fd72c106690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:11:24.376 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:24.374+0000 7fd72ad9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd72c060280 0x7fd72c106690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:38700/0 (socket says 172.21.15.160:38700) 2023-12-25T20:11:24.376 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:24.374+0000 7fd72ad9d700 1 -- 172.21.15.160:0/685982839 learned_addr learned my addr 172.21.15.160:0/685982839 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:11:24.376 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:24.375+0000 7fd72ad9d700 1 -- 172.21.15.160:0/685982839 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd720005040 con 0x7fd72c060280 2023-12-25T20:11:24.376 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:24.375+0000 7fd72ad9d700 1 --2- 172.21.15.160:0/685982839 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd72c060280 0x7fd72c106690 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7fd720009830 tx=0x7fd72000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:11:24.376 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:24.375+0000 7fd7137fe700 1 -- 172.21.15.160:0/685982839 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd720009320 con 0x7fd72c060280 2023-12-25T20:11:24.376 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:24.375+0000 7fd7137fe700 1 -- 172.21.15.160:0/685982839 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd7200136c0 con 0x7fd72c060280 2023-12-25T20:11:24.377 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:24.375+0000 7fd73104f700 1 -- 172.21.15.160:0/685982839 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd72c106d80 con 0x7fd72c060280 2023-12-25T20:11:24.377 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:24.375+0000 7fd7137fe700 1 -- 172.21.15.160:0/685982839 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd72001ee90 con 0x7fd72c060280 2023-12-25T20:11:24.377 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:24.375+0000 7fd73104f700 1 -- 172.21.15.160:0/685982839 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd72c107210 con 0x7fd72c060280 2023-12-25T20:11:24.377 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:24.376+0000 7fd7137fe700 1 -- 172.21.15.160:0/685982839 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd72001e420 con 0x7fd72c060280 2023-12-25T20:11:24.377 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:24.376+0000 7fd73104f700 1 -- 172.21.15.160:0/685982839 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd72c045ad0 con 0x7fd72c060280 2023-12-25T20:11:24.377 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:24.376+0000 7fd7137fe700 1 --2- 172.21.15.160:0/685982839 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fd71402ff10 0x7fd7140323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:11:24.377 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:24.376+0000 7fd7137fe700 1 -- 172.21.15.160:0/685982839 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd720041040 con 0x7fd72c060280 2023-12-25T20:11:24.377 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:24.376+0000 7fd72a59c700 1 --2- 172.21.15.160:0/685982839 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fd71402ff10 0x7fd7140323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:11:24.377 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:24.377+0000 7fd72a59c700 1 --2- 172.21.15.160:0/685982839 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fd71402ff10 0x7fd7140323d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7fd71c0096a0 tx=0x7fd71c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:11:24.380 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:24.381+0000 7fd7137fe700 1 -- 172.21.15.160:0/685982839 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd72006d2b0 con 0x7fd72c060280 2023-12-25T20:11:24.585 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:24.585+0000 7fd73104f700 1 -- 172.21.15.160:0/685982839 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd72c1080c0 con 0x7fd72c060280 2023-12-25T20:11:24.585 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:24.586+0000 7fd7137fe700 1 -- 172.21.15.160:0/685982839 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd72001c790 con 0x7fd72c060280 2023-12-25T20:11:24.585 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:11:24.585 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:11:24.587 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:24.587+0000 7fd73104f700 1 -- 172.21.15.160:0/685982839 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fd71402ff10 msgr2=0x7fd7140323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:11:24.587 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:24.587+0000 7fd73104f700 1 --2- 172.21.15.160:0/685982839 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fd71402ff10 0x7fd7140323d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7fd71c0096a0 tx=0x7fd71c006b40).stop 2023-12-25T20:11:24.587 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:24.587+0000 7fd73104f700 1 -- 172.21.15.160:0/685982839 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd72c060280 msgr2=0x7fd72c106690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:11:24.587 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:24.587+0000 7fd73104f700 1 --2- 172.21.15.160:0/685982839 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd72c060280 0x7fd72c106690 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7fd720009830 tx=0x7fd72000b040).stop 2023-12-25T20:11:24.588 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:24.587+0000 7fd73104f700 1 -- 172.21.15.160:0/685982839 shutdown_connections 2023-12-25T20:11:24.588 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:24.587+0000 7fd73104f700 1 --2- 172.21.15.160:0/685982839 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fd71402ff10 0x7fd7140323d0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:11:24.588 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:24.587+0000 7fd73104f700 1 --2- 172.21.15.160:0/685982839 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd72c060280 0x7fd72c106690 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:11:24.588 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:24.587+0000 7fd73104f700 1 -- 172.21.15.160:0/685982839 >> 172.21.15.160:0/685982839 conn(0x7fd72c0f0b70 msgr2=0x7fd72c0f1600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:11:24.588 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:24.587+0000 7fd73104f700 1 -- 172.21.15.160:0/685982839 shutdown_connections 2023-12-25T20:11:24.588 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:24.587+0000 7fd73104f700 1 -- 172.21.15.160:0/685982839 wait complete. 2023-12-25T20:11:24.588 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:11:25.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:11:24 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:11:24.586221+0000 mon.smithi050 (mon.0) 202 : audit [DBG] from='client.? 172.21.15.160:0/685982839' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:11:26.024 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:11:26.024 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:11:26.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:11:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:11:24.661298+0000 mgr.smithi050.nbzhun (mgr.14162) 126 : cluster [DBG] pgmap v119: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:11:27.244 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:27.242+0000 7f14f732a700 1 -- 172.21.15.160:0/382621200 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f14f00f5410 msgr2=0x7f14f00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:11:27.244 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:27.242+0000 7f14f732a700 1 --2- 172.21.15.160:0/382621200 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f14f00f5410 0x7f14f00f5830 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0x7f14e4009fb0 tx=0x7f14e4009b30).stop 2023-12-25T20:11:27.244 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:27.242+0000 7f14f732a700 1 -- 172.21.15.160:0/382621200 shutdown_connections 2023-12-25T20:11:27.244 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:27.242+0000 7f14f732a700 1 --2- 172.21.15.160:0/382621200 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f14f00f5410 0x7f14f00f5830 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:11:27.244 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:27.242+0000 7f14f732a700 1 -- 172.21.15.160:0/382621200 >> 172.21.15.160:0/382621200 conn(0x7f14f00f0b70 msgr2=0x7f14f00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:11:27.244 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:27.243+0000 7f14f732a700 1 -- 172.21.15.160:0/382621200 shutdown_connections 2023-12-25T20:11:27.244 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:27.243+0000 7f14f732a700 1 -- 172.21.15.160:0/382621200 wait complete. 2023-12-25T20:11:27.244 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:27.243+0000 7f14f732a700 1 Processor -- start 2023-12-25T20:11:27.245 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:27.243+0000 7f14f732a700 1 -- start start 2023-12-25T20:11:27.245 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:27.243+0000 7f14f732a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f14f00f5410 0x7f14f0109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:11:27.245 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:27.243+0000 7f14f732a700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14f0060220 con 0x7f14f00f5410 2023-12-25T20:11:27.245 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:27.243+0000 7f14f50c6700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f14f00f5410 0x7f14f0109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:11:27.245 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:27.243+0000 7f14f50c6700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f14f00f5410 0x7f14f0109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:38712/0 (socket says 172.21.15.160:38712) 2023-12-25T20:11:27.245 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:27.243+0000 7f14f50c6700 1 -- 172.21.15.160:0/3802271027 learned_addr learned my addr 172.21.15.160:0/3802271027 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:11:27.245 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:27.244+0000 7f14f50c6700 1 -- 172.21.15.160:0/3802271027 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14e4005040 con 0x7f14f00f5410 2023-12-25T20:11:27.245 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:27.244+0000 7f14f50c6700 1 --2- 172.21.15.160:0/3802271027 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f14f00f5410 0x7f14f0109450 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7f14e4013510 tx=0x7f14e400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:11:27.245 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:27.244+0000 7f14e1ffb700 1 -- 172.21.15.160:0/3802271027 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f14e4009320 con 0x7f14f00f5410 2023-12-25T20:11:27.245 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:27.244+0000 7f14e1ffb700 1 -- 172.21.15.160:0/3802271027 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f14e4013f00 con 0x7f14f00f5410 2023-12-25T20:11:27.245 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:27.244+0000 7f14f732a700 1 -- 172.21.15.160:0/3802271027 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14f0060420 con 0x7f14f00f5410 2023-12-25T20:11:27.245 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:27.244+0000 7f14e1ffb700 1 -- 172.21.15.160:0/3802271027 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f14e4014d90 con 0x7f14f00f5410 2023-12-25T20:11:27.245 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:27.244+0000 7f14f732a700 1 -- 172.21.15.160:0/3802271027 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14f0109b00 con 0x7f14f00f5410 2023-12-25T20:11:27.246 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:27.245+0000 7f14e1ffb700 1 -- 172.21.15.160:0/3802271027 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f14e40263a0 con 0x7f14f00f5410 2023-12-25T20:11:27.246 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:27.245+0000 7f14f732a700 1 -- 172.21.15.160:0/3802271027 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14f0059100 con 0x7f14f00f5410 2023-12-25T20:11:27.246 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:27.245+0000 7f14e1ffb700 1 --2- 172.21.15.160:0/3802271027 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f14dc02ff60 0x7f14dc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:11:27.247 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:27.245+0000 7f14e1ffb700 1 -- 172.21.15.160:0/3802271027 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f14e4041340 con 0x7f14f00f5410 2023-12-25T20:11:27.247 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:27.246+0000 7f14f48c5700 1 --2- 172.21.15.160:0/3802271027 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f14dc02ff60 0x7f14dc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:11:27.247 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:27.246+0000 7f14f48c5700 1 --2- 172.21.15.160:0/3802271027 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f14dc02ff60 0x7f14dc032420 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f14ec0096a0 tx=0x7f14ec006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:11:27.250 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:27.250+0000 7f14e1ffb700 1 -- 172.21.15.160:0/3802271027 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f14e4012330 con 0x7f14f00f5410 2023-12-25T20:11:27.455 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:27.455+0000 7f14f732a700 1 -- 172.21.15.160:0/3802271027 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f14f0045ad0 con 0x7f14f00f5410 2023-12-25T20:11:27.456 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:27.456+0000 7f14e1ffb700 1 -- 172.21.15.160:0/3802271027 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f14e4012330 con 0x7f14f00f5410 2023-12-25T20:11:27.456 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:11:27.456 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:11:27.457 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:27.458+0000 7f14f732a700 1 -- 172.21.15.160:0/3802271027 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f14dc02ff60 msgr2=0x7f14dc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:11:27.458 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:27.458+0000 7f14f732a700 1 --2- 172.21.15.160:0/3802271027 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f14dc02ff60 0x7f14dc032420 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f14ec0096a0 tx=0x7f14ec006b40).stop 2023-12-25T20:11:27.458 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:27.458+0000 7f14f732a700 1 -- 172.21.15.160:0/3802271027 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f14f00f5410 msgr2=0x7f14f0109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:11:27.458 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:27.458+0000 7f14f732a700 1 --2- 172.21.15.160:0/3802271027 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f14f00f5410 0x7f14f0109450 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7f14e4013510 tx=0x7f14e400b040).stop 2023-12-25T20:11:27.458 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:27.458+0000 7f14f732a700 1 -- 172.21.15.160:0/3802271027 shutdown_connections 2023-12-25T20:11:27.458 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:27.458+0000 7f14f732a700 1 --2- 172.21.15.160:0/3802271027 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f14dc02ff60 0x7f14dc032420 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:11:27.458 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:27.458+0000 7f14f732a700 1 --2- 172.21.15.160:0/3802271027 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f14f00f5410 0x7f14f0109450 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:11:27.458 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:27.458+0000 7f14f732a700 1 -- 172.21.15.160:0/3802271027 >> 172.21.15.160:0/3802271027 conn(0x7f14f00f0b70 msgr2=0x7f14f00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:11:27.458 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:27.458+0000 7f14f732a700 1 -- 172.21.15.160:0/3802271027 shutdown_connections 2023-12-25T20:11:27.458 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:27.458+0000 7f14f732a700 1 -- 172.21.15.160:0/3802271027 wait complete. 2023-12-25T20:11:27.459 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:11:28.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:11:27 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:11:26.661631+0000 mgr.smithi050.nbzhun (mgr.14162) 127 : cluster [DBG] pgmap v120: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:11:28.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:11:27 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:11:27.456558+0000 mon.smithi050 (mon.0) 203 : audit [DBG] from='client.? 172.21.15.160:0/3802271027' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:11:28.904 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:11:28.904 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:11:30.078 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:30.076+0000 7fd356878700 1 -- 172.21.15.160:0/2576686976 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd3500f4860 msgr2=0x7fd3500f4c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:11:30.078 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:30.076+0000 7fd356878700 1 --2- 172.21.15.160:0/2576686976 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd3500f4860 0x7fd3500f4c80 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0x7fd340009fb0 tx=0x7fd340009b30).stop 2023-12-25T20:11:30.078 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:30.077+0000 7fd356878700 1 -- 172.21.15.160:0/2576686976 shutdown_connections 2023-12-25T20:11:30.078 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:30.077+0000 7fd356878700 1 --2- 172.21.15.160:0/2576686976 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd3500f4860 0x7fd3500f4c80 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:11:30.078 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:30.077+0000 7fd356878700 1 -- 172.21.15.160:0/2576686976 >> 172.21.15.160:0/2576686976 conn(0x7fd3500f02e0 msgr2=0x7fd3500f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:11:30.078 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:30.077+0000 7fd356878700 1 -- 172.21.15.160:0/2576686976 shutdown_connections 2023-12-25T20:11:30.078 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:30.077+0000 7fd356878700 1 -- 172.21.15.160:0/2576686976 wait complete. 2023-12-25T20:11:30.078 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:30.077+0000 7fd356878700 1 Processor -- start 2023-12-25T20:11:30.078 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:30.077+0000 7fd356878700 1 -- start start 2023-12-25T20:11:30.080 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:30.077+0000 7fd356878700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd3500f4860 0x7fd3501068e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:11:30.080 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:30.077+0000 7fd356878700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd350106dd0 con 0x7fd3500f4860 2023-12-25T20:11:30.080 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:30.078+0000 7fd34ffff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd3500f4860 0x7fd3501068e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:11:30.080 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:30.078+0000 7fd34ffff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd3500f4860 0x7fd3501068e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:41900/0 (socket says 172.21.15.160:41900) 2023-12-25T20:11:30.080 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:30.078+0000 7fd34ffff700 1 -- 172.21.15.160:0/2448502072 learned_addr learned my addr 172.21.15.160:0/2448502072 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:11:30.080 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:30.078+0000 7fd34ffff700 1 -- 172.21.15.160:0/2448502072 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd340005040 con 0x7fd3500f4860 2023-12-25T20:11:30.080 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:30.078+0000 7fd34ffff700 1 --2- 172.21.15.160:0/2448502072 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd3500f4860 0x7fd3501068e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7fd340013510 tx=0x7fd34000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:11:30.080 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:30.078+0000 7fd34cff9700 1 -- 172.21.15.160:0/2448502072 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd340009320 con 0x7fd3500f4860 2023-12-25T20:11:30.080 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:30.078+0000 7fd356878700 1 -- 172.21.15.160:0/2448502072 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd350106fd0 con 0x7fd3500f4860 2023-12-25T20:11:30.080 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:30.078+0000 7fd34cff9700 1 -- 172.21.15.160:0/2448502072 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd340013f00 con 0x7fd3500f4860 2023-12-25T20:11:30.081 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:30.078+0000 7fd34cff9700 1 -- 172.21.15.160:0/2448502072 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd340014dc0 con 0x7fd3500f4860 2023-12-25T20:11:30.081 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:30.078+0000 7fd356878700 1 -- 172.21.15.160:0/2448502072 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd350107460 con 0x7fd3500f4860 2023-12-25T20:11:30.081 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:30.079+0000 7fd34cff9700 1 -- 172.21.15.160:0/2448502072 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd3400095f0 con 0x7fd3500f4860 2023-12-25T20:11:30.081 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:30.079+0000 7fd356878700 1 -- 172.21.15.160:0/2448502072 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd350045ad0 con 0x7fd3500f4860 2023-12-25T20:11:30.081 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:30.079+0000 7fd34cff9700 1 --2- 172.21.15.160:0/2448502072 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fd33802ff60 0x7fd338032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:11:30.081 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:30.079+0000 7fd34cff9700 1 -- 172.21.15.160:0/2448502072 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd340041080 con 0x7fd3500f4860 2023-12-25T20:11:30.081 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:30.080+0000 7fd34f7fe700 1 --2- 172.21.15.160:0/2448502072 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fd33802ff60 0x7fd338032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:11:30.081 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:30.080+0000 7fd34f7fe700 1 --2- 172.21.15.160:0/2448502072 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fd33802ff60 0x7fd338032420 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7fd3440096a0 tx=0x7fd344006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:11:30.084 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:30.084+0000 7fd34cff9700 1 -- 172.21.15.160:0/2448502072 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd340014420 con 0x7fd3500f4860 2023-12-25T20:11:30.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:11:29 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:11:28.662046+0000 mgr.smithi050.nbzhun (mgr.14162) 128 : cluster [DBG] pgmap v121: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:11:30.295 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:30.294+0000 7fd356878700 1 -- 172.21.15.160:0/2448502072 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd35005f370 con 0x7fd3500f4860 2023-12-25T20:11:30.295 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:30.294+0000 7fd34cff9700 1 -- 172.21.15.160:0/2448502072 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd340010230 con 0x7fd3500f4860 2023-12-25T20:11:30.296 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:11:30.296 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:11:30.297 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:30.296+0000 7fd356878700 1 -- 172.21.15.160:0/2448502072 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fd33802ff60 msgr2=0x7fd338032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:11:30.297 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:30.296+0000 7fd356878700 1 --2- 172.21.15.160:0/2448502072 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fd33802ff60 0x7fd338032420 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7fd3440096a0 tx=0x7fd344006b40).stop 2023-12-25T20:11:30.297 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:30.296+0000 7fd356878700 1 -- 172.21.15.160:0/2448502072 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd3500f4860 msgr2=0x7fd3501068e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:11:30.297 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:30.296+0000 7fd356878700 1 --2- 172.21.15.160:0/2448502072 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd3500f4860 0x7fd3501068e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7fd340013510 tx=0x7fd34000b040).stop 2023-12-25T20:11:30.297 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:30.296+0000 7fd356878700 1 -- 172.21.15.160:0/2448502072 shutdown_connections 2023-12-25T20:11:30.297 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:30.296+0000 7fd356878700 1 --2- 172.21.15.160:0/2448502072 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fd33802ff60 0x7fd338032420 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:11:30.297 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:30.296+0000 7fd356878700 1 --2- 172.21.15.160:0/2448502072 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd3500f4860 0x7fd3501068e0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:11:30.298 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:30.296+0000 7fd356878700 1 -- 172.21.15.160:0/2448502072 >> 172.21.15.160:0/2448502072 conn(0x7fd3500f02e0 msgr2=0x7fd3500f0e40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:11:30.298 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:30.296+0000 7fd356878700 1 -- 172.21.15.160:0/2448502072 shutdown_connections 2023-12-25T20:11:30.298 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:30.296+0000 7fd356878700 1 -- 172.21.15.160:0/2448502072 wait complete. 2023-12-25T20:11:30.298 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:11:31.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:11:30 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:11:30.296156+0000 mon.smithi050 (mon.0) 204 : audit [DBG] from='client.? 172.21.15.160:0/2448502072' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:11:31.710 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:11:31.710 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:11:32.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:11:31 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:11:30.662566+0000 mgr.smithi050.nbzhun (mgr.14162) 129 : cluster [DBG] pgmap v122: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:11:32.938 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:32.937+0000 7f22bdcdd700 1 -- 172.21.15.160:0/2626488617 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f22b80f51a0 msgr2=0x7f22b80f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:11:32.938 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:32.937+0000 7f22bdcdd700 1 --2- 172.21.15.160:0/2626488617 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f22b80f51a0 0x7f22b80f55c0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7f22ac009fb0 tx=0x7f22ac009b30).stop 2023-12-25T20:11:32.938 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:32.937+0000 7f22bdcdd700 1 -- 172.21.15.160:0/2626488617 shutdown_connections 2023-12-25T20:11:32.938 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:32.937+0000 7f22bdcdd700 1 --2- 172.21.15.160:0/2626488617 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f22b80f51a0 0x7f22b80f55c0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:11:32.938 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:32.937+0000 7f22bdcdd700 1 -- 172.21.15.160:0/2626488617 >> 172.21.15.160:0/2626488617 conn(0x7f22b80f09a0 msgr2=0x7f22b80f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:11:32.938 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:32.937+0000 7f22bdcdd700 1 -- 172.21.15.160:0/2626488617 shutdown_connections 2023-12-25T20:11:32.939 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:32.937+0000 7f22bdcdd700 1 -- 172.21.15.160:0/2626488617 wait complete. 2023-12-25T20:11:32.939 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:32.937+0000 7f22bdcdd700 1 Processor -- start 2023-12-25T20:11:32.939 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:32.937+0000 7f22bdcdd700 1 -- start start 2023-12-25T20:11:32.939 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:32.938+0000 7f22bdcdd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f22b80f51a0 0x7f22b81091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:11:32.939 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:32.938+0000 7f22bdcdd700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22b8060240 con 0x7f22b80f51a0 2023-12-25T20:11:32.940 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:32.938+0000 7f22bccdb700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f22b80f51a0 0x7f22b81091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:11:32.940 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:32.938+0000 7f22bccdb700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f22b80f51a0 0x7f22b81091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:41912/0 (socket says 172.21.15.160:41912) 2023-12-25T20:11:32.940 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:32.938+0000 7f22bccdb700 1 -- 172.21.15.160:0/3007840701 learned_addr learned my addr 172.21.15.160:0/3007840701 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:11:32.940 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:32.938+0000 7f22bccdb700 1 -- 172.21.15.160:0/3007840701 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22ac005040 con 0x7f22b80f51a0 2023-12-25T20:11:32.940 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:32.938+0000 7f22bccdb700 1 --2- 172.21.15.160:0/3007840701 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f22b80f51a0 0x7f22b81091b0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7f22ac013510 tx=0x7f22ac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:11:32.941 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:32.938+0000 7f22b57fa700 1 -- 172.21.15.160:0/3007840701 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f22ac009320 con 0x7f22b80f51a0 2023-12-25T20:11:32.941 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:32.939+0000 7f22bdcdd700 1 -- 172.21.15.160:0/3007840701 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22b8060440 con 0x7f22b80f51a0 2023-12-25T20:11:32.941 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:32.939+0000 7f22b57fa700 1 -- 172.21.15.160:0/3007840701 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f22ac013f00 con 0x7f22b80f51a0 2023-12-25T20:11:32.941 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:32.939+0000 7f22b57fa700 1 -- 172.21.15.160:0/3007840701 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f22ac014dc0 con 0x7f22b80f51a0 2023-12-25T20:11:32.941 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:32.939+0000 7f22bdcdd700 1 -- 172.21.15.160:0/3007840701 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22b8109860 con 0x7f22b80f51a0 2023-12-25T20:11:32.941 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:32.940+0000 7f22b57fa700 1 -- 172.21.15.160:0/3007840701 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f22ac0095f0 con 0x7f22b80f51a0 2023-12-25T20:11:32.941 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:32.939+0000 7f22bdcdd700 1 -- 172.21.15.160:0/3007840701 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22b8045ad0 con 0x7f22b80f51a0 2023-12-25T20:11:32.941 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:32.940+0000 7f22b57fa700 1 --2- 172.21.15.160:0/3007840701 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f22a802ff60 0x7f22a8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:11:32.941 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:32.940+0000 7f22b57fa700 1 -- 172.21.15.160:0/3007840701 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f22ac0410c0 con 0x7f22b80f51a0 2023-12-25T20:11:32.942 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:32.940+0000 7f22b7fff700 1 --2- 172.21.15.160:0/3007840701 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f22a802ff60 0x7f22a8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:11:32.942 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:32.940+0000 7f22b7fff700 1 --2- 172.21.15.160:0/3007840701 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f22a802ff60 0x7f22a8032420 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f22a40096a0 tx=0x7f22a4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:11:32.945 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:32.945+0000 7f22b57fa700 1 -- 172.21.15.160:0/3007840701 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f22ac014420 con 0x7f22b80f51a0 2023-12-25T20:11:33.149 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:33.149+0000 7f22bdcdd700 1 -- 172.21.15.160:0/3007840701 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f22b810a080 con 0x7f22b80f51a0 2023-12-25T20:11:33.150 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:33.149+0000 7f22b57fa700 1 -- 172.21.15.160:0/3007840701 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f22ac010230 con 0x7f22b80f51a0 2023-12-25T20:11:33.150 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:11:33.150 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:11:33.152 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:33.151+0000 7f22bdcdd700 1 -- 172.21.15.160:0/3007840701 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f22a802ff60 msgr2=0x7f22a8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:11:33.152 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:33.151+0000 7f22bdcdd700 1 --2- 172.21.15.160:0/3007840701 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f22a802ff60 0x7f22a8032420 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f22a40096a0 tx=0x7f22a4006b40).stop 2023-12-25T20:11:33.152 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:33.151+0000 7f22bdcdd700 1 -- 172.21.15.160:0/3007840701 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f22b80f51a0 msgr2=0x7f22b81091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:11:33.152 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:33.151+0000 7f22bdcdd700 1 --2- 172.21.15.160:0/3007840701 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f22b80f51a0 0x7f22b81091b0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7f22ac013510 tx=0x7f22ac00b040).stop 2023-12-25T20:11:33.152 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:33.151+0000 7f22bdcdd700 1 -- 172.21.15.160:0/3007840701 shutdown_connections 2023-12-25T20:11:33.152 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:33.151+0000 7f22bdcdd700 1 --2- 172.21.15.160:0/3007840701 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f22a802ff60 0x7f22a8032420 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:11:33.152 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:33.151+0000 7f22bdcdd700 1 --2- 172.21.15.160:0/3007840701 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f22b80f51a0 0x7f22b81091b0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:11:33.152 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:33.151+0000 7f22bdcdd700 1 -- 172.21.15.160:0/3007840701 >> 172.21.15.160:0/3007840701 conn(0x7f22b80f09a0 msgr2=0x7f22b80f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:11:33.152 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:33.151+0000 7f22bdcdd700 1 -- 172.21.15.160:0/3007840701 shutdown_connections 2023-12-25T20:11:33.152 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:33.151+0000 7f22bdcdd700 1 -- 172.21.15.160:0/3007840701 wait complete. 2023-12-25T20:11:33.153 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:11:34.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:11:33 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:11:32.662947+0000 mgr.smithi050.nbzhun (mgr.14162) 130 : cluster [DBG] pgmap v123: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:11:34.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:11:33 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:11:33.151160+0000 mon.smithi050 (mon.0) 205 : audit [DBG] from='client.? 172.21.15.160:0/3007840701' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:11:34.534 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:11:34.535 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:11:35.751 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:35.749+0000 7f5c18b2b700 1 -- 172.21.15.160:0/2874631967 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5c140f6510 msgr2=0x7f5c140f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:11:35.751 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:35.749+0000 7f5c18b2b700 1 --2- 172.21.15.160:0/2874631967 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5c140f6510 0x7f5c140f6930 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0x7f5c04009fb0 tx=0x7f5c04009b30).stop 2023-12-25T20:11:35.751 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:35.749+0000 7f5c18b2b700 1 -- 172.21.15.160:0/2874631967 shutdown_connections 2023-12-25T20:11:35.751 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:35.749+0000 7f5c18b2b700 1 --2- 172.21.15.160:0/2874631967 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5c140f6510 0x7f5c140f6930 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:11:35.751 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:35.749+0000 7f5c18b2b700 1 -- 172.21.15.160:0/2874631967 >> 172.21.15.160:0/2874631967 conn(0x7f5c140f1c70 msgr2=0x7f5c140f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:11:35.751 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:35.749+0000 7f5c18b2b700 1 -- 172.21.15.160:0/2874631967 shutdown_connections 2023-12-25T20:11:35.751 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:35.749+0000 7f5c18b2b700 1 -- 172.21.15.160:0/2874631967 wait complete. 2023-12-25T20:11:35.751 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:35.750+0000 7f5c18b2b700 1 Processor -- start 2023-12-25T20:11:35.752 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:35.750+0000 7f5c18b2b700 1 -- start start 2023-12-25T20:11:35.752 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:35.750+0000 7f5c18b2b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5c140f6510 0x7f5c14106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:11:35.752 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:35.750+0000 7f5c18b2b700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c14106c20 con 0x7f5c140f6510 2023-12-25T20:11:35.752 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:35.750+0000 7f5c1259c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5c140f6510 0x7f5c14106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:11:35.753 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:35.750+0000 7f5c1259c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5c140f6510 0x7f5c14106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:41930/0 (socket says 172.21.15.160:41930) 2023-12-25T20:11:35.754 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:35.750+0000 7f5c1259c700 1 -- 172.21.15.160:0/2375044934 learned_addr learned my addr 172.21.15.160:0/2375044934 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:11:35.754 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:35.751+0000 7f5c1259c700 1 -- 172.21.15.160:0/2375044934 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c04005040 con 0x7f5c140f6510 2023-12-25T20:11:35.754 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:35.751+0000 7f5c1259c700 1 --2- 172.21.15.160:0/2375044934 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5c140f6510 0x7f5c14106730 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7f5c04013510 tx=0x7f5c0400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:11:35.754 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:35.751+0000 7f5c02ffd700 1 -- 172.21.15.160:0/2375044934 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5c04009320 con 0x7f5c140f6510 2023-12-25T20:11:35.754 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:35.751+0000 7f5c02ffd700 1 -- 172.21.15.160:0/2375044934 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5c04013f00 con 0x7f5c140f6510 2023-12-25T20:11:35.754 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:35.751+0000 7f5c18b2b700 1 -- 172.21.15.160:0/2375044934 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c14106e20 con 0x7f5c140f6510 2023-12-25T20:11:35.754 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:35.751+0000 7f5c02ffd700 1 -- 172.21.15.160:0/2375044934 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5c04014d90 con 0x7f5c140f6510 2023-12-25T20:11:35.754 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:35.751+0000 7f5c18b2b700 1 -- 172.21.15.160:0/2375044934 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c141072b0 con 0x7f5c140f6510 2023-12-25T20:11:35.754 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:35.752+0000 7f5c18b2b700 1 -- 172.21.15.160:0/2375044934 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c14045ad0 con 0x7f5c140f6510 2023-12-25T20:11:35.754 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:35.752+0000 7f5c02ffd700 1 -- 172.21.15.160:0/2375044934 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5c040095f0 con 0x7f5c140f6510 2023-12-25T20:11:35.754 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:35.752+0000 7f5c02ffd700 1 --2- 172.21.15.160:0/2375044934 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f5bfc02ffb0 0x7f5bfc032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:11:35.754 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:35.752+0000 7f5c02ffd700 1 -- 172.21.15.160:0/2375044934 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5c04040fe0 con 0x7f5c140f6510 2023-12-25T20:11:35.754 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:35.752+0000 7f5c11d9b700 1 --2- 172.21.15.160:0/2375044934 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f5bfc02ffb0 0x7f5bfc032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:11:35.754 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:35.753+0000 7f5c11d9b700 1 --2- 172.21.15.160:0/2375044934 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f5bfc02ffb0 0x7f5bfc032470 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f5c080096a0 tx=0x7f5c08006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:11:35.757 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:35.757+0000 7f5c02ffd700 1 -- 172.21.15.160:0/2375044934 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5c0406d2b0 con 0x7f5c140f6510 2023-12-25T20:11:35.963 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:35.962+0000 7f5c18b2b700 1 -- 172.21.15.160:0/2375044934 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5c140fbf60 con 0x7f5c140f6510 2023-12-25T20:11:35.963 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:35.963+0000 7f5c02ffd700 1 -- 172.21.15.160:0/2375044934 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5c04014420 con 0x7f5c140f6510 2023-12-25T20:11:35.964 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:11:35.964 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:11:35.965 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:35.964+0000 7f5c18b2b700 1 -- 172.21.15.160:0/2375044934 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f5bfc02ffb0 msgr2=0x7f5bfc032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:11:35.965 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:35.964+0000 7f5c18b2b700 1 --2- 172.21.15.160:0/2375044934 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f5bfc02ffb0 0x7f5bfc032470 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f5c080096a0 tx=0x7f5c08006b40).stop 2023-12-25T20:11:35.965 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:35.964+0000 7f5c18b2b700 1 -- 172.21.15.160:0/2375044934 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5c140f6510 msgr2=0x7f5c14106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:11:35.965 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:35.964+0000 7f5c18b2b700 1 --2- 172.21.15.160:0/2375044934 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5c140f6510 0x7f5c14106730 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7f5c04013510 tx=0x7f5c0400b040).stop 2023-12-25T20:11:35.965 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:35.964+0000 7f5c18b2b700 1 -- 172.21.15.160:0/2375044934 shutdown_connections 2023-12-25T20:11:35.965 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:35.964+0000 7f5c18b2b700 1 --2- 172.21.15.160:0/2375044934 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f5bfc02ffb0 0x7f5bfc032470 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:11:35.965 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:35.964+0000 7f5c18b2b700 1 --2- 172.21.15.160:0/2375044934 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5c140f6510 0x7f5c14106730 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:11:35.966 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:35.964+0000 7f5c18b2b700 1 -- 172.21.15.160:0/2375044934 >> 172.21.15.160:0/2375044934 conn(0x7f5c140f1c70 msgr2=0x7f5c140f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:11:35.966 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:35.964+0000 7f5c18b2b700 1 -- 172.21.15.160:0/2375044934 shutdown_connections 2023-12-25T20:11:35.966 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:35.964+0000 7f5c18b2b700 1 -- 172.21.15.160:0/2375044934 wait complete. 2023-12-25T20:11:35.966 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:11:36.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:11:35 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:11:34.663423+0000 mgr.smithi050.nbzhun (mgr.14162) 131 : cluster [DBG] pgmap v124: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:11:37.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:11:36 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:11:35.964481+0000 mon.smithi050 (mon.0) 206 : audit [DBG] from='client.? 172.21.15.160:0/2375044934' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:11:37.359 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:11:37.359 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:11:38.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:11:37 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:11:36.663688+0000 mgr.smithi050.nbzhun (mgr.14162) 132 : cluster [DBG] pgmap v125: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:11:38.551 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:38.549+0000 7f28cd686700 1 -- 172.21.15.160:0/1093677732 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f28c80f51a0 msgr2=0x7f28c80f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:11:38.551 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:38.549+0000 7f28cd686700 1 --2- 172.21.15.160:0/1093677732 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f28c80f51a0 0x7f28c80f55c0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0x7f28b8009fb0 tx=0x7f28b8009b30).stop 2023-12-25T20:11:38.551 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:38.549+0000 7f28cd686700 1 -- 172.21.15.160:0/1093677732 shutdown_connections 2023-12-25T20:11:38.551 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:38.549+0000 7f28cd686700 1 --2- 172.21.15.160:0/1093677732 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f28c80f51a0 0x7f28c80f55c0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:11:38.552 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:38.549+0000 7f28cd686700 1 -- 172.21.15.160:0/1093677732 >> 172.21.15.160:0/1093677732 conn(0x7f28c80f09a0 msgr2=0x7f28c80f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:11:38.552 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:38.549+0000 7f28cd686700 1 -- 172.21.15.160:0/1093677732 shutdown_connections 2023-12-25T20:11:38.552 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:38.549+0000 7f28cd686700 1 -- 172.21.15.160:0/1093677732 wait complete. 2023-12-25T20:11:38.552 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:38.549+0000 7f28cd686700 1 Processor -- start 2023-12-25T20:11:38.552 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:38.549+0000 7f28cd686700 1 -- start start 2023-12-25T20:11:38.552 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:38.550+0000 7f28cd686700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f28c80f51a0 0x7f28c81091d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:11:38.552 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:38.550+0000 7f28cd686700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28c8060240 con 0x7f28c80f51a0 2023-12-25T20:11:38.552 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:38.550+0000 7f28c7fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f28c80f51a0 0x7f28c81091d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:11:38.552 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:38.550+0000 7f28c7fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f28c80f51a0 0x7f28c81091d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:39396/0 (socket says 172.21.15.160:39396) 2023-12-25T20:11:38.552 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:38.550+0000 7f28c7fff700 1 -- 172.21.15.160:0/617115056 learned_addr learned my addr 172.21.15.160:0/617115056 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:11:38.552 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:38.550+0000 7f28c7fff700 1 -- 172.21.15.160:0/617115056 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28b8005040 con 0x7f28c80f51a0 2023-12-25T20:11:38.552 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:38.550+0000 7f28c7fff700 1 --2- 172.21.15.160:0/617115056 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f28c80f51a0 0x7f28c81091d0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7f28b8009830 tx=0x7f28b800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:11:38.552 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:38.550+0000 7f28c4ff9700 1 -- 172.21.15.160:0/617115056 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f28b8009320 con 0x7f28c80f51a0 2023-12-25T20:11:38.552 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:38.550+0000 7f28c4ff9700 1 -- 172.21.15.160:0/617115056 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f28b80136c0 con 0x7f28c80f51a0 2023-12-25T20:11:38.553 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:38.550+0000 7f28cd686700 1 -- 172.21.15.160:0/617115056 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28c8060440 con 0x7f28c80f51a0 2023-12-25T20:11:38.553 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:38.550+0000 7f28c4ff9700 1 -- 172.21.15.160:0/617115056 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f28b801ee90 con 0x7f28c80f51a0 2023-12-25T20:11:38.553 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:38.550+0000 7f28cd686700 1 -- 172.21.15.160:0/617115056 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28c8109880 con 0x7f28c80f51a0 2023-12-25T20:11:38.554 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:38.551+0000 7f28c4ff9700 1 -- 172.21.15.160:0/617115056 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f28b801e420 con 0x7f28c80f51a0 2023-12-25T20:11:38.554 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:38.551+0000 7f28cd686700 1 -- 172.21.15.160:0/617115056 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28c8045ad0 con 0x7f28c80f51a0 2023-12-25T20:11:38.554 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:38.552+0000 7f28c4ff9700 1 --2- 172.21.15.160:0/617115056 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f28b002ff10 0x7f28b00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:11:38.554 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:38.552+0000 7f28c4ff9700 1 -- 172.21.15.160:0/617115056 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f28b8040f40 con 0x7f28c80f51a0 2023-12-25T20:11:38.554 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:38.552+0000 7f28c77fe700 1 --2- 172.21.15.160:0/617115056 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f28b002ff10 0x7f28b00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:11:38.554 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:38.552+0000 7f28c77fe700 1 --2- 172.21.15.160:0/617115056 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f28b002ff10 0x7f28b00323d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f28bc0096a0 tx=0x7f28bc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:11:38.557 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:38.557+0000 7f28c4ff9700 1 -- 172.21.15.160:0/617115056 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f28b806d2b0 con 0x7f28c80f51a0 2023-12-25T20:11:38.762 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:38.762+0000 7f28cd686700 1 -- 172.21.15.160:0/617115056 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f28c810a0a0 con 0x7f28c80f51a0 2023-12-25T20:11:38.763 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:38.762+0000 7f28c4ff9700 1 -- 172.21.15.160:0/617115056 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f28b80095f0 con 0x7f28c80f51a0 2023-12-25T20:11:38.763 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:11:38.763 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:11:38.764 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:38.764+0000 7f28cd686700 1 -- 172.21.15.160:0/617115056 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f28b002ff10 msgr2=0x7f28b00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:11:38.765 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:38.764+0000 7f28cd686700 1 --2- 172.21.15.160:0/617115056 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f28b002ff10 0x7f28b00323d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f28bc0096a0 tx=0x7f28bc006b40).stop 2023-12-25T20:11:38.765 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:38.764+0000 7f28cd686700 1 -- 172.21.15.160:0/617115056 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f28c80f51a0 msgr2=0x7f28c81091d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:11:38.765 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:38.764+0000 7f28cd686700 1 --2- 172.21.15.160:0/617115056 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f28c80f51a0 0x7f28c81091d0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7f28b8009830 tx=0x7f28b800b040).stop 2023-12-25T20:11:38.765 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:38.764+0000 7f28cd686700 1 -- 172.21.15.160:0/617115056 shutdown_connections 2023-12-25T20:11:38.765 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:38.764+0000 7f28cd686700 1 --2- 172.21.15.160:0/617115056 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f28b002ff10 0x7f28b00323d0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:11:38.765 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:38.764+0000 7f28cd686700 1 --2- 172.21.15.160:0/617115056 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f28c80f51a0 0x7f28c81091d0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:11:38.765 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:38.764+0000 7f28cd686700 1 -- 172.21.15.160:0/617115056 >> 172.21.15.160:0/617115056 conn(0x7f28c80f09a0 msgr2=0x7f28c80f1420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:11:38.765 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:38.764+0000 7f28cd686700 1 -- 172.21.15.160:0/617115056 shutdown_connections 2023-12-25T20:11:38.765 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:38.764+0000 7f28cd686700 1 -- 172.21.15.160:0/617115056 wait complete. 2023-12-25T20:11:38.766 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:11:39.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:11:38 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:11:38.763968+0000 mon.smithi050 (mon.0) 207 : audit [DBG] from='client.? 172.21.15.160:0/617115056' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:11:40.152 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:11:40.152 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:11:40.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:11:39 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:11:38.663937+0000 mgr.smithi050.nbzhun (mgr.14162) 133 : cluster [DBG] pgmap v126: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:11:41.427 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:41.426+0000 7fbaf2691700 1 -- 172.21.15.160:0/67423686 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbaec0f5410 msgr2=0x7fbaec0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:11:41.427 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:41.426+0000 7fbaf2691700 1 --2- 172.21.15.160:0/67423686 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbaec0f5410 0x7fbaec0f5830 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0x7fbae0009fb0 tx=0x7fbae0009b30).stop 2023-12-25T20:11:41.427 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:41.426+0000 7fbaf2691700 1 -- 172.21.15.160:0/67423686 shutdown_connections 2023-12-25T20:11:41.427 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:41.426+0000 7fbaf2691700 1 --2- 172.21.15.160:0/67423686 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbaec0f5410 0x7fbaec0f5830 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:11:41.427 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:41.426+0000 7fbaf2691700 1 -- 172.21.15.160:0/67423686 >> 172.21.15.160:0/67423686 conn(0x7fbaec0f0b70 msgr2=0x7fbaec0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:11:41.427 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:41.426+0000 7fbaf2691700 1 -- 172.21.15.160:0/67423686 shutdown_connections 2023-12-25T20:11:41.427 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:41.426+0000 7fbaf2691700 1 -- 172.21.15.160:0/67423686 wait complete. 2023-12-25T20:11:41.429 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:41.427+0000 7fbaf2691700 1 Processor -- start 2023-12-25T20:11:41.429 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:41.427+0000 7fbaf2691700 1 -- start start 2023-12-25T20:11:41.429 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:41.427+0000 7fbaf2691700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbaec0f5410 0x7fbaec106710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:11:41.429 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:41.427+0000 7fbaf2691700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaec106c00 con 0x7fbaec0f5410 2023-12-25T20:11:41.429 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:41.427+0000 7fbaebfff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbaec0f5410 0x7fbaec106710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:11:41.429 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:41.427+0000 7fbaebfff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbaec0f5410 0x7fbaec106710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:39414/0 (socket says 172.21.15.160:39414) 2023-12-25T20:11:41.429 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:41.427+0000 7fbaebfff700 1 -- 172.21.15.160:0/2540814300 learned_addr learned my addr 172.21.15.160:0/2540814300 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:11:41.429 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:41.428+0000 7fbaebfff700 1 -- 172.21.15.160:0/2540814300 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbae0005040 con 0x7fbaec0f5410 2023-12-25T20:11:41.429 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:41.428+0000 7fbaebfff700 1 --2- 172.21.15.160:0/2540814300 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbaec0f5410 0x7fbaec106710 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7fbae0013510 tx=0x7fbae000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:11:41.429 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:41.428+0000 7fbae8ff9700 1 -- 172.21.15.160:0/2540814300 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbae0009320 con 0x7fbaec0f5410 2023-12-25T20:11:41.429 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:41.428+0000 7fbaf2691700 1 -- 172.21.15.160:0/2540814300 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbaec106e00 con 0x7fbaec0f5410 2023-12-25T20:11:41.429 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:41.428+0000 7fbae8ff9700 1 -- 172.21.15.160:0/2540814300 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbae0013f00 con 0x7fbaec0f5410 2023-12-25T20:11:41.429 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:41.428+0000 7fbae8ff9700 1 -- 172.21.15.160:0/2540814300 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbae0014d90 con 0x7fbaec0f5410 2023-12-25T20:11:41.430 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:41.428+0000 7fbaf2691700 1 -- 172.21.15.160:0/2540814300 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbaec107290 con 0x7fbaec0f5410 2023-12-25T20:11:41.430 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:41.429+0000 7fbae8ff9700 1 -- 172.21.15.160:0/2540814300 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbae00263a0 con 0x7fbaec0f5410 2023-12-25T20:11:41.430 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:41.429+0000 7fbae8ff9700 1 --2- 172.21.15.160:0/2540814300 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fbad402ff60 0x7fbad4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:11:41.431 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:41.429+0000 7fbae8ff9700 1 -- 172.21.15.160:0/2540814300 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbae0041ff0 con 0x7fbaec0f5410 2023-12-25T20:11:41.431 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:41.429+0000 7fbaf2691700 1 -- 172.21.15.160:0/2540814300 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbaec045ad0 con 0x7fbaec0f5410 2023-12-25T20:11:41.431 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:41.429+0000 7fbaeb7fe700 1 --2- 172.21.15.160:0/2540814300 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fbad402ff60 0x7fbad4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:11:41.431 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:41.430+0000 7fbaeb7fe700 1 --2- 172.21.15.160:0/2540814300 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fbad402ff60 0x7fbad4032420 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7fbadc0096a0 tx=0x7fbadc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:11:41.435 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:41.434+0000 7fbae8ff9700 1 -- 172.21.15.160:0/2540814300 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbae0012920 con 0x7fbaec0f5410 2023-12-25T20:11:41.640 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:41.639+0000 7fbaf2691700 1 -- 172.21.15.160:0/2540814300 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbaec0fae60 con 0x7fbaec0f5410 2023-12-25T20:11:41.640 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:41.640+0000 7fbae8ff9700 1 -- 172.21.15.160:0/2540814300 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbae002b020 con 0x7fbaec0f5410 2023-12-25T20:11:41.641 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:11:41.641 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:11:41.643 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:41.642+0000 7fbaf2691700 1 -- 172.21.15.160:0/2540814300 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fbad402ff60 msgr2=0x7fbad4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:11:41.643 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:41.642+0000 7fbaf2691700 1 --2- 172.21.15.160:0/2540814300 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fbad402ff60 0x7fbad4032420 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7fbadc0096a0 tx=0x7fbadc006b40).stop 2023-12-25T20:11:41.643 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:41.642+0000 7fbaf2691700 1 -- 172.21.15.160:0/2540814300 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbaec0f5410 msgr2=0x7fbaec106710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:11:41.643 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:41.642+0000 7fbaf2691700 1 --2- 172.21.15.160:0/2540814300 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbaec0f5410 0x7fbaec106710 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7fbae0013510 tx=0x7fbae000b040).stop 2023-12-25T20:11:41.643 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:41.642+0000 7fbaf2691700 1 -- 172.21.15.160:0/2540814300 shutdown_connections 2023-12-25T20:11:41.643 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:41.642+0000 7fbaf2691700 1 --2- 172.21.15.160:0/2540814300 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fbad402ff60 0x7fbad4032420 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:11:41.643 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:41.642+0000 7fbaf2691700 1 --2- 172.21.15.160:0/2540814300 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbaec0f5410 0x7fbaec106710 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:11:41.643 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:41.642+0000 7fbaf2691700 1 -- 172.21.15.160:0/2540814300 >> 172.21.15.160:0/2540814300 conn(0x7fbaec0f0b70 msgr2=0x7fbaec05fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:11:41.643 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:41.642+0000 7fbaf2691700 1 -- 172.21.15.160:0/2540814300 shutdown_connections 2023-12-25T20:11:41.643 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:41.642+0000 7fbaf2691700 1 -- 172.21.15.160:0/2540814300 wait complete. 2023-12-25T20:11:41.644 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:11:42.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:11:41 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:11:40.664259+0000 mgr.smithi050.nbzhun (mgr.14162) 134 : cluster [DBG] pgmap v127: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:11:42.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:11:41 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:11:41.641811+0000 mon.smithi050 (mon.0) 208 : audit [DBG] from='client.? 172.21.15.160:0/2540814300' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:11:43.052 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:11:43.052 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:11:44.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:11:43 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:11:42.664695+0000 mgr.smithi050.nbzhun (mgr.14162) 135 : cluster [DBG] pgmap v128: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:11:44.285 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:44.282+0000 7fb314c35700 1 -- 172.21.15.160:0/3481525531 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb3100f7ae0 msgr2=0x7fb3100f7f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:11:44.285 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:44.282+0000 7fb314c35700 1 --2- 172.21.15.160:0/3481525531 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb3100f7ae0 0x7fb3100f7f00 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0x7fb304009fb0 tx=0x7fb304009b30).stop 2023-12-25T20:11:44.285 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:44.282+0000 7fb314c35700 1 -- 172.21.15.160:0/3481525531 shutdown_connections 2023-12-25T20:11:44.285 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:44.282+0000 7fb314c35700 1 --2- 172.21.15.160:0/3481525531 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb3100f7ae0 0x7fb3100f7f00 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:11:44.285 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:44.282+0000 7fb314c35700 1 -- 172.21.15.160:0/3481525531 >> 172.21.15.160:0/3481525531 conn(0x7fb3100f0260 msgr2=0x7fb3100f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:11:44.285 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:44.282+0000 7fb314c35700 1 -- 172.21.15.160:0/3481525531 shutdown_connections 2023-12-25T20:11:44.285 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:44.282+0000 7fb314c35700 1 -- 172.21.15.160:0/3481525531 wait complete. 2023-12-25T20:11:44.285 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:44.283+0000 7fb314c35700 1 Processor -- start 2023-12-25T20:11:44.285 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:44.283+0000 7fb314c35700 1 -- start start 2023-12-25T20:11:44.285 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:44.283+0000 7fb314c35700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb3100f7ae0 0x7fb310106710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:11:44.285 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:44.283+0000 7fb314c35700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb310106c00 con 0x7fb3100f7ae0 2023-12-25T20:11:44.286 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:44.283+0000 7fb30e59c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb3100f7ae0 0x7fb310106710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:11:44.286 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:44.283+0000 7fb30e59c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb3100f7ae0 0x7fb310106710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:39440/0 (socket says 172.21.15.160:39440) 2023-12-25T20:11:44.286 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:44.283+0000 7fb30e59c700 1 -- 172.21.15.160:0/2726940454 learned_addr learned my addr 172.21.15.160:0/2726940454 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:11:44.286 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:44.283+0000 7fb30e59c700 1 -- 172.21.15.160:0/2726940454 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb304005040 con 0x7fb3100f7ae0 2023-12-25T20:11:44.286 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:44.284+0000 7fb30e59c700 1 --2- 172.21.15.160:0/2726940454 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb3100f7ae0 0x7fb310106710 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7fb304013510 tx=0x7fb30400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:11:44.286 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:44.284+0000 7fb2feffd700 1 -- 172.21.15.160:0/2726940454 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb304009320 con 0x7fb3100f7ae0 2023-12-25T20:11:44.286 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:44.284+0000 7fb314c35700 1 -- 172.21.15.160:0/2726940454 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb310106e00 con 0x7fb3100f7ae0 2023-12-25T20:11:44.286 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:44.284+0000 7fb2feffd700 1 -- 172.21.15.160:0/2726940454 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb304013f00 con 0x7fb3100f7ae0 2023-12-25T20:11:44.286 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:44.284+0000 7fb2feffd700 1 -- 172.21.15.160:0/2726940454 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb304014d90 con 0x7fb3100f7ae0 2023-12-25T20:11:44.286 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:44.284+0000 7fb314c35700 1 -- 172.21.15.160:0/2726940454 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb310107290 con 0x7fb3100f7ae0 2023-12-25T20:11:44.287 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:44.285+0000 7fb2feffd700 1 -- 172.21.15.160:0/2726940454 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb3040095f0 con 0x7fb3100f7ae0 2023-12-25T20:11:44.287 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:44.285+0000 7fb314c35700 1 -- 172.21.15.160:0/2726940454 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb310045ad0 con 0x7fb3100f7ae0 2023-12-25T20:11:44.287 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:44.285+0000 7fb2feffd700 1 --2- 172.21.15.160:0/2726940454 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fb2f802ff60 0x7fb2f8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:11:44.287 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:44.285+0000 7fb2feffd700 1 -- 172.21.15.160:0/2726940454 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb304040fe0 con 0x7fb3100f7ae0 2023-12-25T20:11:44.287 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:44.285+0000 7fb30dd9b700 1 --2- 172.21.15.160:0/2726940454 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fb2f802ff60 0x7fb2f8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:11:44.287 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:44.285+0000 7fb30dd9b700 1 --2- 172.21.15.160:0/2726940454 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fb2f802ff60 0x7fb2f8032420 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7fb3000096a0 tx=0x7fb300006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:11:44.290 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:44.289+0000 7fb2feffd700 1 -- 172.21.15.160:0/2726940454 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb304014420 con 0x7fb3100f7ae0 2023-12-25T20:11:44.496 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:44.495+0000 7fb314c35700 1 -- 172.21.15.160:0/2726940454 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb3100fbfc0 con 0x7fb3100f7ae0 2023-12-25T20:11:44.496 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:44.496+0000 7fb2feffd700 1 -- 172.21.15.160:0/2726940454 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb304010230 con 0x7fb3100f7ae0 2023-12-25T20:11:44.497 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:11:44.497 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:11:44.498 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:44.497+0000 7fb314c35700 1 -- 172.21.15.160:0/2726940454 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fb2f802ff60 msgr2=0x7fb2f8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:11:44.498 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:44.497+0000 7fb314c35700 1 --2- 172.21.15.160:0/2726940454 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fb2f802ff60 0x7fb2f8032420 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7fb3000096a0 tx=0x7fb300006b40).stop 2023-12-25T20:11:44.498 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:44.497+0000 7fb314c35700 1 -- 172.21.15.160:0/2726940454 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb3100f7ae0 msgr2=0x7fb310106710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:11:44.498 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:44.497+0000 7fb314c35700 1 --2- 172.21.15.160:0/2726940454 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb3100f7ae0 0x7fb310106710 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7fb304013510 tx=0x7fb30400b040).stop 2023-12-25T20:11:44.498 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:44.497+0000 7fb314c35700 1 -- 172.21.15.160:0/2726940454 shutdown_connections 2023-12-25T20:11:44.498 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:44.497+0000 7fb314c35700 1 --2- 172.21.15.160:0/2726940454 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fb2f802ff60 0x7fb2f8032420 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:11:44.498 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:44.497+0000 7fb314c35700 1 --2- 172.21.15.160:0/2726940454 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb3100f7ae0 0x7fb310106710 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:11:44.499 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:44.497+0000 7fb314c35700 1 -- 172.21.15.160:0/2726940454 >> 172.21.15.160:0/2726940454 conn(0x7fb3100f0260 msgr2=0x7fb3100f0d50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:11:44.499 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:44.497+0000 7fb314c35700 1 -- 172.21.15.160:0/2726940454 shutdown_connections 2023-12-25T20:11:44.499 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:44.497+0000 7fb314c35700 1 -- 172.21.15.160:0/2726940454 wait complete. 2023-12-25T20:11:44.499 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:11:45.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:11:44 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:11:44.497485+0000 mon.smithi050 (mon.0) 209 : audit [DBG] from='client.? 172.21.15.160:0/2726940454' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:11:45.890 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:11:45.891 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:11:46.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:11:45 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:11:44.665064+0000 mgr.smithi050.nbzhun (mgr.14162) 136 : cluster [DBG] pgmap v129: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:11:47.105 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:47.102+0000 7fb50b7c7700 1 -- 172.21.15.160:0/2650669810 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb5040f5410 msgr2=0x7fb5040f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:11:47.105 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:47.102+0000 7fb50b7c7700 1 --2- 172.21.15.160:0/2650669810 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb5040f5410 0x7fb5040f5830 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0x7fb4f4009fb0 tx=0x7fb4f4009b30).stop 2023-12-25T20:11:47.105 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:47.102+0000 7fb50b7c7700 1 -- 172.21.15.160:0/2650669810 shutdown_connections 2023-12-25T20:11:47.105 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:47.102+0000 7fb50b7c7700 1 --2- 172.21.15.160:0/2650669810 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb5040f5410 0x7fb5040f5830 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:11:47.105 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:47.102+0000 7fb50b7c7700 1 -- 172.21.15.160:0/2650669810 >> 172.21.15.160:0/2650669810 conn(0x7fb5040f0b70 msgr2=0x7fb5040f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:11:47.105 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:47.102+0000 7fb50b7c7700 1 -- 172.21.15.160:0/2650669810 shutdown_connections 2023-12-25T20:11:47.106 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:47.102+0000 7fb50b7c7700 1 -- 172.21.15.160:0/2650669810 wait complete. 2023-12-25T20:11:47.106 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:47.103+0000 7fb50b7c7700 1 Processor -- start 2023-12-25T20:11:47.106 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:47.103+0000 7fb50b7c7700 1 -- start start 2023-12-25T20:11:47.106 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:47.103+0000 7fb50b7c7700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb5040f5410 0x7fb5040fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:11:47.106 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:47.103+0000 7fb50b7c7700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5040fb820 con 0x7fb5040f5410 2023-12-25T20:11:47.106 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:47.103+0000 7fb509563700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb5040f5410 0x7fb5040fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:11:47.106 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:47.103+0000 7fb509563700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb5040f5410 0x7fb5040fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:39452/0 (socket says 172.21.15.160:39452) 2023-12-25T20:11:47.106 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:47.103+0000 7fb509563700 1 -- 172.21.15.160:0/2661211135 learned_addr learned my addr 172.21.15.160:0/2661211135 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:11:47.106 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:47.104+0000 7fb509563700 1 -- 172.21.15.160:0/2661211135 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4f4005040 con 0x7fb5040f5410 2023-12-25T20:11:47.106 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:47.104+0000 7fb509563700 1 --2- 172.21.15.160:0/2661211135 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb5040f5410 0x7fb5040fd3a0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7fb4f4013510 tx=0x7fb4f400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:11:47.106 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:47.104+0000 7fb4f9ffb700 1 -- 172.21.15.160:0/2661211135 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb4f4009320 con 0x7fb5040f5410 2023-12-25T20:11:47.106 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:47.104+0000 7fb50b7c7700 1 -- 172.21.15.160:0/2661211135 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5040fb9c0 con 0x7fb5040f5410 2023-12-25T20:11:47.106 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:47.104+0000 7fb4f9ffb700 1 -- 172.21.15.160:0/2661211135 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb4f4013f00 con 0x7fb5040f5410 2023-12-25T20:11:47.106 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:47.104+0000 7fb4f9ffb700 1 -- 172.21.15.160:0/2661211135 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb4f4014d90 con 0x7fb5040f5410 2023-12-25T20:11:47.107 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:47.104+0000 7fb50b7c7700 1 -- 172.21.15.160:0/2661211135 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5040fbe50 con 0x7fb5040f5410 2023-12-25T20:11:47.108 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:47.105+0000 7fb4f9ffb700 1 -- 172.21.15.160:0/2661211135 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb4f40095f0 con 0x7fb5040f5410 2023-12-25T20:11:47.108 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:47.105+0000 7fb50b7c7700 1 -- 172.21.15.160:0/2661211135 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb504059100 con 0x7fb5040f5410 2023-12-25T20:11:47.108 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:47.105+0000 7fb4f9ffb700 1 --2- 172.21.15.160:0/2661211135 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fb4f002ff60 0x7fb4f0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:11:47.108 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:47.105+0000 7fb4f9ffb700 1 -- 172.21.15.160:0/2661211135 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb4f4041040 con 0x7fb5040f5410 2023-12-25T20:11:47.108 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:47.105+0000 7fb508d62700 1 --2- 172.21.15.160:0/2661211135 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fb4f002ff60 0x7fb4f0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:11:47.108 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:47.106+0000 7fb508d62700 1 --2- 172.21.15.160:0/2661211135 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fb4f002ff60 0x7fb4f0032420 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7fb5000096a0 tx=0x7fb500006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:11:47.110 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:47.110+0000 7fb4f9ffb700 1 -- 172.21.15.160:0/2661211135 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb4f4014420 con 0x7fb5040f5410 2023-12-25T20:11:47.317 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:47.316+0000 7fb50b7c7700 1 -- 172.21.15.160:0/2661211135 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb504045ad0 con 0x7fb5040f5410 2023-12-25T20:11:47.318 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:47.317+0000 7fb4f9ffb700 1 -- 172.21.15.160:0/2661211135 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb4f4010230 con 0x7fb5040f5410 2023-12-25T20:11:47.318 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:11:47.319 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:11:47.320 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:47.318+0000 7fb50b7c7700 1 -- 172.21.15.160:0/2661211135 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fb4f002ff60 msgr2=0x7fb4f0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:11:47.320 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:47.319+0000 7fb50b7c7700 1 --2- 172.21.15.160:0/2661211135 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fb4f002ff60 0x7fb4f0032420 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7fb5000096a0 tx=0x7fb500006b40).stop 2023-12-25T20:11:47.320 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:47.319+0000 7fb50b7c7700 1 -- 172.21.15.160:0/2661211135 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb5040f5410 msgr2=0x7fb5040fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:11:47.321 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:47.319+0000 7fb50b7c7700 1 --2- 172.21.15.160:0/2661211135 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb5040f5410 0x7fb5040fd3a0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7fb4f4013510 tx=0x7fb4f400b040).stop 2023-12-25T20:11:47.321 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:47.319+0000 7fb50b7c7700 1 -- 172.21.15.160:0/2661211135 shutdown_connections 2023-12-25T20:11:47.321 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:47.319+0000 7fb50b7c7700 1 --2- 172.21.15.160:0/2661211135 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fb4f002ff60 0x7fb4f0032420 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:11:47.321 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:47.319+0000 7fb50b7c7700 1 --2- 172.21.15.160:0/2661211135 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb5040f5410 0x7fb5040fd3a0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:11:47.321 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:47.319+0000 7fb50b7c7700 1 -- 172.21.15.160:0/2661211135 >> 172.21.15.160:0/2661211135 conn(0x7fb5040f0b70 msgr2=0x7fb50405fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:11:47.321 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:47.319+0000 7fb50b7c7700 1 -- 172.21.15.160:0/2661211135 shutdown_connections 2023-12-25T20:11:47.322 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:47.319+0000 7fb50b7c7700 1 -- 172.21.15.160:0/2661211135 wait complete. 2023-12-25T20:11:47.322 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:11:48.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:11:47 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:11:46.665530+0000 mgr.smithi050.nbzhun (mgr.14162) 137 : cluster [DBG] pgmap v130: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:11:48.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:11:47 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:11:47.318725+0000 mon.smithi050 (mon.0) 210 : audit [DBG] from='client.? 172.21.15.160:0/2661211135' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:11:48.713 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:11:48.714 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:11:49.996 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:49.994+0000 7fbcae90f700 1 -- 172.21.15.160:0/3764280000 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbca8060220 msgr2=0x7fbca80f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:11:49.996 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:49.994+0000 7fbcae90f700 1 --2- 172.21.15.160:0/3764280000 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbca8060220 0x7fbca80f9150 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 rx=0x7fbc98009fb0 tx=0x7fbc98009b30).stop 2023-12-25T20:11:49.996 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:49.994+0000 7fbcae90f700 1 -- 172.21.15.160:0/3764280000 shutdown_connections 2023-12-25T20:11:49.996 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:49.994+0000 7fbcae90f700 1 --2- 172.21.15.160:0/3764280000 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbca8060220 0x7fbca80f9150 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:11:49.996 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:49.994+0000 7fbcae90f700 1 -- 172.21.15.160:0/3764280000 >> 172.21.15.160:0/3764280000 conn(0x7fbca80f0b70 msgr2=0x7fbca80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:11:49.997 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:49.994+0000 7fbcae90f700 1 -- 172.21.15.160:0/3764280000 shutdown_connections 2023-12-25T20:11:49.997 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:49.994+0000 7fbcae90f700 1 -- 172.21.15.160:0/3764280000 wait complete. 2023-12-25T20:11:49.997 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:49.995+0000 7fbcae90f700 1 Processor -- start 2023-12-25T20:11:49.997 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:49.995+0000 7fbcae90f700 1 -- start start 2023-12-25T20:11:49.997 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:49.995+0000 7fbcae90f700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbca8060220 0x7fbca81066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:11:49.997 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:49.995+0000 7fbcae90f700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbca8106bd0 con 0x7fbca8060220 2023-12-25T20:11:49.997 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:49.996+0000 7fbca7fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbca8060220 0x7fbca81066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:11:49.997 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:49.996+0000 7fbca7fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbca8060220 0x7fbca81066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:33100/0 (socket says 172.21.15.160:33100) 2023-12-25T20:11:49.997 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:49.996+0000 7fbca7fff700 1 -- 172.21.15.160:0/2260599144 learned_addr learned my addr 172.21.15.160:0/2260599144 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:11:49.999 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:49.996+0000 7fbca7fff700 1 -- 172.21.15.160:0/2260599144 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc98005040 con 0x7fbca8060220 2023-12-25T20:11:49.999 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:49.996+0000 7fbca7fff700 1 --2- 172.21.15.160:0/2260599144 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbca8060220 0x7fbca81066e0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7fbc98013510 tx=0x7fbc9800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:11:49.999 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:49.996+0000 7fbca4ff9700 1 -- 172.21.15.160:0/2260599144 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbc98009320 con 0x7fbca8060220 2023-12-25T20:11:49.999 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:49.996+0000 7fbcae90f700 1 -- 172.21.15.160:0/2260599144 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbca8106dd0 con 0x7fbca8060220 2023-12-25T20:11:49.999 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:49.996+0000 7fbca4ff9700 1 -- 172.21.15.160:0/2260599144 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbc98013f00 con 0x7fbca8060220 2023-12-25T20:11:49.999 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:49.996+0000 7fbca4ff9700 1 -- 172.21.15.160:0/2260599144 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbc98014dc0 con 0x7fbca8060220 2023-12-25T20:11:49.999 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:49.996+0000 7fbcae90f700 1 -- 172.21.15.160:0/2260599144 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbca8107260 con 0x7fbca8060220 2023-12-25T20:11:49.999 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:49.997+0000 7fbca4ff9700 1 -- 172.21.15.160:0/2260599144 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbc9801d3d0 con 0x7fbca8060220 2023-12-25T20:11:49.999 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:49.997+0000 7fbcae90f700 1 -- 172.21.15.160:0/2260599144 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbca8045ad0 con 0x7fbca8060220 2023-12-25T20:11:49.999 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:49.997+0000 7fbca4ff9700 1 --2- 172.21.15.160:0/2260599144 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fbc9002ff60 0x7fbc90032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:11:49.999 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:49.997+0000 7fbca4ff9700 1 -- 172.21.15.160:0/2260599144 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbc980410d0 con 0x7fbca8060220 2023-12-25T20:11:49.999 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:49.997+0000 7fbca77fe700 1 --2- 172.21.15.160:0/2260599144 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fbc9002ff60 0x7fbc90032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:11:50.000 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:49.998+0000 7fbca77fe700 1 --2- 172.21.15.160:0/2260599144 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fbc9002ff60 0x7fbc90032420 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7fbc9c0096a0 tx=0x7fbc9c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:11:50.002 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:50.002+0000 7fbca4ff9700 1 -- 172.21.15.160:0/2260599144 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbc980095f0 con 0x7fbca8060220 2023-12-25T20:11:50.206 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:50.205+0000 7fbcae90f700 1 -- 172.21.15.160:0/2260599144 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbca8108110 con 0x7fbca8060220 2023-12-25T20:11:50.206 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:50.206+0000 7fbca4ff9700 1 -- 172.21.15.160:0/2260599144 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbc9801cb10 con 0x7fbca8060220 2023-12-25T20:11:50.207 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:11:50.207 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:11:50.208 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:50.207+0000 7fbcae90f700 1 -- 172.21.15.160:0/2260599144 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fbc9002ff60 msgr2=0x7fbc90032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:11:50.208 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:50.207+0000 7fbcae90f700 1 --2- 172.21.15.160:0/2260599144 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fbc9002ff60 0x7fbc90032420 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7fbc9c0096a0 tx=0x7fbc9c006b40).stop 2023-12-25T20:11:50.208 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:50.207+0000 7fbcae90f700 1 -- 172.21.15.160:0/2260599144 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbca8060220 msgr2=0x7fbca81066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:11:50.208 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:50.207+0000 7fbcae90f700 1 --2- 172.21.15.160:0/2260599144 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbca8060220 0x7fbca81066e0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7fbc98013510 tx=0x7fbc9800b040).stop 2023-12-25T20:11:50.209 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:50.207+0000 7fbcae90f700 1 -- 172.21.15.160:0/2260599144 shutdown_connections 2023-12-25T20:11:50.209 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:50.207+0000 7fbcae90f700 1 --2- 172.21.15.160:0/2260599144 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fbc9002ff60 0x7fbc90032420 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:11:50.209 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:50.207+0000 7fbcae90f700 1 --2- 172.21.15.160:0/2260599144 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbca8060220 0x7fbca81066e0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:11:50.209 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:50.207+0000 7fbcae90f700 1 -- 172.21.15.160:0/2260599144 >> 172.21.15.160:0/2260599144 conn(0x7fbca80f0b70 msgr2=0x7fbca80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:11:50.209 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:50.207+0000 7fbcae90f700 1 -- 172.21.15.160:0/2260599144 shutdown_connections 2023-12-25T20:11:50.209 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:50.207+0000 7fbcae90f700 1 -- 172.21.15.160:0/2260599144 wait complete. 2023-12-25T20:11:50.209 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:11:50.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:11:49 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:11:48.666006+0000 mgr.smithi050.nbzhun (mgr.14162) 138 : cluster [DBG] pgmap v131: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:11:51.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:11:50 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:11:50.207555+0000 mon.smithi050 (mon.0) 211 : audit [DBG] from='client.? 172.21.15.160:0/2260599144' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:11:51.590 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:11:51.590 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:11:52.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:11:51 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:11:50.666438+0000 mgr.smithi050.nbzhun (mgr.14162) 139 : cluster [DBG] pgmap v132: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:11:52.827 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:52.826+0000 7f1bd0af1700 1 -- 172.21.15.160:0/1274797052 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1bcc0f4740 msgr2=0x7f1bcc0f4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:11:52.827 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:52.826+0000 7f1bd0af1700 1 --2- 172.21.15.160:0/1274797052 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1bcc0f4740 0x7f1bcc0f4b60 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0x7f1bc0009fb0 tx=0x7f1bc0009b30).stop 2023-12-25T20:11:52.827 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:52.826+0000 7f1bd0af1700 1 -- 172.21.15.160:0/1274797052 shutdown_connections 2023-12-25T20:11:52.827 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:52.826+0000 7f1bd0af1700 1 --2- 172.21.15.160:0/1274797052 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1bcc0f4740 0x7f1bcc0f4b60 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:11:52.828 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:52.826+0000 7f1bd0af1700 1 -- 172.21.15.160:0/1274797052 >> 172.21.15.160:0/1274797052 conn(0x7f1bcc0f0260 msgr2=0x7f1bcc0f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:11:52.828 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:52.826+0000 7f1bd0af1700 1 -- 172.21.15.160:0/1274797052 shutdown_connections 2023-12-25T20:11:52.828 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:52.826+0000 7f1bd0af1700 1 -- 172.21.15.160:0/1274797052 wait complete. 2023-12-25T20:11:52.829 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:52.827+0000 7f1bd0af1700 1 Processor -- start 2023-12-25T20:11:52.829 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:52.827+0000 7f1bd0af1700 1 -- start start 2023-12-25T20:11:52.829 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:52.827+0000 7f1bd0af1700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1bcc0f4740 0x7f1bcc1066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:11:52.829 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:52.827+0000 7f1bd0af1700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bcc106bc0 con 0x7f1bcc0f4740 2023-12-25T20:11:52.829 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:52.827+0000 7f1bca59c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1bcc0f4740 0x7f1bcc1066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:11:52.829 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:52.827+0000 7f1bca59c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1bcc0f4740 0x7f1bcc1066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:33118/0 (socket says 172.21.15.160:33118) 2023-12-25T20:11:52.830 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:52.827+0000 7f1bca59c700 1 -- 172.21.15.160:0/3599411517 learned_addr learned my addr 172.21.15.160:0/3599411517 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:11:52.830 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:52.828+0000 7f1bca59c700 1 -- 172.21.15.160:0/3599411517 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bc0005040 con 0x7f1bcc0f4740 2023-12-25T20:11:52.830 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:52.828+0000 7f1bca59c700 1 --2- 172.21.15.160:0/3599411517 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1bcc0f4740 0x7f1bcc1066d0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7f1bc0013510 tx=0x7f1bc000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:11:52.830 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:52.828+0000 7f1bbaffd700 1 -- 172.21.15.160:0/3599411517 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1bc0009320 con 0x7f1bcc0f4740 2023-12-25T20:11:52.830 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:52.828+0000 7f1bd0af1700 1 -- 172.21.15.160:0/3599411517 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bcc106dc0 con 0x7f1bcc0f4740 2023-12-25T20:11:52.830 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:52.828+0000 7f1bbaffd700 1 -- 172.21.15.160:0/3599411517 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1bc0013f00 con 0x7f1bcc0f4740 2023-12-25T20:11:52.830 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:52.828+0000 7f1bbaffd700 1 -- 172.21.15.160:0/3599411517 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1bc0014d90 con 0x7f1bcc0f4740 2023-12-25T20:11:52.830 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:52.828+0000 7f1bd0af1700 1 -- 172.21.15.160:0/3599411517 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bcc107250 con 0x7f1bcc0f4740 2023-12-25T20:11:52.831 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:52.829+0000 7f1bd0af1700 1 -- 172.21.15.160:0/3599411517 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bcc045ad0 con 0x7f1bcc0f4740 2023-12-25T20:11:52.831 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:52.829+0000 7f1bbaffd700 1 -- 172.21.15.160:0/3599411517 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1bc00095f0 con 0x7f1bcc0f4740 2023-12-25T20:11:52.831 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:52.829+0000 7f1bbaffd700 1 --2- 172.21.15.160:0/3599411517 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f1bb402ff60 0x7f1bb4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:11:52.831 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:52.829+0000 7f1bbaffd700 1 -- 172.21.15.160:0/3599411517 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1bc0040fe0 con 0x7f1bcc0f4740 2023-12-25T20:11:52.831 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:52.829+0000 7f1bc9d9b700 1 --2- 172.21.15.160:0/3599411517 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f1bb402ff60 0x7f1bb4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:11:52.831 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:52.830+0000 7f1bc9d9b700 1 --2- 172.21.15.160:0/3599411517 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f1bb402ff60 0x7f1bb4032420 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f1bbc0096a0 tx=0x7f1bbc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:11:52.834 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:52.834+0000 7f1bbaffd700 1 -- 172.21.15.160:0/3599411517 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1bc0014420 con 0x7f1bcc0f4740 2023-12-25T20:11:53.038 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:53.038+0000 7f1bd0af1700 1 -- 172.21.15.160:0/3599411517 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1bcc1080a0 con 0x7f1bcc0f4740 2023-12-25T20:11:53.039 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:53.038+0000 7f1bbaffd700 1 -- 172.21.15.160:0/3599411517 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1bc0010230 con 0x7f1bcc0f4740 2023-12-25T20:11:53.039 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:11:53.039 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:11:53.042 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:53.040+0000 7f1bd0af1700 1 -- 172.21.15.160:0/3599411517 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f1bb402ff60 msgr2=0x7f1bb4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:11:53.042 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:53.040+0000 7f1bd0af1700 1 --2- 172.21.15.160:0/3599411517 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f1bb402ff60 0x7f1bb4032420 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f1bbc0096a0 tx=0x7f1bbc006b40).stop 2023-12-25T20:11:53.042 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:53.040+0000 7f1bd0af1700 1 -- 172.21.15.160:0/3599411517 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1bcc0f4740 msgr2=0x7f1bcc1066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:11:53.042 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:53.040+0000 7f1bd0af1700 1 --2- 172.21.15.160:0/3599411517 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1bcc0f4740 0x7f1bcc1066d0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7f1bc0013510 tx=0x7f1bc000b040).stop 2023-12-25T20:11:53.042 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:53.040+0000 7f1bd0af1700 1 -- 172.21.15.160:0/3599411517 shutdown_connections 2023-12-25T20:11:53.042 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:53.040+0000 7f1bd0af1700 1 --2- 172.21.15.160:0/3599411517 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f1bb402ff60 0x7f1bb4032420 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:11:53.042 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:53.040+0000 7f1bd0af1700 1 --2- 172.21.15.160:0/3599411517 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1bcc0f4740 0x7f1bcc1066d0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:11:53.042 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:53.040+0000 7f1bd0af1700 1 -- 172.21.15.160:0/3599411517 >> 172.21.15.160:0/3599411517 conn(0x7f1bcc0f0260 msgr2=0x7f1bcc0f0d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:11:53.042 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:53.040+0000 7f1bd0af1700 1 -- 172.21.15.160:0/3599411517 shutdown_connections 2023-12-25T20:11:53.042 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:53.040+0000 7f1bd0af1700 1 -- 172.21.15.160:0/3599411517 wait complete. 2023-12-25T20:11:53.042 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:11:54.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:11:53 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:11:52.666843+0000 mgr.smithi050.nbzhun (mgr.14162) 140 : cluster [DBG] pgmap v133: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:11:54.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:11:53 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:11:53.040162+0000 mon.smithi050 (mon.0) 212 : audit [DBG] from='client.? 172.21.15.160:0/3599411517' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:11:54.431 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:11:54.431 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:11:55.700 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:55.699+0000 7f0d9234d700 1 -- 172.21.15.160:0/116516440 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0d8c0f5410 msgr2=0x7f0d8c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:11:55.700 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:55.699+0000 7f0d9234d700 1 --2- 172.21.15.160:0/116516440 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0d8c0f5410 0x7f0d8c0f5830 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 rx=0x7f0d7c009fb0 tx=0x7f0d7c009b30).stop 2023-12-25T20:11:55.700 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:55.699+0000 7f0d9234d700 1 -- 172.21.15.160:0/116516440 shutdown_connections 2023-12-25T20:11:55.700 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:55.699+0000 7f0d9234d700 1 --2- 172.21.15.160:0/116516440 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0d8c0f5410 0x7f0d8c0f5830 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:11:55.700 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:55.699+0000 7f0d9234d700 1 -- 172.21.15.160:0/116516440 >> 172.21.15.160:0/116516440 conn(0x7f0d8c0f0b70 msgr2=0x7f0d8c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:11:55.700 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:55.699+0000 7f0d9234d700 1 -- 172.21.15.160:0/116516440 shutdown_connections 2023-12-25T20:11:55.701 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:55.699+0000 7f0d9234d700 1 -- 172.21.15.160:0/116516440 wait complete. 2023-12-25T20:11:55.701 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:55.700+0000 7f0d9234d700 1 Processor -- start 2023-12-25T20:11:55.702 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:55.700+0000 7f0d9234d700 1 -- start start 2023-12-25T20:11:55.702 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:55.700+0000 7f0d9234d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0d8c0f5410 0x7f0d8c1055d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:11:55.702 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:55.700+0000 7f0d9234d700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d8c105ac0 con 0x7f0d8c0f5410 2023-12-25T20:11:55.702 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:55.700+0000 7f0d8bfff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0d8c0f5410 0x7f0d8c1055d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:11:55.702 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:55.700+0000 7f0d8bfff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0d8c0f5410 0x7f0d8c1055d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:33136/0 (socket says 172.21.15.160:33136) 2023-12-25T20:11:55.702 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:55.700+0000 7f0d8bfff700 1 -- 172.21.15.160:0/3026216023 learned_addr learned my addr 172.21.15.160:0/3026216023 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:11:55.702 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:55.701+0000 7f0d8bfff700 1 -- 172.21.15.160:0/3026216023 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d7c005040 con 0x7f0d8c0f5410 2023-12-25T20:11:55.702 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:55.701+0000 7f0d8bfff700 1 --2- 172.21.15.160:0/3026216023 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0d8c0f5410 0x7f0d8c1055d0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7f0d7c013510 tx=0x7f0d7c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:11:55.702 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:55.701+0000 7f0d88ff9700 1 -- 172.21.15.160:0/3026216023 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0d7c009320 con 0x7f0d8c0f5410 2023-12-25T20:11:55.703 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:55.701+0000 7f0d9234d700 1 -- 172.21.15.160:0/3026216023 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d8c105cc0 con 0x7f0d8c0f5410 2023-12-25T20:11:55.703 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:55.701+0000 7f0d88ff9700 1 -- 172.21.15.160:0/3026216023 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0d7c013f00 con 0x7f0d8c0f5410 2023-12-25T20:11:55.703 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:55.701+0000 7f0d88ff9700 1 -- 172.21.15.160:0/3026216023 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0d7c014d90 con 0x7f0d8c0f5410 2023-12-25T20:11:55.703 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:55.701+0000 7f0d9234d700 1 -- 172.21.15.160:0/3026216023 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d8c106150 con 0x7f0d8c0f5410 2023-12-25T20:11:55.703 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:55.702+0000 7f0d88ff9700 1 -- 172.21.15.160:0/3026216023 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0d7c0095f0 con 0x7f0d8c0f5410 2023-12-25T20:11:55.704 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:55.702+0000 7f0d9234d700 1 -- 172.21.15.160:0/3026216023 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d8c045ad0 con 0x7f0d8c0f5410 2023-12-25T20:11:55.704 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:55.702+0000 7f0d88ff9700 1 --2- 172.21.15.160:0/3026216023 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f0d7402ff60 0x7f0d74032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:11:55.704 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:55.702+0000 7f0d88ff9700 1 -- 172.21.15.160:0/3026216023 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0d7c0410c0 con 0x7f0d8c0f5410 2023-12-25T20:11:55.704 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:55.702+0000 7f0d8b7fe700 1 --2- 172.21.15.160:0/3026216023 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f0d7402ff60 0x7f0d74032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:11:55.704 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:55.703+0000 7f0d8b7fe700 1 --2- 172.21.15.160:0/3026216023 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f0d7402ff60 0x7f0d74032420 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f0d800096a0 tx=0x7f0d80006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:11:55.707 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:55.707+0000 7f0d88ff9700 1 -- 172.21.15.160:0/3026216023 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0d7c014420 con 0x7f0d8c0f5410 2023-12-25T20:11:55.911 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:55.910+0000 7f0d9234d700 1 -- 172.21.15.160:0/3026216023 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0d8c107000 con 0x7f0d8c0f5410 2023-12-25T20:11:55.911 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:55.911+0000 7f0d88ff9700 1 -- 172.21.15.160:0/3026216023 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0d7c010230 con 0x7f0d8c0f5410 2023-12-25T20:11:55.912 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:11:55.912 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:11:55.913 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:55.912+0000 7f0d9234d700 1 -- 172.21.15.160:0/3026216023 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f0d7402ff60 msgr2=0x7f0d74032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:11:55.913 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:55.912+0000 7f0d9234d700 1 --2- 172.21.15.160:0/3026216023 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f0d7402ff60 0x7f0d74032420 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f0d800096a0 tx=0x7f0d80006b40).stop 2023-12-25T20:11:55.913 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:55.912+0000 7f0d9234d700 1 -- 172.21.15.160:0/3026216023 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0d8c0f5410 msgr2=0x7f0d8c1055d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:11:55.913 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:55.912+0000 7f0d9234d700 1 --2- 172.21.15.160:0/3026216023 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0d8c0f5410 0x7f0d8c1055d0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7f0d7c013510 tx=0x7f0d7c00b040).stop 2023-12-25T20:11:55.913 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:55.912+0000 7f0d9234d700 1 -- 172.21.15.160:0/3026216023 shutdown_connections 2023-12-25T20:11:55.913 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:55.912+0000 7f0d9234d700 1 --2- 172.21.15.160:0/3026216023 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f0d7402ff60 0x7f0d74032420 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:11:55.913 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:55.912+0000 7f0d9234d700 1 --2- 172.21.15.160:0/3026216023 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0d8c0f5410 0x7f0d8c1055d0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:11:55.913 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:55.912+0000 7f0d9234d700 1 -- 172.21.15.160:0/3026216023 >> 172.21.15.160:0/3026216023 conn(0x7f0d8c0f0b70 msgr2=0x7f0d8c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:11:55.913 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:55.912+0000 7f0d9234d700 1 -- 172.21.15.160:0/3026216023 shutdown_connections 2023-12-25T20:11:55.914 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:55.912+0000 7f0d9234d700 1 -- 172.21.15.160:0/3026216023 wait complete. 2023-12-25T20:11:55.914 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:11:56.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:11:55 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:11:54.667370+0000 mgr.smithi050.nbzhun (mgr.14162) 141 : cluster [DBG] pgmap v134: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:11:56.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:11:55 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:11:55.912539+0000 mon.smithi050 (mon.0) 213 : audit [DBG] from='client.? 172.21.15.160:0/3026216023' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:11:57.316 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:11:57.317 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:11:58.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:11:57 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:11:56.667742+0000 mgr.smithi050.nbzhun (mgr.14162) 142 : cluster [DBG] pgmap v135: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:11:58.564 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:58.563+0000 7faf230f9700 1 -- 172.21.15.160:0/617290038 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faf1c0f5410 msgr2=0x7faf1c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:11:58.565 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:58.563+0000 7faf230f9700 1 --2- 172.21.15.160:0/617290038 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faf1c0f5410 0x7faf1c0f5830 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 rx=0x7faf10009fb0 tx=0x7faf10009b30).stop 2023-12-25T20:11:58.565 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:58.563+0000 7faf230f9700 1 -- 172.21.15.160:0/617290038 shutdown_connections 2023-12-25T20:11:58.565 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:58.563+0000 7faf230f9700 1 --2- 172.21.15.160:0/617290038 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faf1c0f5410 0x7faf1c0f5830 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:11:58.565 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:58.563+0000 7faf230f9700 1 -- 172.21.15.160:0/617290038 >> 172.21.15.160:0/617290038 conn(0x7faf1c0f0b70 msgr2=0x7faf1c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:11:58.565 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:58.563+0000 7faf230f9700 1 -- 172.21.15.160:0/617290038 shutdown_connections 2023-12-25T20:11:58.565 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:58.563+0000 7faf230f9700 1 -- 172.21.15.160:0/617290038 wait complete. 2023-12-25T20:11:58.565 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:58.564+0000 7faf230f9700 1 Processor -- start 2023-12-25T20:11:58.565 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:58.564+0000 7faf230f9700 1 -- start start 2023-12-25T20:11:58.565 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:58.564+0000 7faf230f9700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faf1c0f5410 0x7faf1c106710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:11:58.565 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:58.564+0000 7faf230f9700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf1c106c00 con 0x7faf1c0f5410 2023-12-25T20:11:58.567 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:58.564+0000 7faf20e95700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faf1c0f5410 0x7faf1c106710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:11:58.567 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:58.564+0000 7faf20e95700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faf1c0f5410 0x7faf1c106710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:37612/0 (socket says 172.21.15.160:37612) 2023-12-25T20:11:58.567 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:58.564+0000 7faf20e95700 1 -- 172.21.15.160:0/3734722671 learned_addr learned my addr 172.21.15.160:0/3734722671 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:11:58.567 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:58.565+0000 7faf20e95700 1 -- 172.21.15.160:0/3734722671 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf10005040 con 0x7faf1c0f5410 2023-12-25T20:11:58.567 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:58.565+0000 7faf20e95700 1 --2- 172.21.15.160:0/3734722671 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faf1c0f5410 0x7faf1c106710 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7faf10013510 tx=0x7faf1000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:11:58.568 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:58.565+0000 7faf197fa700 1 -- 172.21.15.160:0/3734722671 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faf10009320 con 0x7faf1c0f5410 2023-12-25T20:11:58.568 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:58.565+0000 7faf197fa700 1 -- 172.21.15.160:0/3734722671 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7faf10013f00 con 0x7faf1c0f5410 2023-12-25T20:11:58.568 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:58.565+0000 7faf230f9700 1 -- 172.21.15.160:0/3734722671 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf1c106e00 con 0x7faf1c0f5410 2023-12-25T20:11:58.568 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:58.565+0000 7faf197fa700 1 -- 172.21.15.160:0/3734722671 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faf10014d90 con 0x7faf1c0f5410 2023-12-25T20:11:58.568 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:58.565+0000 7faf230f9700 1 -- 172.21.15.160:0/3734722671 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf1c107290 con 0x7faf1c0f5410 2023-12-25T20:11:58.568 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:58.566+0000 7faf197fa700 1 -- 172.21.15.160:0/3734722671 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7faf100095f0 con 0x7faf1c0f5410 2023-12-25T20:11:58.568 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:58.566+0000 7faf230f9700 1 -- 172.21.15.160:0/3734722671 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf1c045ad0 con 0x7faf1c0f5410 2023-12-25T20:11:58.568 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:58.566+0000 7faf197fa700 1 --2- 172.21.15.160:0/3734722671 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7faf0c02ff60 0x7faf0c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:11:58.568 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:58.566+0000 7faf197fa700 1 -- 172.21.15.160:0/3734722671 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7faf100410c0 con 0x7faf1c0f5410 2023-12-25T20:11:58.568 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:58.566+0000 7faf1bfff700 1 --2- 172.21.15.160:0/3734722671 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7faf0c02ff60 0x7faf0c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:11:58.568 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:58.567+0000 7faf1bfff700 1 --2- 172.21.15.160:0/3734722671 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7faf0c02ff60 0x7faf0c032420 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7faf080096a0 tx=0x7faf08006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:11:58.571 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:58.571+0000 7faf197fa700 1 -- 172.21.15.160:0/3734722671 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7faf1001d7f0 con 0x7faf1c0f5410 2023-12-25T20:11:58.779 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:58.778+0000 7faf230f9700 1 -- 172.21.15.160:0/3734722671 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7faf1c108140 con 0x7faf1c0f5410 2023-12-25T20:11:58.780 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:58.779+0000 7faf197fa700 1 -- 172.21.15.160:0/3734722671 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7faf10010230 con 0x7faf1c0f5410 2023-12-25T20:11:58.780 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:11:58.780 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:11:58.782 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:58.781+0000 7faf230f9700 1 -- 172.21.15.160:0/3734722671 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7faf0c02ff60 msgr2=0x7faf0c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:11:58.782 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:58.781+0000 7faf230f9700 1 --2- 172.21.15.160:0/3734722671 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7faf0c02ff60 0x7faf0c032420 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7faf080096a0 tx=0x7faf08006b40).stop 2023-12-25T20:11:58.782 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:58.781+0000 7faf230f9700 1 -- 172.21.15.160:0/3734722671 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faf1c0f5410 msgr2=0x7faf1c106710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:11:58.782 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:58.781+0000 7faf230f9700 1 --2- 172.21.15.160:0/3734722671 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faf1c0f5410 0x7faf1c106710 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7faf10013510 tx=0x7faf1000b040).stop 2023-12-25T20:11:58.782 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:58.781+0000 7faf230f9700 1 -- 172.21.15.160:0/3734722671 shutdown_connections 2023-12-25T20:11:58.782 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:58.781+0000 7faf230f9700 1 --2- 172.21.15.160:0/3734722671 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7faf0c02ff60 0x7faf0c032420 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:11:58.782 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:58.781+0000 7faf230f9700 1 --2- 172.21.15.160:0/3734722671 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faf1c0f5410 0x7faf1c106710 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:11:58.782 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:58.781+0000 7faf230f9700 1 -- 172.21.15.160:0/3734722671 >> 172.21.15.160:0/3734722671 conn(0x7faf1c0f0b70 msgr2=0x7faf1c05fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:11:58.782 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:58.781+0000 7faf230f9700 1 -- 172.21.15.160:0/3734722671 shutdown_connections 2023-12-25T20:11:58.782 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:11:58.781+0000 7faf230f9700 1 -- 172.21.15.160:0/3734722671 wait complete. 2023-12-25T20:11:58.783 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:11:59.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:11:58 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:11:58.780792+0000 mon.smithi050 (mon.0) 214 : audit [DBG] from='client.? 172.21.15.160:0/3734722671' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:12:00.180 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:12:00.181 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:12:00.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:11:59 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:11:58.668087+0000 mgr.smithi050.nbzhun (mgr.14162) 143 : cluster [DBG] pgmap v136: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:12:01.423 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:01.422+0000 7f592359e700 1 -- 172.21.15.160:0/1172165150 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f59240f51a0 msgr2=0x7f59240f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:12:01.424 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:01.422+0000 7f592359e700 1 --2- 172.21.15.160:0/1172165150 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f59240f51a0 0x7f59240f55c0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 rx=0x7f5914009fb0 tx=0x7f5914009b30).stop 2023-12-25T20:12:01.424 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:01.422+0000 7f592359e700 1 -- 172.21.15.160:0/1172165150 shutdown_connections 2023-12-25T20:12:01.424 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:01.422+0000 7f592359e700 1 --2- 172.21.15.160:0/1172165150 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f59240f51a0 0x7f59240f55c0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:12:01.424 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:01.422+0000 7f592359e700 1 -- 172.21.15.160:0/1172165150 >> 172.21.15.160:0/1172165150 conn(0x7f59240f09a0 msgr2=0x7f59240f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:12:01.424 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:01.422+0000 7f592359e700 1 -- 172.21.15.160:0/1172165150 shutdown_connections 2023-12-25T20:12:01.424 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:01.422+0000 7f592359e700 1 -- 172.21.15.160:0/1172165150 wait complete. 2023-12-25T20:12:01.424 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:01.422+0000 7f592359e700 1 Processor -- start 2023-12-25T20:12:01.424 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:01.422+0000 7f592359e700 1 -- start start 2023-12-25T20:12:01.424 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:01.423+0000 7f592359e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f59240f51a0 0x7f59241091d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:12:01.424 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:01.423+0000 7f592359e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5924060220 con 0x7f59240f51a0 2023-12-25T20:12:01.424 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:01.423+0000 7f592259c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f59240f51a0 0x7f59241091d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:12:01.424 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:01.423+0000 7f592259c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f59240f51a0 0x7f59241091d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:37628/0 (socket says 172.21.15.160:37628) 2023-12-25T20:12:01.425 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:01.423+0000 7f592259c700 1 -- 172.21.15.160:0/3312068756 learned_addr learned my addr 172.21.15.160:0/3312068756 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:12:01.426 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:01.423+0000 7f592259c700 1 -- 172.21.15.160:0/3312068756 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5914005040 con 0x7f59240f51a0 2023-12-25T20:12:01.426 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:01.423+0000 7f592259c700 1 --2- 172.21.15.160:0/3312068756 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f59240f51a0 0x7f59241091d0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7f5914013510 tx=0x7f591400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:12:01.426 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:01.424+0000 7f5912ffd700 1 -- 172.21.15.160:0/3312068756 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5914009320 con 0x7f59240f51a0 2023-12-25T20:12:01.426 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:01.424+0000 7f592359e700 1 -- 172.21.15.160:0/3312068756 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5924060420 con 0x7f59240f51a0 2023-12-25T20:12:01.426 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:01.424+0000 7f5912ffd700 1 -- 172.21.15.160:0/3312068756 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5914013f00 con 0x7f59240f51a0 2023-12-25T20:12:01.426 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:01.424+0000 7f5912ffd700 1 -- 172.21.15.160:0/3312068756 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5914014dc0 con 0x7f59240f51a0 2023-12-25T20:12:01.426 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:01.424+0000 7f592359e700 1 -- 172.21.15.160:0/3312068756 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5924109880 con 0x7f59240f51a0 2023-12-25T20:12:01.426 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:01.424+0000 7f5912ffd700 1 -- 172.21.15.160:0/3312068756 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f59140095f0 con 0x7f59240f51a0 2023-12-25T20:12:01.427 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:01.424+0000 7f592359e700 1 -- 172.21.15.160:0/3312068756 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5924059100 con 0x7f59240f51a0 2023-12-25T20:12:01.427 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:01.425+0000 7f5912ffd700 1 --2- 172.21.15.160:0/3312068756 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f590c02ff60 0x7f590c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:12:01.427 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:01.425+0000 7f5912ffd700 1 -- 172.21.15.160:0/3312068756 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5914041080 con 0x7f59240f51a0 2023-12-25T20:12:01.427 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:01.425+0000 7f5921d9b700 1 --2- 172.21.15.160:0/3312068756 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f590c02ff60 0x7f590c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:12:01.427 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:01.425+0000 7f5921d9b700 1 --2- 172.21.15.160:0/3312068756 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f590c02ff60 0x7f590c032420 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f59180096a0 tx=0x7f5918006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:12:01.430 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:01.429+0000 7f5912ffd700 1 -- 172.21.15.160:0/3312068756 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f591401c790 con 0x7f59240f51a0 2023-12-25T20:12:01.632 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:01.631+0000 7f592359e700 1 -- 172.21.15.160:0/3312068756 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f59240fabe0 con 0x7f59240f51a0 2023-12-25T20:12:01.633 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:01.632+0000 7f5912ffd700 1 -- 172.21.15.160:0/3312068756 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f591401c790 con 0x7f59240f51a0 2023-12-25T20:12:01.633 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:12:01.633 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:12:01.634 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:01.633+0000 7f592359e700 1 -- 172.21.15.160:0/3312068756 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f590c02ff60 msgr2=0x7f590c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:12:01.634 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:01.633+0000 7f592359e700 1 --2- 172.21.15.160:0/3312068756 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f590c02ff60 0x7f590c032420 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f59180096a0 tx=0x7f5918006b40).stop 2023-12-25T20:12:01.634 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:01.634+0000 7f592359e700 1 -- 172.21.15.160:0/3312068756 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f59240f51a0 msgr2=0x7f59241091d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:12:01.634 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:01.634+0000 7f592359e700 1 --2- 172.21.15.160:0/3312068756 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f59240f51a0 0x7f59241091d0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7f5914013510 tx=0x7f591400b040).stop 2023-12-25T20:12:01.635 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:01.634+0000 7f592359e700 1 -- 172.21.15.160:0/3312068756 shutdown_connections 2023-12-25T20:12:01.635 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:01.634+0000 7f592359e700 1 --2- 172.21.15.160:0/3312068756 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f590c02ff60 0x7f590c032420 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:12:01.635 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:01.634+0000 7f592359e700 1 --2- 172.21.15.160:0/3312068756 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f59240f51a0 0x7f59241091d0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:12:01.635 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:01.634+0000 7f592359e700 1 -- 172.21.15.160:0/3312068756 >> 172.21.15.160:0/3312068756 conn(0x7f59240f09a0 msgr2=0x7f59240f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:12:01.635 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:01.634+0000 7f592359e700 1 -- 172.21.15.160:0/3312068756 shutdown_connections 2023-12-25T20:12:01.635 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:01.634+0000 7f592359e700 1 -- 172.21.15.160:0/3312068756 wait complete. 2023-12-25T20:12:01.635 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:12:02.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:01 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:12:00.668421+0000 mgr.smithi050.nbzhun (mgr.14162) 144 : cluster [DBG] pgmap v137: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:12:02.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:01 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:12:01.633691+0000 mon.smithi050 (mon.0) 215 : audit [DBG] from='client.? 172.21.15.160:0/3312068756' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:12:03.030 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:12:03.030 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:12:04.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:03 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:12:02.668819+0000 mgr.smithi050.nbzhun (mgr.14162) 145 : cluster [DBG] pgmap v138: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:12:04.256 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:04.255+0000 7f3436797700 1 -- 172.21.15.160:0/1214260004 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f34300f5410 msgr2=0x7f34300f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:12:04.256 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:04.255+0000 7f3436797700 1 --2- 172.21.15.160:0/1214260004 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f34300f5410 0x7f34300f5830 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0x7f3420009fb0 tx=0x7f3420009b30).stop 2023-12-25T20:12:04.257 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:04.255+0000 7f3436797700 1 -- 172.21.15.160:0/1214260004 shutdown_connections 2023-12-25T20:12:04.257 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:04.255+0000 7f3436797700 1 --2- 172.21.15.160:0/1214260004 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f34300f5410 0x7f34300f5830 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:12:04.257 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:04.255+0000 7f3436797700 1 -- 172.21.15.160:0/1214260004 >> 172.21.15.160:0/1214260004 conn(0x7f34300f0b70 msgr2=0x7f34300f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:12:04.257 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:04.255+0000 7f3436797700 1 -- 172.21.15.160:0/1214260004 shutdown_connections 2023-12-25T20:12:04.257 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:04.256+0000 7f3436797700 1 -- 172.21.15.160:0/1214260004 wait complete. 2023-12-25T20:12:04.257 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:04.256+0000 7f3436797700 1 Processor -- start 2023-12-25T20:12:04.257 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:04.256+0000 7f3436797700 1 -- start start 2023-12-25T20:12:04.257 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:04.257+0000 7f3436797700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f34300f5410 0x7f3430105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:12:04.257 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:04.257+0000 7f3436797700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3430105b20 con 0x7f34300f5410 2023-12-25T20:12:04.259 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:04.257+0000 7f342ffff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f34300f5410 0x7f3430105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:12:04.259 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:04.257+0000 7f342ffff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f34300f5410 0x7f3430105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:37648/0 (socket says 172.21.15.160:37648) 2023-12-25T20:12:04.259 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:04.257+0000 7f342ffff700 1 -- 172.21.15.160:0/2232849767 learned_addr learned my addr 172.21.15.160:0/2232849767 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:12:04.259 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:04.257+0000 7f342ffff700 1 -- 172.21.15.160:0/2232849767 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3420005040 con 0x7f34300f5410 2023-12-25T20:12:04.259 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:04.257+0000 7f342ffff700 1 --2- 172.21.15.160:0/2232849767 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f34300f5410 0x7f3430105630 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7f3420013510 tx=0x7f342000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:12:04.260 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:04.257+0000 7f342cff9700 1 -- 172.21.15.160:0/2232849767 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3420009320 con 0x7f34300f5410 2023-12-25T20:12:04.260 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:04.257+0000 7f3436797700 1 -- 172.21.15.160:0/2232849767 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3430105d20 con 0x7f34300f5410 2023-12-25T20:12:04.260 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:04.258+0000 7f342cff9700 1 -- 172.21.15.160:0/2232849767 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3420013f00 con 0x7f34300f5410 2023-12-25T20:12:04.260 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:04.258+0000 7f342cff9700 1 -- 172.21.15.160:0/2232849767 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3420014d90 con 0x7f34300f5410 2023-12-25T20:12:04.260 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:04.258+0000 7f3436797700 1 -- 172.21.15.160:0/2232849767 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34301061b0 con 0x7f34300f5410 2023-12-25T20:12:04.260 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:04.258+0000 7f342cff9700 1 -- 172.21.15.160:0/2232849767 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f34200095f0 con 0x7f34300f5410 2023-12-25T20:12:04.260 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:04.258+0000 7f3436797700 1 -- 172.21.15.160:0/2232849767 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3430045ad0 con 0x7f34300f5410 2023-12-25T20:12:04.260 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:04.259+0000 7f342cff9700 1 --2- 172.21.15.160:0/2232849767 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f341802ff60 0x7f3418032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:12:04.260 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:04.259+0000 7f342cff9700 1 -- 172.21.15.160:0/2232849767 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f34200410c0 con 0x7f34300f5410 2023-12-25T20:12:04.260 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:04.259+0000 7f342f7fe700 1 --2- 172.21.15.160:0/2232849767 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f341802ff60 0x7f3418032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:12:04.261 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:04.259+0000 7f342f7fe700 1 --2- 172.21.15.160:0/2232849767 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f341802ff60 0x7f3418032420 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f34240096a0 tx=0x7f3424006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:12:04.263 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:04.263+0000 7f342cff9700 1 -- 172.21.15.160:0/2232849767 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3420014420 con 0x7f34300f5410 2023-12-25T20:12:04.467 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:04.467+0000 7f3436797700 1 -- 172.21.15.160:0/2232849767 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f34300fae60 con 0x7f34300f5410 2023-12-25T20:12:04.467 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:04.468+0000 7f342cff9700 1 -- 172.21.15.160:0/2232849767 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3420010230 con 0x7f34300f5410 2023-12-25T20:12:04.468 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:12:04.468 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:12:04.469 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:04.469+0000 7f3436797700 1 -- 172.21.15.160:0/2232849767 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f341802ff60 msgr2=0x7f3418032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:12:04.469 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:04.469+0000 7f3436797700 1 --2- 172.21.15.160:0/2232849767 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f341802ff60 0x7f3418032420 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f34240096a0 tx=0x7f3424006b40).stop 2023-12-25T20:12:04.469 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:04.469+0000 7f3436797700 1 -- 172.21.15.160:0/2232849767 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f34300f5410 msgr2=0x7f3430105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:12:04.470 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:04.469+0000 7f3436797700 1 --2- 172.21.15.160:0/2232849767 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f34300f5410 0x7f3430105630 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7f3420013510 tx=0x7f342000b040).stop 2023-12-25T20:12:04.470 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:04.469+0000 7f3436797700 1 -- 172.21.15.160:0/2232849767 shutdown_connections 2023-12-25T20:12:04.470 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:04.469+0000 7f3436797700 1 --2- 172.21.15.160:0/2232849767 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f341802ff60 0x7f3418032420 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:12:04.470 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:04.469+0000 7f3436797700 1 --2- 172.21.15.160:0/2232849767 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f34300f5410 0x7f3430105630 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:12:04.470 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:04.469+0000 7f3436797700 1 -- 172.21.15.160:0/2232849767 >> 172.21.15.160:0/2232849767 conn(0x7f34300f0b70 msgr2=0x7f34300f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:12:04.470 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:04.470+0000 7f3436797700 1 -- 172.21.15.160:0/2232849767 shutdown_connections 2023-12-25T20:12:04.470 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:04.470+0000 7f3436797700 1 -- 172.21.15.160:0/2232849767 wait complete. 2023-12-25T20:12:04.470 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:12:05.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:04 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:12:04.468634+0000 mon.smithi050 (mon.0) 216 : audit [DBG] from='client.? 172.21.15.160:0/2232849767' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:12:05.877 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:12:05.877 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:12:06.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:05 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:12:04.669215+0000 mgr.smithi050.nbzhun (mgr.14162) 146 : cluster [DBG] pgmap v139: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:12:07.116 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:07.116+0000 7fb41c96e700 1 -- 172.21.15.160:0/529858790 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb418060280 msgr2=0x7fb4180f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:12:07.116 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:07.116+0000 7fb41c96e700 1 --2- 172.21.15.160:0/529858790 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb418060280 0x7fb4180f90f0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 rx=0x7fb408009fb0 tx=0x7fb408009b30).stop 2023-12-25T20:12:07.116 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:07.116+0000 7fb41c96e700 1 -- 172.21.15.160:0/529858790 shutdown_connections 2023-12-25T20:12:07.116 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:07.116+0000 7fb41c96e700 1 --2- 172.21.15.160:0/529858790 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb418060280 0x7fb4180f90f0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:12:07.116 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:07.116+0000 7fb41c96e700 1 -- 172.21.15.160:0/529858790 >> 172.21.15.160:0/529858790 conn(0x7fb4180f0b70 msgr2=0x7fb4180f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:12:07.117 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:07.116+0000 7fb41c96e700 1 -- 172.21.15.160:0/529858790 shutdown_connections 2023-12-25T20:12:07.117 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:07.116+0000 7fb41c96e700 1 -- 172.21.15.160:0/529858790 wait complete. 2023-12-25T20:12:07.118 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:07.117+0000 7fb41c96e700 1 Processor -- start 2023-12-25T20:12:07.118 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:07.117+0000 7fb41c96e700 1 -- start start 2023-12-25T20:12:07.118 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:07.117+0000 7fb41c96e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb418060280 0x7fb4181066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:12:07.118 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:07.117+0000 7fb41c96e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb418106bd0 con 0x7fb418060280 2023-12-25T20:12:07.118 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:07.117+0000 7fb41659c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb418060280 0x7fb4181066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:12:07.118 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:07.117+0000 7fb41659c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb418060280 0x7fb4181066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:37670/0 (socket says 172.21.15.160:37670) 2023-12-25T20:12:07.118 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:07.117+0000 7fb41659c700 1 -- 172.21.15.160:0/2104995972 learned_addr learned my addr 172.21.15.160:0/2104995972 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:12:07.119 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:07.118+0000 7fb41659c700 1 -- 172.21.15.160:0/2104995972 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb408005040 con 0x7fb418060280 2023-12-25T20:12:07.119 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:07.118+0000 7fb41659c700 1 --2- 172.21.15.160:0/2104995972 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb418060280 0x7fb4181066e0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7fb408013510 tx=0x7fb40800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:12:07.119 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:07.118+0000 7fb40effd700 1 -- 172.21.15.160:0/2104995972 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb408009320 con 0x7fb418060280 2023-12-25T20:12:07.119 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:07.118+0000 7fb41c96e700 1 -- 172.21.15.160:0/2104995972 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb418106dd0 con 0x7fb418060280 2023-12-25T20:12:07.119 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:07.118+0000 7fb40effd700 1 -- 172.21.15.160:0/2104995972 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb408013f00 con 0x7fb418060280 2023-12-25T20:12:07.119 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:07.118+0000 7fb40effd700 1 -- 172.21.15.160:0/2104995972 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb408014dc0 con 0x7fb418060280 2023-12-25T20:12:07.119 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:07.118+0000 7fb41c96e700 1 -- 172.21.15.160:0/2104995972 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb418107260 con 0x7fb418060280 2023-12-25T20:12:07.120 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:07.119+0000 7fb40effd700 1 -- 172.21.15.160:0/2104995972 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb4080095f0 con 0x7fb418060280 2023-12-25T20:12:07.120 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:07.119+0000 7fb41c96e700 1 -- 172.21.15.160:0/2104995972 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb418045ad0 con 0x7fb418060280 2023-12-25T20:12:07.120 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:07.119+0000 7fb40effd700 1 --2- 172.21.15.160:0/2104995972 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fb40402ff60 0x7fb404032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:12:07.120 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:07.119+0000 7fb40effd700 1 -- 172.21.15.160:0/2104995972 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb4080410c0 con 0x7fb418060280 2023-12-25T20:12:07.120 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:07.119+0000 7fb415d9b700 1 --2- 172.21.15.160:0/2104995972 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fb40402ff60 0x7fb404032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:12:07.120 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:07.119+0000 7fb415d9b700 1 --2- 172.21.15.160:0/2104995972 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fb40402ff60 0x7fb404032420 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7fb4000096a0 tx=0x7fb400006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:12:07.123 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:07.124+0000 7fb40effd700 1 -- 172.21.15.160:0/2104995972 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb40801c790 con 0x7fb418060280 2023-12-25T20:12:07.327 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:07.327+0000 7fb41c96e700 1 -- 172.21.15.160:0/2104995972 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb4181080b0 con 0x7fb418060280 2023-12-25T20:12:07.327 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:07.328+0000 7fb40effd700 1 -- 172.21.15.160:0/2104995972 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb40801c790 con 0x7fb418060280 2023-12-25T20:12:07.327 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:12:07.328 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:12:07.329 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:07.329+0000 7fb41c96e700 1 -- 172.21.15.160:0/2104995972 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fb40402ff60 msgr2=0x7fb404032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:12:07.329 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:07.329+0000 7fb41c96e700 1 --2- 172.21.15.160:0/2104995972 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fb40402ff60 0x7fb404032420 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7fb4000096a0 tx=0x7fb400006b40).stop 2023-12-25T20:12:07.329 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:07.329+0000 7fb41c96e700 1 -- 172.21.15.160:0/2104995972 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb418060280 msgr2=0x7fb4181066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:12:07.329 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:07.329+0000 7fb41c96e700 1 --2- 172.21.15.160:0/2104995972 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb418060280 0x7fb4181066e0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7fb408013510 tx=0x7fb40800b040).stop 2023-12-25T20:12:07.330 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:07.329+0000 7fb41c96e700 1 -- 172.21.15.160:0/2104995972 shutdown_connections 2023-12-25T20:12:07.330 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:07.329+0000 7fb41c96e700 1 --2- 172.21.15.160:0/2104995972 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fb40402ff60 0x7fb404032420 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:12:07.330 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:07.329+0000 7fb41c96e700 1 --2- 172.21.15.160:0/2104995972 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb418060280 0x7fb4181066e0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:12:07.330 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:07.329+0000 7fb41c96e700 1 -- 172.21.15.160:0/2104995972 >> 172.21.15.160:0/2104995972 conn(0x7fb4180f0b70 msgr2=0x7fb4180f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:12:07.330 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:07.329+0000 7fb41c96e700 1 -- 172.21.15.160:0/2104995972 shutdown_connections 2023-12-25T20:12:07.330 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:07.329+0000 7fb41c96e700 1 -- 172.21.15.160:0/2104995972 wait complete. 2023-12-25T20:12:07.330 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:12:08.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:07 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:12:06.669525+0000 mgr.smithi050.nbzhun (mgr.14162) 147 : cluster [DBG] pgmap v140: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:12:08.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:07 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:12:07.328464+0000 mon.smithi050 (mon.0) 217 : audit [DBG] from='client.? 172.21.15.160:0/2104995972' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:12:08.713 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:12:08.713 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:12:09.997 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:09.995+0000 7f0b2efa5700 1 -- 172.21.15.160:0/2958181225 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0b280f51a0 msgr2=0x7f0b280f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:12:09.997 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:09.995+0000 7f0b2efa5700 1 --2- 172.21.15.160:0/2958181225 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0b280f51a0 0x7f0b280f55c0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 rx=0x7f0b24009fb0 tx=0x7f0b24009b30).stop 2023-12-25T20:12:09.997 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:09.995+0000 7f0b2efa5700 1 -- 172.21.15.160:0/2958181225 shutdown_connections 2023-12-25T20:12:09.997 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:09.995+0000 7f0b2efa5700 1 --2- 172.21.15.160:0/2958181225 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0b280f51a0 0x7f0b280f55c0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:12:09.997 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:09.995+0000 7f0b2efa5700 1 -- 172.21.15.160:0/2958181225 >> 172.21.15.160:0/2958181225 conn(0x7f0b280f09a0 msgr2=0x7f0b280f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:12:09.997 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:09.996+0000 7f0b2efa5700 1 -- 172.21.15.160:0/2958181225 shutdown_connections 2023-12-25T20:12:09.997 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:09.996+0000 7f0b2efa5700 1 -- 172.21.15.160:0/2958181225 wait complete. 2023-12-25T20:12:09.997 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:09.996+0000 7f0b2efa5700 1 Processor -- start 2023-12-25T20:12:09.997 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:09.996+0000 7f0b2efa5700 1 -- start start 2023-12-25T20:12:09.997 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:09.996+0000 7f0b2efa5700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0b280f51a0 0x7f0b281064a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:12:09.997 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:09.996+0000 7f0b2efa5700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b28106990 con 0x7f0b280f51a0 2023-12-25T20:12:09.998 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:09.996+0000 7f0b2dfa3700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0b280f51a0 0x7f0b281064a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:12:09.998 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:09.996+0000 7f0b2dfa3700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0b280f51a0 0x7f0b281064a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:58616/0 (socket says 172.21.15.160:58616) 2023-12-25T20:12:09.998 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:09.996+0000 7f0b2dfa3700 1 -- 172.21.15.160:0/4170976106 learned_addr learned my addr 172.21.15.160:0/4170976106 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:12:09.998 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:09.997+0000 7f0b2dfa3700 1 -- 172.21.15.160:0/4170976106 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b24005040 con 0x7f0b280f51a0 2023-12-25T20:12:09.998 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:09.997+0000 7f0b2dfa3700 1 --2- 172.21.15.160:0/4170976106 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0b280f51a0 0x7f0b281064a0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7f0b24013510 tx=0x7f0b2400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:12:09.998 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:09.997+0000 7f0b1e7fc700 1 -- 172.21.15.160:0/4170976106 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0b24009320 con 0x7f0b280f51a0 2023-12-25T20:12:09.998 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:09.997+0000 7f0b2efa5700 1 -- 172.21.15.160:0/4170976106 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b28106b90 con 0x7f0b280f51a0 2023-12-25T20:12:09.998 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:09.997+0000 7f0b1e7fc700 1 -- 172.21.15.160:0/4170976106 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0b24013f00 con 0x7f0b280f51a0 2023-12-25T20:12:09.998 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:09.997+0000 7f0b1e7fc700 1 -- 172.21.15.160:0/4170976106 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0b24014dc0 con 0x7f0b280f51a0 2023-12-25T20:12:09.998 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:09.997+0000 7f0b2efa5700 1 -- 172.21.15.160:0/4170976106 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b28107020 con 0x7f0b280f51a0 2023-12-25T20:12:09.999 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:09.998+0000 7f0b1e7fc700 1 -- 172.21.15.160:0/4170976106 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0b240095f0 con 0x7f0b280f51a0 2023-12-25T20:12:09.999 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:09.998+0000 7f0b2efa5700 1 -- 172.21.15.160:0/4170976106 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b28045ad0 con 0x7f0b280f51a0 2023-12-25T20:12:09.999 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:09.998+0000 7f0b1e7fc700 1 --2- 172.21.15.160:0/4170976106 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f0b1402ff60 0x7f0b14032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:12:09.999 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:09.998+0000 7f0b1e7fc700 1 -- 172.21.15.160:0/4170976106 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0b24041000 con 0x7f0b280f51a0 2023-12-25T20:12:09.999 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:09.998+0000 7f0b2d7a2700 1 --2- 172.21.15.160:0/4170976106 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f0b1402ff60 0x7f0b14032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:12:09.999 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:09.999+0000 7f0b2d7a2700 1 --2- 172.21.15.160:0/4170976106 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f0b1402ff60 0x7f0b14032420 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f0b180096a0 tx=0x7f0b18006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:12:10.002 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:10.003+0000 7f0b1e7fc700 1 -- 172.21.15.160:0/4170976106 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0b240268c0 con 0x7f0b280f51a0 2023-12-25T20:12:10.205 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:10.205+0000 7f0b2efa5700 1 -- 172.21.15.160:0/4170976106 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0b28107ed0 con 0x7f0b280f51a0 2023-12-25T20:12:10.205 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:10.206+0000 7f0b1e7fc700 1 -- 172.21.15.160:0/4170976106 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0b24010230 con 0x7f0b280f51a0 2023-12-25T20:12:10.206 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:12:10.206 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:12:10.207 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:10.207+0000 7f0b2efa5700 1 -- 172.21.15.160:0/4170976106 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f0b1402ff60 msgr2=0x7f0b14032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:12:10.207 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:10.207+0000 7f0b2efa5700 1 --2- 172.21.15.160:0/4170976106 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f0b1402ff60 0x7f0b14032420 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f0b180096a0 tx=0x7f0b18006b40).stop 2023-12-25T20:12:10.208 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:10.207+0000 7f0b2efa5700 1 -- 172.21.15.160:0/4170976106 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0b280f51a0 msgr2=0x7f0b281064a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:12:10.208 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:10.207+0000 7f0b2efa5700 1 --2- 172.21.15.160:0/4170976106 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0b280f51a0 0x7f0b281064a0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7f0b24013510 tx=0x7f0b2400b040).stop 2023-12-25T20:12:10.208 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:10.207+0000 7f0b2efa5700 1 -- 172.21.15.160:0/4170976106 shutdown_connections 2023-12-25T20:12:10.208 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:10.207+0000 7f0b2efa5700 1 --2- 172.21.15.160:0/4170976106 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f0b1402ff60 0x7f0b14032420 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:12:10.208 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:10.207+0000 7f0b2efa5700 1 --2- 172.21.15.160:0/4170976106 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0b280f51a0 0x7f0b281064a0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:12:10.208 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:10.207+0000 7f0b2efa5700 1 -- 172.21.15.160:0/4170976106 >> 172.21.15.160:0/4170976106 conn(0x7f0b280f09a0 msgr2=0x7f0b2805fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:12:10.208 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:10.207+0000 7f0b2efa5700 1 -- 172.21.15.160:0/4170976106 shutdown_connections 2023-12-25T20:12:10.208 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:10.207+0000 7f0b2efa5700 1 -- 172.21.15.160:0/4170976106 wait complete. 2023-12-25T20:12:10.208 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:12:10.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:09 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:12:08.669843+0000 mgr.smithi050.nbzhun (mgr.14162) 148 : cluster [DBG] pgmap v141: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:12:11.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:10 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:12:10.206478+0000 mon.smithi050 (mon.0) 218 : audit [DBG] from='client.? 172.21.15.160:0/4170976106' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:12:11.594 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:12:11.594 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:12:12.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:11 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:12:10.670298+0000 mgr.smithi050.nbzhun (mgr.14162) 149 : cluster [DBG] pgmap v142: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:12:12.843 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:12.843+0000 7f0f45937700 1 -- 172.21.15.160:0/389182673 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f400f47c0 msgr2=0x7f0f400f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:12:12.843 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:12.843+0000 7f0f45937700 1 --2- 172.21.15.160:0/389182673 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f400f47c0 0x7f0f400f4be0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0x7f0f34009fb0 tx=0x7f0f34009b30).stop 2023-12-25T20:12:12.843 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:12.843+0000 7f0f45937700 1 -- 172.21.15.160:0/389182673 shutdown_connections 2023-12-25T20:12:12.844 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:12.843+0000 7f0f45937700 1 --2- 172.21.15.160:0/389182673 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f400f47c0 0x7f0f400f4be0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:12:12.844 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:12.843+0000 7f0f45937700 1 -- 172.21.15.160:0/389182673 >> 172.21.15.160:0/389182673 conn(0x7f0f400f02e0 msgr2=0x7f0f400f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:12:12.844 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:12.844+0000 7f0f45937700 1 -- 172.21.15.160:0/389182673 shutdown_connections 2023-12-25T20:12:12.844 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:12.844+0000 7f0f45937700 1 -- 172.21.15.160:0/389182673 wait complete. 2023-12-25T20:12:12.845 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:12.844+0000 7f0f45937700 1 Processor -- start 2023-12-25T20:12:12.845 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:12.844+0000 7f0f45937700 1 -- start start 2023-12-25T20:12:12.845 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:12.844+0000 7f0f45937700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f400f47c0 0x7f0f40106700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:12:12.845 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:12.844+0000 7f0f45937700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f40106bf0 con 0x7f0f400f47c0 2023-12-25T20:12:12.845 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:12.845+0000 7f0f3effd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f400f47c0 0x7f0f40106700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:12:12.846 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:12.845+0000 7f0f3effd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f400f47c0 0x7f0f40106700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:58636/0 (socket says 172.21.15.160:58636) 2023-12-25T20:12:12.846 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:12.845+0000 7f0f3effd700 1 -- 172.21.15.160:0/433834799 learned_addr learned my addr 172.21.15.160:0/433834799 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:12:12.846 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:12.845+0000 7f0f3effd700 1 -- 172.21.15.160:0/433834799 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f34005040 con 0x7f0f400f47c0 2023-12-25T20:12:12.846 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:12.845+0000 7f0f3effd700 1 --2- 172.21.15.160:0/433834799 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f400f47c0 0x7f0f40106700 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7f0f34013510 tx=0x7f0f3400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:12:12.846 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:12.845+0000 7f0f277fe700 1 -- 172.21.15.160:0/433834799 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0f34009320 con 0x7f0f400f47c0 2023-12-25T20:12:12.846 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:12.845+0000 7f0f277fe700 1 -- 172.21.15.160:0/433834799 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0f34013f00 con 0x7f0f400f47c0 2023-12-25T20:12:12.846 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:12.845+0000 7f0f277fe700 1 -- 172.21.15.160:0/433834799 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0f34014d90 con 0x7f0f400f47c0 2023-12-25T20:12:12.846 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:12.845+0000 7f0f45937700 1 -- 172.21.15.160:0/433834799 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f40106df0 con 0x7f0f400f47c0 2023-12-25T20:12:12.846 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:12.845+0000 7f0f45937700 1 -- 172.21.15.160:0/433834799 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f40107280 con 0x7f0f400f47c0 2023-12-25T20:12:12.847 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:12.846+0000 7f0f277fe700 1 -- 172.21.15.160:0/433834799 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0f340095f0 con 0x7f0f400f47c0 2023-12-25T20:12:12.847 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:12.846+0000 7f0f45937700 1 -- 172.21.15.160:0/433834799 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f40045ad0 con 0x7f0f400f47c0 2023-12-25T20:12:12.847 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:12.846+0000 7f0f277fe700 1 --2- 172.21.15.160:0/433834799 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f0f2802ff60 0x7f0f28032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:12:12.847 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:12.846+0000 7f0f277fe700 1 -- 172.21.15.160:0/433834799 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0f34041000 con 0x7f0f400f47c0 2023-12-25T20:12:12.847 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:12.846+0000 7f0f3e7fc700 1 --2- 172.21.15.160:0/433834799 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f0f2802ff60 0x7f0f28032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:12:12.847 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:12.847+0000 7f0f3e7fc700 1 --2- 172.21.15.160:0/433834799 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f0f2802ff60 0x7f0f28032420 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f0f300096a0 tx=0x7f0f30006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:12:12.853 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:12.851+0000 7f0f277fe700 1 -- 172.21.15.160:0/433834799 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0f3401d7f0 con 0x7f0f400f47c0 2023-12-25T20:12:13.052 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:13.052+0000 7f0f45937700 1 -- 172.21.15.160:0/433834799 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0f40108130 con 0x7f0f400f47c0 2023-12-25T20:12:13.052 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:13.053+0000 7f0f277fe700 1 -- 172.21.15.160:0/433834799 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0f34010230 con 0x7f0f400f47c0 2023-12-25T20:12:13.053 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:12:13.053 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:12:13.058 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:13.054+0000 7f0f45937700 1 -- 172.21.15.160:0/433834799 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f0f2802ff60 msgr2=0x7f0f28032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:12:13.059 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:13.054+0000 7f0f45937700 1 --2- 172.21.15.160:0/433834799 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f0f2802ff60 0x7f0f28032420 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f0f300096a0 tx=0x7f0f30006b40).stop 2023-12-25T20:12:13.059 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:13.054+0000 7f0f45937700 1 -- 172.21.15.160:0/433834799 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f400f47c0 msgr2=0x7f0f40106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:12:13.059 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:13.054+0000 7f0f45937700 1 --2- 172.21.15.160:0/433834799 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f400f47c0 0x7f0f40106700 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7f0f34013510 tx=0x7f0f3400b040).stop 2023-12-25T20:12:13.059 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:13.054+0000 7f0f45937700 1 -- 172.21.15.160:0/433834799 shutdown_connections 2023-12-25T20:12:13.059 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:13.054+0000 7f0f45937700 1 --2- 172.21.15.160:0/433834799 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f0f2802ff60 0x7f0f28032420 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:12:13.059 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:13.054+0000 7f0f45937700 1 --2- 172.21.15.160:0/433834799 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f400f47c0 0x7f0f40106700 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:12:13.059 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:13.054+0000 7f0f45937700 1 -- 172.21.15.160:0/433834799 >> 172.21.15.160:0/433834799 conn(0x7f0f400f02e0 msgr2=0x7f0f400f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:12:13.059 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:13.055+0000 7f0f45937700 1 -- 172.21.15.160:0/433834799 shutdown_connections 2023-12-25T20:12:13.059 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:13.055+0000 7f0f45937700 1 -- 172.21.15.160:0/433834799 wait complete. 2023-12-25T20:12:13.059 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:12:14.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:13 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:12:12.670640+0000 mgr.smithi050.nbzhun (mgr.14162) 150 : cluster [DBG] pgmap v143: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:12:14.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:13 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:12:13.053359+0000 mon.smithi050 (mon.0) 219 : audit [DBG] from='client.? 172.21.15.160:0/433834799' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:12:14.441 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:12:14.441 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:12:15.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:14 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:12:14.766349+0000 mon.smithi050 (mon.0) 220 : audit [INF] from='mgr.14162 172.21.15.50:0/3178367813' entity='mgr.smithi050.nbzhun' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi050.nbzhun/mirror_snapshot_schedule"}]: dispatch 2023-12-25T20:12:15.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:14 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:12:14.767064+0000 mon.smithi050 (mon.0) 221 : audit [INF] from='mgr.14162 172.21.15.50:0/3178367813' entity='mgr.smithi050.nbzhun' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi050.nbzhun/trash_purge_schedule"}]: dispatch 2023-12-25T20:12:15.663 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:15.661+0000 7f3300c90700 1 -- 172.21.15.160:0/3878720420 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f32fc060280 msgr2=0x7f32fc0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:12:15.663 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:15.661+0000 7f3300c90700 1 --2- 172.21.15.160:0/3878720420 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f32fc060280 0x7f32fc0f90f0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 rx=0x7f32f0009fb0 tx=0x7f32f0009b30).stop 2023-12-25T20:12:15.663 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:15.661+0000 7f3300c90700 1 -- 172.21.15.160:0/3878720420 shutdown_connections 2023-12-25T20:12:15.663 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:15.661+0000 7f3300c90700 1 --2- 172.21.15.160:0/3878720420 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f32fc060280 0x7f32fc0f90f0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:12:15.663 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:15.661+0000 7f3300c90700 1 -- 172.21.15.160:0/3878720420 >> 172.21.15.160:0/3878720420 conn(0x7f32fc0f0b70 msgr2=0x7f32fc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:12:15.663 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:15.661+0000 7f3300c90700 1 -- 172.21.15.160:0/3878720420 shutdown_connections 2023-12-25T20:12:15.663 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:15.661+0000 7f3300c90700 1 -- 172.21.15.160:0/3878720420 wait complete. 2023-12-25T20:12:15.663 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:15.662+0000 7f3300c90700 1 Processor -- start 2023-12-25T20:12:15.663 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:15.662+0000 7f3300c90700 1 -- start start 2023-12-25T20:12:15.663 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:15.662+0000 7f3300c90700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f32fc060280 0x7f32fc1066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:12:15.663 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:15.662+0000 7f3300c90700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32fc106bd0 con 0x7f32fc060280 2023-12-25T20:12:15.663 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:15.662+0000 7f32fa59c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f32fc060280 0x7f32fc1066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:12:15.664 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:15.662+0000 7f32fa59c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f32fc060280 0x7f32fc1066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:58662/0 (socket says 172.21.15.160:58662) 2023-12-25T20:12:15.664 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:15.662+0000 7f32fa59c700 1 -- 172.21.15.160:0/3513569540 learned_addr learned my addr 172.21.15.160:0/3513569540 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:12:15.664 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:15.663+0000 7f32fa59c700 1 -- 172.21.15.160:0/3513569540 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32f0005040 con 0x7f32fc060280 2023-12-25T20:12:15.664 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:15.663+0000 7f32fa59c700 1 --2- 172.21.15.160:0/3513569540 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f32fc060280 0x7f32fc1066e0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7f32f0013510 tx=0x7f32f000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:12:15.664 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:15.663+0000 7f32eaffd700 1 -- 172.21.15.160:0/3513569540 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f32f0009320 con 0x7f32fc060280 2023-12-25T20:12:15.664 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:15.663+0000 7f3300c90700 1 -- 172.21.15.160:0/3513569540 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32fc106dd0 con 0x7f32fc060280 2023-12-25T20:12:15.664 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:15.663+0000 7f32eaffd700 1 -- 172.21.15.160:0/3513569540 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f32f0013f00 con 0x7f32fc060280 2023-12-25T20:12:15.664 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:15.663+0000 7f32eaffd700 1 -- 172.21.15.160:0/3513569540 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f32f0014dc0 con 0x7f32fc060280 2023-12-25T20:12:15.664 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:15.663+0000 7f3300c90700 1 -- 172.21.15.160:0/3513569540 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32fc107260 con 0x7f32fc060280 2023-12-25T20:12:15.665 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:15.664+0000 7f32eaffd700 1 -- 172.21.15.160:0/3513569540 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f32f00095f0 con 0x7f32fc060280 2023-12-25T20:12:15.665 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:15.664+0000 7f3300c90700 1 -- 172.21.15.160:0/3513569540 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32fc045ad0 con 0x7f32fc060280 2023-12-25T20:12:15.665 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:15.664+0000 7f32eaffd700 1 --2- 172.21.15.160:0/3513569540 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f32e402ff60 0x7f32e4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:12:15.665 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:15.664+0000 7f32eaffd700 1 -- 172.21.15.160:0/3513569540 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f32f0041000 con 0x7f32fc060280 2023-12-25T20:12:15.665 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:15.664+0000 7f32f9d9b700 1 --2- 172.21.15.160:0/3513569540 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f32e402ff60 0x7f32e4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:12:15.665 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:15.665+0000 7f32f9d9b700 1 --2- 172.21.15.160:0/3513569540 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f32e402ff60 0x7f32e4032420 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f32ec0096a0 tx=0x7f32ec006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:12:15.668 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:15.669+0000 7f32eaffd700 1 -- 172.21.15.160:0/3513569540 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f32f0014420 con 0x7f32fc060280 2023-12-25T20:12:15.873 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:15.873+0000 7f3300c90700 1 -- 172.21.15.160:0/3513569540 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f32fc108110 con 0x7f32fc060280 2023-12-25T20:12:15.874 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:15.874+0000 7f32eaffd700 1 -- 172.21.15.160:0/3513569540 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f32f0010230 con 0x7f32fc060280 2023-12-25T20:12:15.874 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:12:15.874 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:12:15.875 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:15.876+0000 7f3300c90700 1 -- 172.21.15.160:0/3513569540 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f32e402ff60 msgr2=0x7f32e4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:12:15.876 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:15.876+0000 7f3300c90700 1 --2- 172.21.15.160:0/3513569540 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f32e402ff60 0x7f32e4032420 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f32ec0096a0 tx=0x7f32ec006b40).stop 2023-12-25T20:12:15.876 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:15.876+0000 7f3300c90700 1 -- 172.21.15.160:0/3513569540 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f32fc060280 msgr2=0x7f32fc1066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:12:15.876 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:15.876+0000 7f3300c90700 1 --2- 172.21.15.160:0/3513569540 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f32fc060280 0x7f32fc1066e0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7f32f0013510 tx=0x7f32f000b040).stop 2023-12-25T20:12:15.876 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:15.876+0000 7f3300c90700 1 -- 172.21.15.160:0/3513569540 shutdown_connections 2023-12-25T20:12:15.876 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:15.876+0000 7f3300c90700 1 --2- 172.21.15.160:0/3513569540 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f32e402ff60 0x7f32e4032420 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:12:15.876 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:15.876+0000 7f3300c90700 1 --2- 172.21.15.160:0/3513569540 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f32fc060280 0x7f32fc1066e0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:12:15.876 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:15.876+0000 7f3300c90700 1 -- 172.21.15.160:0/3513569540 >> 172.21.15.160:0/3513569540 conn(0x7f32fc0f0b70 msgr2=0x7f32fc0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:12:15.876 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:15.876+0000 7f3300c90700 1 -- 172.21.15.160:0/3513569540 shutdown_connections 2023-12-25T20:12:15.876 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:15.876+0000 7f3300c90700 1 -- 172.21.15.160:0/3513569540 wait complete. 2023-12-25T20:12:15.876 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:12:16.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:15 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:12:14.671063+0000 mgr.smithi050.nbzhun (mgr.14162) 151 : cluster [DBG] pgmap v144: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:12:16.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:15 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:12:15.874589+0000 mon.smithi050 (mon.0) 222 : audit [DBG] from='client.? 172.21.15.160:0/3513569540' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:12:17.312 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:12:17.312 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:12:18.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:17 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:12:16.671484+0000 mgr.smithi050.nbzhun (mgr.14162) 152 : cluster [DBG] pgmap v145: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:12:18.545 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:18.544+0000 7f6be67c1700 1 -- 172.21.15.160:0/2330504362 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6be00f5410 msgr2=0x7f6be00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:12:18.545 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:18.544+0000 7f6be67c1700 1 --2- 172.21.15.160:0/2330504362 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6be00f5410 0x7f6be00f5830 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 rx=0x7f6bd0009fb0 tx=0x7f6bd0009b30).stop 2023-12-25T20:12:18.545 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:18.544+0000 7f6be67c1700 1 -- 172.21.15.160:0/2330504362 shutdown_connections 2023-12-25T20:12:18.545 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:18.544+0000 7f6be67c1700 1 --2- 172.21.15.160:0/2330504362 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6be00f5410 0x7f6be00f5830 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:12:18.545 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:18.544+0000 7f6be67c1700 1 -- 172.21.15.160:0/2330504362 >> 172.21.15.160:0/2330504362 conn(0x7f6be00f0b70 msgr2=0x7f6be00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:12:18.545 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:18.544+0000 7f6be67c1700 1 -- 172.21.15.160:0/2330504362 shutdown_connections 2023-12-25T20:12:18.546 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:18.544+0000 7f6be67c1700 1 -- 172.21.15.160:0/2330504362 wait complete. 2023-12-25T20:12:18.546 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:18.545+0000 7f6be67c1700 1 Processor -- start 2023-12-25T20:12:18.546 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:18.545+0000 7f6be67c1700 1 -- start start 2023-12-25T20:12:18.546 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:18.545+0000 7f6be67c1700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6be00f5410 0x7f6be0109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:12:18.546 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:18.545+0000 7f6be67c1700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6be0060240 con 0x7f6be00f5410 2023-12-25T20:12:18.546 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:18.545+0000 7f6bdffff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6be00f5410 0x7f6be0109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:12:18.546 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:18.545+0000 7f6bdffff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6be00f5410 0x7f6be0109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:52956/0 (socket says 172.21.15.160:52956) 2023-12-25T20:12:18.546 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:18.545+0000 7f6bdffff700 1 -- 172.21.15.160:0/3752654331 learned_addr learned my addr 172.21.15.160:0/3752654331 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:12:18.546 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:18.545+0000 7f6bdffff700 1 -- 172.21.15.160:0/3752654331 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bd0005040 con 0x7f6be00f5410 2023-12-25T20:12:18.546 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:18.546+0000 7f6bdffff700 1 --2- 172.21.15.160:0/3752654331 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6be00f5410 0x7f6be0109430 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7f6bd0014000 tx=0x7f6bd000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:12:18.547 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:18.546+0000 7f6bdcff9700 1 -- 172.21.15.160:0/3752654331 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6bd00095f0 con 0x7f6be00f5410 2023-12-25T20:12:18.547 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:18.546+0000 7f6be67c1700 1 -- 172.21.15.160:0/3752654331 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6be0060440 con 0x7f6be00f5410 2023-12-25T20:12:18.548 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:18.546+0000 7f6bdcff9700 1 -- 172.21.15.160:0/3752654331 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6bd0007e70 con 0x7f6be00f5410 2023-12-25T20:12:18.548 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:18.546+0000 7f6bdcff9700 1 -- 172.21.15.160:0/3752654331 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6bd0014ea0 con 0x7f6be00f5410 2023-12-25T20:12:18.548 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:18.546+0000 7f6be67c1700 1 -- 172.21.15.160:0/3752654331 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6be0109ae0 con 0x7f6be00f5410 2023-12-25T20:12:18.548 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:18.547+0000 7f6bdcff9700 1 -- 172.21.15.160:0/3752654331 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6bd0025070 con 0x7f6be00f5410 2023-12-25T20:12:18.548 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:18.547+0000 7f6be67c1700 1 -- 172.21.15.160:0/3752654331 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6be0045ad0 con 0x7f6be00f5410 2023-12-25T20:12:18.548 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:18.547+0000 7f6bdcff9700 1 --2- 172.21.15.160:0/3752654331 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f6bc802ff60 0x7f6bc8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:12:18.548 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:18.547+0000 7f6bdcff9700 1 -- 172.21.15.160:0/3752654331 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6bd00409f0 con 0x7f6be00f5410 2023-12-25T20:12:18.548 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:18.547+0000 7f6bdf7fe700 1 --2- 172.21.15.160:0/3752654331 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f6bc802ff60 0x7f6bc8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:12:18.548 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:18.547+0000 7f6bdf7fe700 1 --2- 172.21.15.160:0/3752654331 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f6bc802ff60 0x7f6bc8032420 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f6bd40096a0 tx=0x7f6bd4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:12:18.551 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:18.552+0000 7f6bdcff9700 1 -- 172.21.15.160:0/3752654331 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6bd000f4e0 con 0x7f6be00f5410 2023-12-25T20:12:18.759 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:18.759+0000 7f6be67c1700 1 -- 172.21.15.160:0/3752654331 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6be00fae60 con 0x7f6be00f5410 2023-12-25T20:12:18.759 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:18.759+0000 7f6bdcff9700 1 -- 172.21.15.160:0/3752654331 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6bd000f300 con 0x7f6be00f5410 2023-12-25T20:12:18.759 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:12:18.759 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:12:18.761 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:18.761+0000 7f6be67c1700 1 -- 172.21.15.160:0/3752654331 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f6bc802ff60 msgr2=0x7f6bc8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:12:18.761 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:18.761+0000 7f6be67c1700 1 --2- 172.21.15.160:0/3752654331 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f6bc802ff60 0x7f6bc8032420 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f6bd40096a0 tx=0x7f6bd4006b40).stop 2023-12-25T20:12:18.761 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:18.761+0000 7f6be67c1700 1 -- 172.21.15.160:0/3752654331 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6be00f5410 msgr2=0x7f6be0109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:12:18.761 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:18.761+0000 7f6be67c1700 1 --2- 172.21.15.160:0/3752654331 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6be00f5410 0x7f6be0109430 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7f6bd0014000 tx=0x7f6bd000b040).stop 2023-12-25T20:12:18.761 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:18.761+0000 7f6be67c1700 1 -- 172.21.15.160:0/3752654331 shutdown_connections 2023-12-25T20:12:18.761 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:18.761+0000 7f6be67c1700 1 --2- 172.21.15.160:0/3752654331 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f6bc802ff60 0x7f6bc8032420 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:12:18.761 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:18.761+0000 7f6be67c1700 1 --2- 172.21.15.160:0/3752654331 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6be00f5410 0x7f6be0109430 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:12:18.761 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:18.761+0000 7f6be67c1700 1 -- 172.21.15.160:0/3752654331 >> 172.21.15.160:0/3752654331 conn(0x7f6be00f0b70 msgr2=0x7f6be00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:12:18.761 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:18.761+0000 7f6be67c1700 1 -- 172.21.15.160:0/3752654331 shutdown_connections 2023-12-25T20:12:18.761 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:18.761+0000 7f6be67c1700 1 -- 172.21.15.160:0/3752654331 wait complete. 2023-12-25T20:12:18.762 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:12:19.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:18 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:12:18.760023+0000 mon.smithi050 (mon.0) 223 : audit [DBG] from='client.? 172.21.15.160:0/3752654331' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:12:20.187 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:12:20.187 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:12:20.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:19 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:12:18.671771+0000 mgr.smithi050.nbzhun (mgr.14162) 153 : cluster [DBG] pgmap v146: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:12:21.422 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:21.421+0000 7f6bf8de3700 1 -- 172.21.15.160:0/1859599850 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6bf40f6510 msgr2=0x7f6bf40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:12:21.422 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:21.421+0000 7f6bf8de3700 1 --2- 172.21.15.160:0/1859599850 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6bf40f6510 0x7f6bf40f6930 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 rx=0x7f6bdc009fb0 tx=0x7f6bdc009b30).stop 2023-12-25T20:12:21.422 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:21.422+0000 7f6bf8de3700 1 -- 172.21.15.160:0/1859599850 shutdown_connections 2023-12-25T20:12:21.422 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:21.422+0000 7f6bf8de3700 1 --2- 172.21.15.160:0/1859599850 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6bf40f6510 0x7f6bf40f6930 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:12:21.422 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:21.422+0000 7f6bf8de3700 1 -- 172.21.15.160:0/1859599850 >> 172.21.15.160:0/1859599850 conn(0x7f6bf40f1c70 msgr2=0x7f6bf40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:12:21.422 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:21.422+0000 7f6bf8de3700 1 -- 172.21.15.160:0/1859599850 shutdown_connections 2023-12-25T20:12:21.422 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:21.422+0000 7f6bf8de3700 1 -- 172.21.15.160:0/1859599850 wait complete. 2023-12-25T20:12:21.423 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:21.422+0000 7f6bf8de3700 1 Processor -- start 2023-12-25T20:12:21.423 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:21.422+0000 7f6bf8de3700 1 -- start start 2023-12-25T20:12:21.423 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:21.423+0000 7f6bf8de3700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6bf40f6510 0x7f6bf4106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:12:21.423 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:21.423+0000 7f6bf8de3700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bf4106c20 con 0x7f6bf40f6510 2023-12-25T20:12:21.425 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:21.423+0000 7f6bf259c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6bf40f6510 0x7f6bf4106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:12:21.425 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:21.423+0000 7f6bf259c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6bf40f6510 0x7f6bf4106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:52968/0 (socket says 172.21.15.160:52968) 2023-12-25T20:12:21.425 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:21.423+0000 7f6bf259c700 1 -- 172.21.15.160:0/3224289753 learned_addr learned my addr 172.21.15.160:0/3224289753 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:12:21.425 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:21.423+0000 7f6bf259c700 1 -- 172.21.15.160:0/3224289753 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bdc005040 con 0x7f6bf40f6510 2023-12-25T20:12:21.425 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:21.423+0000 7f6bf259c700 1 --2- 172.21.15.160:0/3224289753 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6bf40f6510 0x7f6bf4106730 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7f6bdc013510 tx=0x7f6bdc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:12:21.425 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:21.424+0000 7f6beaffd700 1 -- 172.21.15.160:0/3224289753 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6bdc009320 con 0x7f6bf40f6510 2023-12-25T20:12:21.425 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:21.424+0000 7f6bf8de3700 1 -- 172.21.15.160:0/3224289753 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bf4106e20 con 0x7f6bf40f6510 2023-12-25T20:12:21.425 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:21.424+0000 7f6beaffd700 1 -- 172.21.15.160:0/3224289753 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6bdc013f00 con 0x7f6bf40f6510 2023-12-25T20:12:21.425 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:21.424+0000 7f6beaffd700 1 -- 172.21.15.160:0/3224289753 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6bdc014d90 con 0x7f6bf40f6510 2023-12-25T20:12:21.425 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:21.424+0000 7f6bf8de3700 1 -- 172.21.15.160:0/3224289753 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bf41072b0 con 0x7f6bf40f6510 2023-12-25T20:12:21.426 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:21.424+0000 7f6beaffd700 1 -- 172.21.15.160:0/3224289753 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6bdc0095f0 con 0x7f6bf40f6510 2023-12-25T20:12:21.426 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:21.424+0000 7f6bf8de3700 1 -- 172.21.15.160:0/3224289753 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bf4045ad0 con 0x7f6bf40f6510 2023-12-25T20:12:21.426 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:21.425+0000 7f6beaffd700 1 --2- 172.21.15.160:0/3224289753 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f6be002ff60 0x7f6be0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:12:21.426 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:21.425+0000 7f6beaffd700 1 -- 172.21.15.160:0/3224289753 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6bdc041000 con 0x7f6bf40f6510 2023-12-25T20:12:21.426 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:21.425+0000 7f6bf1d9b700 1 --2- 172.21.15.160:0/3224289753 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f6be002ff60 0x7f6be0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:12:21.426 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:21.425+0000 7f6bf1d9b700 1 --2- 172.21.15.160:0/3224289753 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f6be002ff60 0x7f6be0032420 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f6be40096a0 tx=0x7f6be4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:12:21.429 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:21.429+0000 7f6beaffd700 1 -- 172.21.15.160:0/3224289753 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6bdc0268c0 con 0x7f6bf40f6510 2023-12-25T20:12:21.632 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:21.633+0000 7f6bf8de3700 1 -- 172.21.15.160:0/3224289753 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6bf40fbf60 con 0x7f6bf40f6510 2023-12-25T20:12:21.633 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:21.633+0000 7f6beaffd700 1 -- 172.21.15.160:0/3224289753 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6bdc010230 con 0x7f6bf40f6510 2023-12-25T20:12:21.633 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:12:21.633 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:12:21.634 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:21.635+0000 7f6bf8de3700 1 -- 172.21.15.160:0/3224289753 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f6be002ff60 msgr2=0x7f6be0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:12:21.635 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:21.635+0000 7f6bf8de3700 1 --2- 172.21.15.160:0/3224289753 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f6be002ff60 0x7f6be0032420 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f6be40096a0 tx=0x7f6be4006b40).stop 2023-12-25T20:12:21.635 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:21.635+0000 7f6bf8de3700 1 -- 172.21.15.160:0/3224289753 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6bf40f6510 msgr2=0x7f6bf4106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:12:21.635 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:21.635+0000 7f6bf8de3700 1 --2- 172.21.15.160:0/3224289753 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6bf40f6510 0x7f6bf4106730 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7f6bdc013510 tx=0x7f6bdc00b040).stop 2023-12-25T20:12:21.635 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:21.635+0000 7f6bf8de3700 1 -- 172.21.15.160:0/3224289753 shutdown_connections 2023-12-25T20:12:21.635 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:21.635+0000 7f6bf8de3700 1 --2- 172.21.15.160:0/3224289753 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f6be002ff60 0x7f6be0032420 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:12:21.635 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:21.635+0000 7f6bf8de3700 1 --2- 172.21.15.160:0/3224289753 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6bf40f6510 0x7f6bf4106730 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:12:21.635 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:21.635+0000 7f6bf8de3700 1 -- 172.21.15.160:0/3224289753 >> 172.21.15.160:0/3224289753 conn(0x7f6bf40f1c70 msgr2=0x7f6bf40f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:12:21.635 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:21.635+0000 7f6bf8de3700 1 -- 172.21.15.160:0/3224289753 shutdown_connections 2023-12-25T20:12:21.635 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:21.635+0000 7f6bf8de3700 1 -- 172.21.15.160:0/3224289753 wait complete. 2023-12-25T20:12:21.636 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:12:22.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:21 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:12:20.672168+0000 mgr.smithi050.nbzhun (mgr.14162) 154 : cluster [DBG] pgmap v147: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:12:22.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:21 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:12:21.634100+0000 mon.smithi050 (mon.0) 224 : audit [DBG] from='client.? 172.21.15.160:0/3224289753' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:12:23.058 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:12:23.058 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:12:24.177 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:23 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:12:22.672449+0000 mgr.smithi050.nbzhun (mgr.14162) 155 : cluster [DBG] pgmap v148: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:12:24.177 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:23 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:12:23.823397+0000 mon.smithi050 (mon.0) 225 : audit [INF] from='mgr.14162 172.21.15.50:0/3178367813' entity='mgr.smithi050.nbzhun' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi050", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2023-12-25T20:12:24.177 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:23 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:12:23.825802+0000 mon.smithi050 (mon.0) 226 : audit [INF] from='mgr.14162 172.21.15.50:0/3178367813' entity='mgr.smithi050.nbzhun' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi050", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2023-12-25T20:12:24.177 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:23 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:12:23.826716+0000 mon.smithi050 (mon.0) 227 : audit [DBG] from='mgr.14162 172.21.15.50:0/3178367813' entity='mgr.smithi050.nbzhun' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-25T20:12:24.227 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:24.226+0000 7f08b2389700 1 -- 172.21.15.160:0/2258401010 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f08ac0f5410 msgr2=0x7f08ac0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:12:24.227 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:24.226+0000 7f08b2389700 1 --2- 172.21.15.160:0/2258401010 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f08ac0f5410 0x7f08ac0f5830 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 rx=0x7f089c009fb0 tx=0x7f089c009b30).stop 2023-12-25T20:12:24.227 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:24.226+0000 7f08b2389700 1 -- 172.21.15.160:0/2258401010 shutdown_connections 2023-12-25T20:12:24.227 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:24.226+0000 7f08b2389700 1 --2- 172.21.15.160:0/2258401010 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f08ac0f5410 0x7f08ac0f5830 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:12:24.227 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:24.226+0000 7f08b2389700 1 -- 172.21.15.160:0/2258401010 >> 172.21.15.160:0/2258401010 conn(0x7f08ac0f0b70 msgr2=0x7f08ac0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:12:24.227 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:24.226+0000 7f08b2389700 1 -- 172.21.15.160:0/2258401010 shutdown_connections 2023-12-25T20:12:24.227 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:24.226+0000 7f08b2389700 1 -- 172.21.15.160:0/2258401010 wait complete. 2023-12-25T20:12:24.227 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:24.227+0000 7f08b2389700 1 Processor -- start 2023-12-25T20:12:24.228 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:24.227+0000 7f08b2389700 1 -- start start 2023-12-25T20:12:24.228 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:24.227+0000 7f08b2389700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f08ac0f5410 0x7f08ac106710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:12:24.228 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:24.227+0000 7f08b2389700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08ac106c00 con 0x7f08ac0f5410 2023-12-25T20:12:24.228 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:24.227+0000 7f08abfff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f08ac0f5410 0x7f08ac106710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:12:24.228 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:24.227+0000 7f08abfff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f08ac0f5410 0x7f08ac106710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:52980/0 (socket says 172.21.15.160:52980) 2023-12-25T20:12:24.228 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:24.227+0000 7f08abfff700 1 -- 172.21.15.160:0/2733760707 learned_addr learned my addr 172.21.15.160:0/2733760707 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:12:24.228 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:24.227+0000 7f08abfff700 1 -- 172.21.15.160:0/2733760707 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f089c005040 con 0x7f08ac0f5410 2023-12-25T20:12:24.228 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:24.227+0000 7f08abfff700 1 --2- 172.21.15.160:0/2733760707 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f08ac0f5410 0x7f08ac106710 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7f089c013510 tx=0x7f089c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:12:24.229 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:24.228+0000 7f08a8ff9700 1 -- 172.21.15.160:0/2733760707 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f089c009320 con 0x7f08ac0f5410 2023-12-25T20:12:24.229 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:24.228+0000 7f08a8ff9700 1 -- 172.21.15.160:0/2733760707 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f089c013f00 con 0x7f08ac0f5410 2023-12-25T20:12:24.229 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:24.228+0000 7f08a8ff9700 1 -- 172.21.15.160:0/2733760707 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f089c014d90 con 0x7f08ac0f5410 2023-12-25T20:12:24.230 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:24.228+0000 7f08b2389700 1 -- 172.21.15.160:0/2733760707 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08ac106e00 con 0x7f08ac0f5410 2023-12-25T20:12:24.230 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:24.228+0000 7f08b2389700 1 -- 172.21.15.160:0/2733760707 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08ac107290 con 0x7f08ac0f5410 2023-12-25T20:12:24.230 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:24.229+0000 7f08a8ff9700 1 -- 172.21.15.160:0/2733760707 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f089c014420 con 0x7f08ac0f5410 2023-12-25T20:12:24.230 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:24.229+0000 7f08a8ff9700 1 --2- 172.21.15.160:0/2733760707 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f089802ffb0 0x7f0898032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:12:24.230 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:24.229+0000 7f08b2389700 1 -- 172.21.15.160:0/2733760707 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08ac045ad0 con 0x7f08ac0f5410 2023-12-25T20:12:24.230 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:24.229+0000 7f08a8ff9700 1 -- 172.21.15.160:0/2733760707 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f089c040fa0 con 0x7f08ac0f5410 2023-12-25T20:12:24.230 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:24.229+0000 7f08ab7fe700 1 --2- 172.21.15.160:0/2733760707 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f089802ffb0 0x7f0898032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:12:24.230 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:24.229+0000 7f08ab7fe700 1 --2- 172.21.15.160:0/2733760707 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f089802ffb0 0x7f0898032470 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f08940096a0 tx=0x7f0894006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:12:24.233 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:24.233+0000 7f08a8ff9700 1 -- 172.21.15.160:0/2733760707 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f089c0095f0 con 0x7f08ac0f5410 2023-12-25T20:12:24.439 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:24.439+0000 7f08b2389700 1 -- 172.21.15.160:0/2733760707 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f08ac0faf50 con 0x7f08ac0f5410 2023-12-25T20:12:24.439 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:24.440+0000 7f08a8ff9700 1 -- 172.21.15.160:0/2733760707 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f089c01db70 con 0x7f08ac0f5410 2023-12-25T20:12:24.439 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:12:24.439 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:12:24.441 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:24.441+0000 7f08b2389700 1 -- 172.21.15.160:0/2733760707 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f089802ffb0 msgr2=0x7f0898032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:12:24.441 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:24.441+0000 7f08b2389700 1 --2- 172.21.15.160:0/2733760707 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f089802ffb0 0x7f0898032470 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f08940096a0 tx=0x7f0894006b40).stop 2023-12-25T20:12:24.441 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:24.441+0000 7f08b2389700 1 -- 172.21.15.160:0/2733760707 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f08ac0f5410 msgr2=0x7f08ac106710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:12:24.441 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:24.441+0000 7f08b2389700 1 --2- 172.21.15.160:0/2733760707 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f08ac0f5410 0x7f08ac106710 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7f089c013510 tx=0x7f089c00b040).stop 2023-12-25T20:12:24.441 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:24.441+0000 7f08b2389700 1 -- 172.21.15.160:0/2733760707 shutdown_connections 2023-12-25T20:12:24.442 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:24.441+0000 7f08b2389700 1 --2- 172.21.15.160:0/2733760707 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f089802ffb0 0x7f0898032470 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:12:24.442 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:24.441+0000 7f08b2389700 1 --2- 172.21.15.160:0/2733760707 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f08ac0f5410 0x7f08ac106710 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:12:24.442 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:24.441+0000 7f08b2389700 1 -- 172.21.15.160:0/2733760707 >> 172.21.15.160:0/2733760707 conn(0x7f08ac0f0b70 msgr2=0x7f08ac05fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:12:24.442 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:24.441+0000 7f08b2389700 1 -- 172.21.15.160:0/2733760707 shutdown_connections 2023-12-25T20:12:24.442 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:24.441+0000 7f08b2389700 1 -- 172.21.15.160:0/2733760707 wait complete. 2023-12-25T20:12:24.442 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:12:25.215 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:24 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cephadm 2023-12-25T20:12:23.821728+0000 mgr.smithi050.nbzhun (mgr.14162) 156 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon alertmanager.smithi050 ... 2023-12-25T20:12:25.216 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:24 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi050 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/alertmanager 2023-12-25T20:12:25.216 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:24 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2023-12-25T20:12:25.216 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:24 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: stat: stderr time="2023-12-25T20:08:02Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-12-25T20:12:25.216 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:24 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: stat: stderr time="2023-12-25T20:08:39Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-12-25T20:12:25.216 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:24 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: stat: stderr time="2023-12-25T20:09:17Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-12-25T20:12:25.216 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:24 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2023-12-25T20:12:25.216 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:24 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi050 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/prometheus 2023-12-25T20:12:25.216 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:24 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2023-12-25T20:12:25.216 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:24 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: stat: stderr time="2023-12-25T20:10:31Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-12-25T20:12:25.216 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:24 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: stat: stderr time="2023-12-25T20:11:08Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-12-25T20:12:25.216 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:24 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: stat: stderr time="2023-12-25T20:11:46Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-12-25T20:12:25.216 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:24 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2023-12-25T20:12:25.216 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:24 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: Traceback (most recent call last): 2023-12-25T20:12:25.217 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:24 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: File "/var/lib/ceph/02c8f25c-a361-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2023-12-25T20:12:25.217 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:24 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: main() 2023-12-25T20:12:25.217 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:24 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: File "/var/lib/ceph/02c8f25c-a361-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2023-12-25T20:12:25.217 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:24 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: r = ctx.func(ctx) 2023-12-25T20:12:25.217 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:24 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: File "/var/lib/ceph/02c8f25c-a361-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2023-12-25T20:12:25.217 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:24 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: return func(ctx) 2023-12-25T20:12:25.217 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:24 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: File "/var/lib/ceph/02c8f25c-a361-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2023-12-25T20:12:25.217 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:24 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2023-12-25T20:12:25.217 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:24 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: File "/var/lib/ceph/02c8f25c-a361-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4191, in extract_uid_gid_monitoring 2023-12-25T20:12:25.217 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:24 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: uid, gid = extract_uid_gid(ctx, file_path=['/etc/alertmanager', '/etc/prometheus']) 2023-12-25T20:12:25.217 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:24 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: File "/var/lib/ceph/02c8f25c-a361-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2023-12-25T20:12:25.217 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:24 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: raise RuntimeError('uid/gid not found') 2023-12-25T20:12:25.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:24 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: RuntimeError: uid/gid not found 2023-12-25T20:12:25.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:24 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: Traceback (most recent call last): 2023-12-25T20:12:25.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:24 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2023-12-25T20:12:25.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:24 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: yield (conn, connr) 2023-12-25T20:12:25.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:24 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2023-12-25T20:12:25.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:24 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: code, '\n'.join(err))) 2023-12-25T20:12:25.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:24 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon alertmanager.smithi050 ... 2023-12-25T20:12:25.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:24 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi050 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/alertmanager 2023-12-25T20:12:25.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:24 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2023-12-25T20:12:25.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:24 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: stat: stderr time="2023-12-25T20:08:02Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-12-25T20:12:25.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:24 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: stat: stderr time="2023-12-25T20:08:39Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-12-25T20:12:25.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:24 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: stat: stderr time="2023-12-25T20:09:17Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-12-25T20:12:25.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:24 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2023-12-25T20:12:25.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:24 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi050 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/prometheus 2023-12-25T20:12:25.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:24 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2023-12-25T20:12:25.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:24 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: stat: stderr time="2023-12-25T20:10:31Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-12-25T20:12:25.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:24 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: stat: stderr time="2023-12-25T20:11:08Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-12-25T20:12:25.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:24 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: stat: stderr time="2023-12-25T20:11:46Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-12-25T20:12:25.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:24 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2023-12-25T20:12:25.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:24 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: Traceback (most recent call last): 2023-12-25T20:12:25.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:24 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: File "/var/lib/ceph/02c8f25c-a361-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2023-12-25T20:12:25.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:24 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: main() 2023-12-25T20:12:25.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:24 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: File "/var/lib/ceph/02c8f25c-a361-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2023-12-25T20:12:25.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:24 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: r = ctx.func(ctx) 2023-12-25T20:12:25.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:24 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: File "/var/lib/ceph/02c8f25c-a361-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2023-12-25T20:12:25.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:24 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: return func(ctx) 2023-12-25T20:12:25.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:24 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: File "/var/lib/ceph/02c8f25c-a361-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2023-12-25T20:12:25.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:24 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2023-12-25T20:12:25.220 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:24 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: File "/var/lib/ceph/02c8f25c-a361-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4191, in extract_uid_gid_monitoring 2023-12-25T20:12:25.220 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:24 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: uid, gid = extract_uid_gid(ctx, file_path=['/etc/alertmanager', '/etc/prometheus']) 2023-12-25T20:12:25.220 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:24 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: File "/var/lib/ceph/02c8f25c-a361-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2023-12-25T20:12:25.220 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:24 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: raise RuntimeError('uid/gid not found') 2023-12-25T20:12:25.220 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:24 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: RuntimeError: uid/gid not found 2023-12-25T20:12:25.220 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:24 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cephadm 2023-12-25T20:12:23.827506+0000 mgr.smithi050.nbzhun (mgr.14162) 157 : cephadm [INF] Deploying daemon crash.smithi050 on smithi050 2023-12-25T20:12:25.220 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:24 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:12:24.440351+0000 mon.smithi050 (mon.0) 228 : audit [DBG] from='client.? 172.21.15.160:0/2733760707' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:12:25.865 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:12:25.866 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:12:26.718 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:26 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:12:24.672776+0000 mgr.smithi050.nbzhun (mgr.14162) 158 : cluster [DBG] pgmap v149: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:12:26.719 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:26 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:12:25.421410+0000 mon.smithi050 (mon.0) 229 : audit [INF] from='mgr.14162 172.21.15.50:0/3178367813' entity='mgr.smithi050.nbzhun' 2023-12-25T20:12:26.719 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:26 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:12:25.670372+0000 mon.smithi050 (mon.0) 230 : audit [INF] from='mgr.14162 172.21.15.50:0/3178367813' entity='mgr.smithi050.nbzhun' 2023-12-25T20:12:26.719 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:26 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:12:25.672669+0000 mon.smithi050 (mon.0) 231 : audit [INF] from='mgr.14162 172.21.15.50:0/3178367813' entity='mgr.smithi050.nbzhun' 2023-12-25T20:12:26.719 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:26 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:12:25.673125+0000 mon.smithi050 (mon.0) 232 : audit [INF] from='mgr.14162 172.21.15.50:0/3178367813' entity='mgr.smithi050.nbzhun' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2023-12-25T20:12:26.719 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:26 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:12:25.675791+0000 mon.smithi050 (mon.0) 233 : audit [INF] from='mgr.14162 172.21.15.50:0/3178367813' entity='mgr.smithi050.nbzhun' 2023-12-25T20:12:27.118 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:27.118+0000 7fe02ecb9700 1 -- 172.21.15.160:0/737720236 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe0280f5410 msgr2=0x7fe0280f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:12:27.118 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:27.118+0000 7fe02ecb9700 1 --2- 172.21.15.160:0/737720236 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe0280f5410 0x7fe0280f5830 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 rx=0x7fe01c009fb0 tx=0x7fe01c009b30).stop 2023-12-25T20:12:27.118 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:27.118+0000 7fe02ecb9700 1 -- 172.21.15.160:0/737720236 shutdown_connections 2023-12-25T20:12:27.118 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:27.118+0000 7fe02ecb9700 1 --2- 172.21.15.160:0/737720236 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe0280f5410 0x7fe0280f5830 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:12:27.118 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:27.118+0000 7fe02ecb9700 1 -- 172.21.15.160:0/737720236 >> 172.21.15.160:0/737720236 conn(0x7fe0280f0b70 msgr2=0x7fe0280f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:12:27.119 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:27.118+0000 7fe02ecb9700 1 -- 172.21.15.160:0/737720236 shutdown_connections 2023-12-25T20:12:27.119 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:27.118+0000 7fe02ecb9700 1 -- 172.21.15.160:0/737720236 wait complete. 2023-12-25T20:12:27.119 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:27.118+0000 7fe02ecb9700 1 Processor -- start 2023-12-25T20:12:27.119 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:27.118+0000 7fe02ecb9700 1 -- start start 2023-12-25T20:12:27.119 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:27.119+0000 7fe02ecb9700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe0280f5410 0x7fe0280fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:12:27.119 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:27.119+0000 7fe02ecb9700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0280fb820 con 0x7fe0280f5410 2023-12-25T20:12:27.121 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:27.119+0000 7fe02ca55700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe0280f5410 0x7fe0280fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:12:27.121 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:27.119+0000 7fe02ca55700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe0280f5410 0x7fe0280fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:52996/0 (socket says 172.21.15.160:52996) 2023-12-25T20:12:27.121 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:27.119+0000 7fe02ca55700 1 -- 172.21.15.160:0/2365163990 learned_addr learned my addr 172.21.15.160:0/2365163990 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:12:27.121 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:27.119+0000 7fe02ca55700 1 -- 172.21.15.160:0/2365163990 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe01c005040 con 0x7fe0280f5410 2023-12-25T20:12:27.121 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:27.119+0000 7fe02ca55700 1 --2- 172.21.15.160:0/2365163990 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe0280f5410 0x7fe0280fd3a0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7fe01c013510 tx=0x7fe01c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:12:27.121 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:27.120+0000 7fe0257fa700 1 -- 172.21.15.160:0/2365163990 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe01c009320 con 0x7fe0280f5410 2023-12-25T20:12:27.121 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:27.120+0000 7fe02ecb9700 1 -- 172.21.15.160:0/2365163990 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0280fb9c0 con 0x7fe0280f5410 2023-12-25T20:12:27.121 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:27.120+0000 7fe0257fa700 1 -- 172.21.15.160:0/2365163990 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe01c013f00 con 0x7fe0280f5410 2023-12-25T20:12:27.121 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:27.120+0000 7fe0257fa700 1 -- 172.21.15.160:0/2365163990 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe01c014d90 con 0x7fe0280f5410 2023-12-25T20:12:27.121 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:27.120+0000 7fe02ecb9700 1 -- 172.21.15.160:0/2365163990 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0280fbe50 con 0x7fe0280f5410 2023-12-25T20:12:27.121 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:27.120+0000 7fe0257fa700 1 -- 172.21.15.160:0/2365163990 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe01c0095f0 con 0x7fe0280f5410 2023-12-25T20:12:27.122 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:27.121+0000 7fe02ecb9700 1 -- 172.21.15.160:0/2365163990 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe028059100 con 0x7fe0280f5410 2023-12-25T20:12:27.122 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:27.121+0000 7fe0257fa700 1 --2- 172.21.15.160:0/2365163990 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fe01802ff60 0x7fe018032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:12:27.122 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:27.121+0000 7fe0257fa700 1 -- 172.21.15.160:0/2365163990 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe01c041000 con 0x7fe0280f5410 2023-12-25T20:12:27.122 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:27.121+0000 7fe027fff700 1 --2- 172.21.15.160:0/2365163990 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fe01802ff60 0x7fe018032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:12:27.122 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:27.121+0000 7fe027fff700 1 --2- 172.21.15.160:0/2365163990 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fe01802ff60 0x7fe018032420 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7fe0140096a0 tx=0x7fe014006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:12:27.125 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:27.126+0000 7fe0257fa700 1 -- 172.21.15.160:0/2365163990 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe01c014420 con 0x7fe0280f5410 2023-12-25T20:12:27.329 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:27.329+0000 7fe02ecb9700 1 -- 172.21.15.160:0/2365163990 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe0280fae60 con 0x7fe0280f5410 2023-12-25T20:12:27.329 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:27.330+0000 7fe0257fa700 1 -- 172.21.15.160:0/2365163990 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe01c010230 con 0x7fe0280f5410 2023-12-25T20:12:27.329 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:12:27.330 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:12:27.331 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:27.331+0000 7fe02ecb9700 1 -- 172.21.15.160:0/2365163990 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fe01802ff60 msgr2=0x7fe018032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:12:27.331 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:27.331+0000 7fe02ecb9700 1 --2- 172.21.15.160:0/2365163990 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fe01802ff60 0x7fe018032420 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7fe0140096a0 tx=0x7fe014006b40).stop 2023-12-25T20:12:27.331 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:27.331+0000 7fe02ecb9700 1 -- 172.21.15.160:0/2365163990 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe0280f5410 msgr2=0x7fe0280fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:12:27.331 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:27.331+0000 7fe02ecb9700 1 --2- 172.21.15.160:0/2365163990 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe0280f5410 0x7fe0280fd3a0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7fe01c013510 tx=0x7fe01c00b040).stop 2023-12-25T20:12:27.331 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:27.331+0000 7fe02ecb9700 1 -- 172.21.15.160:0/2365163990 shutdown_connections 2023-12-25T20:12:27.331 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:27.331+0000 7fe02ecb9700 1 --2- 172.21.15.160:0/2365163990 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fe01802ff60 0x7fe018032420 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:12:27.331 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:27.331+0000 7fe02ecb9700 1 --2- 172.21.15.160:0/2365163990 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe0280f5410 0x7fe0280fd3a0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:12:27.332 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:27.331+0000 7fe02ecb9700 1 -- 172.21.15.160:0/2365163990 >> 172.21.15.160:0/2365163990 conn(0x7fe0280f0b70 msgr2=0x7fe02805fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:12:27.332 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:27.331+0000 7fe02ecb9700 1 -- 172.21.15.160:0/2365163990 shutdown_connections 2023-12-25T20:12:27.332 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:27.331+0000 7fe02ecb9700 1 -- 172.21.15.160:0/2365163990 wait complete. 2023-12-25T20:12:27.332 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:12:27.719 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:27 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:12:25.673522+0000 mgr.smithi050.nbzhun (mgr.14162) 159 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2023-12-25T20:12:27.719 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:27 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cephadm 2023-12-25T20:12:25.678745+0000 mgr.smithi050.nbzhun (mgr.14162) 160 : cephadm [INF] Deploying daemon grafana.smithi050 on smithi050 2023-12-25T20:12:27.719 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:27 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:12:27.330334+0000 mon.smithi050 (mon.0) 234 : audit [DBG] from='client.? 172.21.15.160:0/2365163990' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:12:28.718 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:28 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:12:26.673119+0000 mgr.smithi050.nbzhun (mgr.14162) 161 : cluster [DBG] pgmap v150: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:12:28.719 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:12:28.719 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:12:29.990 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:29.988+0000 7f2e1bf7e700 1 -- 172.21.15.160:0/3532694453 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2e140f5410 msgr2=0x7f2e140f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:12:29.990 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:29.988+0000 7f2e1bf7e700 1 --2- 172.21.15.160:0/3532694453 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2e140f5410 0x7f2e140f5830 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0x7f2e10009fb0 tx=0x7f2e10009b30).stop 2023-12-25T20:12:29.990 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:29.988+0000 7f2e1bf7e700 1 -- 172.21.15.160:0/3532694453 shutdown_connections 2023-12-25T20:12:29.990 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:29.988+0000 7f2e1bf7e700 1 --2- 172.21.15.160:0/3532694453 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2e140f5410 0x7f2e140f5830 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:12:29.990 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:29.988+0000 7f2e1bf7e700 1 -- 172.21.15.160:0/3532694453 >> 172.21.15.160:0/3532694453 conn(0x7f2e140f0b70 msgr2=0x7f2e140f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:12:29.990 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:29.988+0000 7f2e1bf7e700 1 -- 172.21.15.160:0/3532694453 shutdown_connections 2023-12-25T20:12:29.990 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:29.988+0000 7f2e1bf7e700 1 -- 172.21.15.160:0/3532694453 wait complete. 2023-12-25T20:12:29.990 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:29.989+0000 7f2e1bf7e700 1 Processor -- start 2023-12-25T20:12:29.990 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:29.989+0000 7f2e1bf7e700 1 -- start start 2023-12-25T20:12:29.990 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:29.989+0000 7f2e1bf7e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2e140f5410 0x7f2e14109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:12:29.991 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:29.989+0000 7f2e1bf7e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e14060220 con 0x7f2e140f5410 2023-12-25T20:12:29.991 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:29.989+0000 7f2e19d1a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2e140f5410 0x7f2e14109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:12:29.991 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:29.989+0000 7f2e19d1a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2e140f5410 0x7f2e14109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:56734/0 (socket says 172.21.15.160:56734) 2023-12-25T20:12:29.991 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:29.989+0000 7f2e19d1a700 1 -- 172.21.15.160:0/2906083058 learned_addr learned my addr 172.21.15.160:0/2906083058 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:12:29.991 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:29.989+0000 7f2e19d1a700 1 -- 172.21.15.160:0/2906083058 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e10005040 con 0x7f2e140f5410 2023-12-25T20:12:29.991 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:29.989+0000 7f2e19d1a700 1 --2- 172.21.15.160:0/2906083058 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2e140f5410 0x7f2e14109450 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7f2e10013510 tx=0x7f2e1000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:12:29.991 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:29.990+0000 7f2e0a7fc700 1 -- 172.21.15.160:0/2906083058 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2e10009320 con 0x7f2e140f5410 2023-12-25T20:12:29.991 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:29.990+0000 7f2e1bf7e700 1 -- 172.21.15.160:0/2906083058 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e14060420 con 0x7f2e140f5410 2023-12-25T20:12:29.991 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:29.990+0000 7f2e0a7fc700 1 -- 172.21.15.160:0/2906083058 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2e10013f00 con 0x7f2e140f5410 2023-12-25T20:12:29.991 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:29.990+0000 7f2e0a7fc700 1 -- 172.21.15.160:0/2906083058 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2e10014d90 con 0x7f2e140f5410 2023-12-25T20:12:29.991 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:29.990+0000 7f2e1bf7e700 1 -- 172.21.15.160:0/2906083058 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e14109b00 con 0x7f2e140f5410 2023-12-25T20:12:29.992 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:29.990+0000 7f2e1bf7e700 1 -- 172.21.15.160:0/2906083058 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e14059100 con 0x7f2e140f5410 2023-12-25T20:12:29.992 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:29.991+0000 7f2e0a7fc700 1 -- 172.21.15.160:0/2906083058 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2e100095f0 con 0x7f2e140f5410 2023-12-25T20:12:29.992 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:29.991+0000 7f2e0a7fc700 1 --2- 172.21.15.160:0/2906083058 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f2e0002ff60 0x7f2e00032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:12:29.992 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:29.991+0000 7f2e0a7fc700 1 -- 172.21.15.160:0/2906083058 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2e10040fe0 con 0x7f2e140f5410 2023-12-25T20:12:29.992 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:29.991+0000 7f2e19519700 1 --2- 172.21.15.160:0/2906083058 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f2e0002ff60 0x7f2e00032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:12:29.992 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:29.991+0000 7f2e19519700 1 --2- 172.21.15.160:0/2906083058 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f2e0002ff60 0x7f2e00032420 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f2e040096a0 tx=0x7f2e04006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:12:29.995 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:29.995+0000 7f2e0a7fc700 1 -- 172.21.15.160:0/2906083058 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2e10010230 con 0x7f2e140f5410 2023-12-25T20:12:30.198 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:30.198+0000 7f2e1bf7e700 1 -- 172.21.15.160:0/2906083058 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2e14045ad0 con 0x7f2e140f5410 2023-12-25T20:12:30.198 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:30.199+0000 7f2e0a7fc700 1 -- 172.21.15.160:0/2906083058 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2e10010230 con 0x7f2e140f5410 2023-12-25T20:12:30.199 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:12:30.199 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:12:30.201 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:30.200+0000 7f2e1bf7e700 1 -- 172.21.15.160:0/2906083058 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f2e0002ff60 msgr2=0x7f2e00032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:12:30.201 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:30.200+0000 7f2e1bf7e700 1 --2- 172.21.15.160:0/2906083058 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f2e0002ff60 0x7f2e00032420 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f2e040096a0 tx=0x7f2e04006b40).stop 2023-12-25T20:12:30.201 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:30.200+0000 7f2e1bf7e700 1 -- 172.21.15.160:0/2906083058 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2e140f5410 msgr2=0x7f2e14109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:12:30.201 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:30.200+0000 7f2e1bf7e700 1 --2- 172.21.15.160:0/2906083058 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2e140f5410 0x7f2e14109450 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7f2e10013510 tx=0x7f2e1000b040).stop 2023-12-25T20:12:30.201 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:30.200+0000 7f2e1bf7e700 1 -- 172.21.15.160:0/2906083058 shutdown_connections 2023-12-25T20:12:30.201 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:30.200+0000 7f2e1bf7e700 1 --2- 172.21.15.160:0/2906083058 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f2e0002ff60 0x7f2e00032420 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:12:30.201 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:30.200+0000 7f2e1bf7e700 1 --2- 172.21.15.160:0/2906083058 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2e140f5410 0x7f2e14109450 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:12:30.201 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:30.200+0000 7f2e1bf7e700 1 -- 172.21.15.160:0/2906083058 >> 172.21.15.160:0/2906083058 conn(0x7f2e140f0b70 msgr2=0x7f2e140f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:12:30.201 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:30.201+0000 7f2e1bf7e700 1 -- 172.21.15.160:0/2906083058 shutdown_connections 2023-12-25T20:12:30.201 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:30.201+0000 7f2e1bf7e700 1 -- 172.21.15.160:0/2906083058 wait complete. 2023-12-25T20:12:30.202 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:12:30.718 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:30 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:12:28.673392+0000 mgr.smithi050.nbzhun (mgr.14162) 162 : cluster [DBG] pgmap v151: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:12:30.719 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:30 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:12:30.199481+0000 mon.smithi050 (mon.0) 235 : audit [DBG] from='client.? 172.21.15.160:0/2906083058' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:12:31.619 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:12:31.620 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:12:32.718 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:32 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:12:30.673753+0000 mgr.smithi050.nbzhun (mgr.14162) 163 : cluster [DBG] pgmap v152: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:12:32.850 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:32.849+0000 7f1cf23db700 1 -- 172.21.15.160:0/722302877 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1cec0f6510 msgr2=0x7f1cec0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:12:32.851 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:32.849+0000 7f1cf23db700 1 --2- 172.21.15.160:0/722302877 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1cec0f6510 0x7f1cec0f6930 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 rx=0x7f1cdc009fb0 tx=0x7f1cdc009b30).stop 2023-12-25T20:12:32.851 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:32.849+0000 7f1cf23db700 1 -- 172.21.15.160:0/722302877 shutdown_connections 2023-12-25T20:12:32.851 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:32.849+0000 7f1cf23db700 1 --2- 172.21.15.160:0/722302877 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1cec0f6510 0x7f1cec0f6930 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:12:32.851 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:32.849+0000 7f1cf23db700 1 -- 172.21.15.160:0/722302877 >> 172.21.15.160:0/722302877 conn(0x7f1cec0f1c70 msgr2=0x7f1cec0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:12:32.851 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:32.849+0000 7f1cf23db700 1 -- 172.21.15.160:0/722302877 shutdown_connections 2023-12-25T20:12:32.851 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:32.849+0000 7f1cf23db700 1 -- 172.21.15.160:0/722302877 wait complete. 2023-12-25T20:12:32.851 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:32.850+0000 7f1cf23db700 1 Processor -- start 2023-12-25T20:12:32.851 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:32.850+0000 7f1cf23db700 1 -- start start 2023-12-25T20:12:32.851 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:32.850+0000 7f1cf23db700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1cec0f6510 0x7f1cec106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:12:32.851 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:32.850+0000 7f1cf23db700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cec106c20 con 0x7f1cec0f6510 2023-12-25T20:12:32.851 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:32.850+0000 7f1cebfff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1cec0f6510 0x7f1cec106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:12:32.851 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:32.850+0000 7f1cebfff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1cec0f6510 0x7f1cec106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:56752/0 (socket says 172.21.15.160:56752) 2023-12-25T20:12:32.852 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:32.850+0000 7f1cebfff700 1 -- 172.21.15.160:0/3311272112 learned_addr learned my addr 172.21.15.160:0/3311272112 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:12:32.852 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:32.850+0000 7f1cebfff700 1 -- 172.21.15.160:0/3311272112 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cdc005040 con 0x7f1cec0f6510 2023-12-25T20:12:32.852 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:32.851+0000 7f1cebfff700 1 --2- 172.21.15.160:0/3311272112 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1cec0f6510 0x7f1cec106730 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7f1cdc013510 tx=0x7f1cdc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:12:32.852 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:32.851+0000 7f1ce8ff9700 1 -- 172.21.15.160:0/3311272112 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1cdc009320 con 0x7f1cec0f6510 2023-12-25T20:12:32.852 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:32.851+0000 7f1cf23db700 1 -- 172.21.15.160:0/3311272112 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cec106e20 con 0x7f1cec0f6510 2023-12-25T20:12:32.852 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:32.851+0000 7f1ce8ff9700 1 -- 172.21.15.160:0/3311272112 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1cdc013f00 con 0x7f1cec0f6510 2023-12-25T20:12:32.852 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:32.851+0000 7f1ce8ff9700 1 -- 172.21.15.160:0/3311272112 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1cdc014d90 con 0x7f1cec0f6510 2023-12-25T20:12:32.852 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:32.851+0000 7f1cf23db700 1 -- 172.21.15.160:0/3311272112 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cec1072b0 con 0x7f1cec0f6510 2023-12-25T20:12:32.853 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:32.852+0000 7f1ce8ff9700 1 -- 172.21.15.160:0/3311272112 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1cdc0095f0 con 0x7f1cec0f6510 2023-12-25T20:12:32.853 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:32.852+0000 7f1cf23db700 1 -- 172.21.15.160:0/3311272112 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cec045ad0 con 0x7f1cec0f6510 2023-12-25T20:12:32.853 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:32.852+0000 7f1ce8ff9700 1 --2- 172.21.15.160:0/3311272112 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f1cd402ff60 0x7f1cd4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:12:32.853 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:32.852+0000 7f1ce8ff9700 1 -- 172.21.15.160:0/3311272112 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1cdc041000 con 0x7f1cec0f6510 2023-12-25T20:12:32.853 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:32.852+0000 7f1ceb7fe700 1 --2- 172.21.15.160:0/3311272112 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f1cd402ff60 0x7f1cd4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:12:32.853 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:32.852+0000 7f1ceb7fe700 1 --2- 172.21.15.160:0/3311272112 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f1cd402ff60 0x7f1cd4032420 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f1ce00096a0 tx=0x7f1ce0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:12:32.858 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:32.858+0000 7f1ce8ff9700 1 -- 172.21.15.160:0/3311272112 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1cdc014420 con 0x7f1cec0f6510 2023-12-25T20:12:33.061 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:33.061+0000 7f1cf23db700 1 -- 172.21.15.160:0/3311272112 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1cec0fbf60 con 0x7f1cec0f6510 2023-12-25T20:12:33.061 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:33.062+0000 7f1ce8ff9700 1 -- 172.21.15.160:0/3311272112 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1cdc01d7f0 con 0x7f1cec0f6510 2023-12-25T20:12:33.062 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:12:33.062 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:12:33.063 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:33.063+0000 7f1cf23db700 1 -- 172.21.15.160:0/3311272112 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f1cd402ff60 msgr2=0x7f1cd4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:12:33.063 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:33.063+0000 7f1cf23db700 1 --2- 172.21.15.160:0/3311272112 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f1cd402ff60 0x7f1cd4032420 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f1ce00096a0 tx=0x7f1ce0006b40).stop 2023-12-25T20:12:33.063 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:33.063+0000 7f1cf23db700 1 -- 172.21.15.160:0/3311272112 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1cec0f6510 msgr2=0x7f1cec106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:12:33.063 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:33.063+0000 7f1cf23db700 1 --2- 172.21.15.160:0/3311272112 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1cec0f6510 0x7f1cec106730 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7f1cdc013510 tx=0x7f1cdc00b040).stop 2023-12-25T20:12:33.063 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:33.063+0000 7f1cf23db700 1 -- 172.21.15.160:0/3311272112 shutdown_connections 2023-12-25T20:12:33.063 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:33.063+0000 7f1cf23db700 1 --2- 172.21.15.160:0/3311272112 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f1cd402ff60 0x7f1cd4032420 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:12:33.063 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:33.063+0000 7f1cf23db700 1 --2- 172.21.15.160:0/3311272112 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1cec0f6510 0x7f1cec106730 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:12:33.063 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:33.063+0000 7f1cf23db700 1 -- 172.21.15.160:0/3311272112 >> 172.21.15.160:0/3311272112 conn(0x7f1cec0f1c70 msgr2=0x7f1cec0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:12:33.064 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:33.063+0000 7f1cf23db700 1 -- 172.21.15.160:0/3311272112 shutdown_connections 2023-12-25T20:12:33.064 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:33.063+0000 7f1cf23db700 1 -- 172.21.15.160:0/3311272112 wait complete. 2023-12-25T20:12:33.064 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:12:33.718 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:33 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:12:33.062355+0000 mon.smithi050 (mon.0) 236 : audit [DBG] from='client.? 172.21.15.160:0/3311272112' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:12:34.495 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:12:34.496 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:12:34.718 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:34 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:12:32.674156+0000 mgr.smithi050.nbzhun (mgr.14162) 164 : cluster [DBG] pgmap v153: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:12:35.718 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:35.716+0000 7f0696b09700 1 -- 172.21.15.160:0/3544737291 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f06900f5410 msgr2=0x7f06900f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:12:35.718 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:35.716+0000 7f0696b09700 1 --2- 172.21.15.160:0/3544737291 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f06900f5410 0x7f06900f5830 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 rx=0x7f0684009fb0 tx=0x7f0684009b30).stop 2023-12-25T20:12:35.718 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:35.716+0000 7f0696b09700 1 -- 172.21.15.160:0/3544737291 shutdown_connections 2023-12-25T20:12:35.718 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:35.716+0000 7f0696b09700 1 --2- 172.21.15.160:0/3544737291 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f06900f5410 0x7f06900f5830 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:12:35.718 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:35.716+0000 7f0696b09700 1 -- 172.21.15.160:0/3544737291 >> 172.21.15.160:0/3544737291 conn(0x7f06900f0b70 msgr2=0x7f06900f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:12:35.718 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:35.716+0000 7f0696b09700 1 -- 172.21.15.160:0/3544737291 shutdown_connections 2023-12-25T20:12:35.718 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:35.716+0000 7f0696b09700 1 -- 172.21.15.160:0/3544737291 wait complete. 2023-12-25T20:12:35.718 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:35.717+0000 7f0696b09700 1 Processor -- start 2023-12-25T20:12:35.718 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:35.717+0000 7f0696b09700 1 -- start start 2023-12-25T20:12:35.718 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:35.717+0000 7f0696b09700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f06900f5410 0x7f0690105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:12:35.719 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:35.717+0000 7f0696b09700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0690105b20 con 0x7f06900f5410 2023-12-25T20:12:35.719 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:35.717+0000 7f06948a5700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f06900f5410 0x7f0690105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:12:35.719 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:35.717+0000 7f06948a5700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f06900f5410 0x7f0690105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:56766/0 (socket says 172.21.15.160:56766) 2023-12-25T20:12:35.719 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:35.717+0000 7f06948a5700 1 -- 172.21.15.160:0/1708409991 learned_addr learned my addr 172.21.15.160:0/1708409991 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:12:35.719 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:35.718+0000 7f06948a5700 1 -- 172.21.15.160:0/1708409991 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0684005040 con 0x7f06900f5410 2023-12-25T20:12:35.719 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:35.718+0000 7f06948a5700 1 --2- 172.21.15.160:0/1708409991 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f06900f5410 0x7f0690105630 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7f0684013510 tx=0x7f068400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:12:35.719 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:35.718+0000 7f068d7fa700 1 -- 172.21.15.160:0/1708409991 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0684009320 con 0x7f06900f5410 2023-12-25T20:12:35.719 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:35.718+0000 7f068d7fa700 1 -- 172.21.15.160:0/1708409991 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0684013f00 con 0x7f06900f5410 2023-12-25T20:12:35.719 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:35.718+0000 7f0696b09700 1 -- 172.21.15.160:0/1708409991 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0690105d20 con 0x7f06900f5410 2023-12-25T20:12:35.720 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:35.718+0000 7f068d7fa700 1 -- 172.21.15.160:0/1708409991 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0684014d90 con 0x7f06900f5410 2023-12-25T20:12:35.720 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:35.718+0000 7f0696b09700 1 -- 172.21.15.160:0/1708409991 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06901061b0 con 0x7f06900f5410 2023-12-25T20:12:35.720 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:35.719+0000 7f068d7fa700 1 -- 172.21.15.160:0/1708409991 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f068400f410 con 0x7f06900f5410 2023-12-25T20:12:35.721 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:35.719+0000 7f068d7fa700 1 --2- 172.21.15.160:0/1708409991 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f068002ffb0 0x7f0680032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:12:35.721 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:35.719+0000 7f068d7fa700 1 -- 172.21.15.160:0/1708409991 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f06840430a0 con 0x7f06900f5410 2023-12-25T20:12:35.721 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:35.719+0000 7f0696b09700 1 -- 172.21.15.160:0/1708409991 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0690045ad0 con 0x7f06900f5410 2023-12-25T20:12:35.721 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:35.719+0000 7f068ffff700 1 --2- 172.21.15.160:0/1708409991 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f068002ffb0 0x7f0680032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:12:35.721 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:35.719+0000 7f068ffff700 1 --2- 172.21.15.160:0/1708409991 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f068002ffb0 0x7f0680032470 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f067c0096a0 tx=0x7f067c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:12:35.723 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:35.724+0000 7f068d7fa700 1 -- 172.21.15.160:0/1708409991 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0684012920 con 0x7f06900f5410 2023-12-25T20:12:35.930 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:35.930+0000 7f0696b09700 1 -- 172.21.15.160:0/1708409991 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f06900faf50 con 0x7f06900f5410 2023-12-25T20:12:35.930 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:35.930+0000 7f068d7fa700 1 -- 172.21.15.160:0/1708409991 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f068402d030 con 0x7f06900f5410 2023-12-25T20:12:35.930 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:12:35.930 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:12:35.931 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:35.932+0000 7f0696b09700 1 -- 172.21.15.160:0/1708409991 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f068002ffb0 msgr2=0x7f0680032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:12:35.932 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:35.932+0000 7f0696b09700 1 --2- 172.21.15.160:0/1708409991 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f068002ffb0 0x7f0680032470 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f067c0096a0 tx=0x7f067c006b40).stop 2023-12-25T20:12:35.932 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:35.932+0000 7f0696b09700 1 -- 172.21.15.160:0/1708409991 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f06900f5410 msgr2=0x7f0690105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:12:35.932 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:35.932+0000 7f0696b09700 1 --2- 172.21.15.160:0/1708409991 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f06900f5410 0x7f0690105630 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7f0684013510 tx=0x7f068400b040).stop 2023-12-25T20:12:35.932 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:35.932+0000 7f0696b09700 1 -- 172.21.15.160:0/1708409991 shutdown_connections 2023-12-25T20:12:35.932 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:35.932+0000 7f0696b09700 1 --2- 172.21.15.160:0/1708409991 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f068002ffb0 0x7f0680032470 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:12:35.932 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:35.932+0000 7f0696b09700 1 --2- 172.21.15.160:0/1708409991 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f06900f5410 0x7f0690105630 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:12:35.932 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:35.932+0000 7f0696b09700 1 -- 172.21.15.160:0/1708409991 >> 172.21.15.160:0/1708409991 conn(0x7f06900f0b70 msgr2=0x7f06900f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:12:35.932 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:35.932+0000 7f0696b09700 1 -- 172.21.15.160:0/1708409991 shutdown_connections 2023-12-25T20:12:35.932 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:35.932+0000 7f0696b09700 1 -- 172.21.15.160:0/1708409991 wait complete. 2023-12-25T20:12:35.933 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:12:36.718 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:36 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:12:34.674332+0000 mgr.smithi050.nbzhun (mgr.14162) 165 : cluster [DBG] pgmap v154: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:12:36.718 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:36 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:12:35.930961+0000 mon.smithi050 (mon.0) 237 : audit [DBG] from='client.? 172.21.15.160:0/1708409991' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:12:37.351 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:12:37.351 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:12:38.551 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:38.549+0000 7fc93d6d1700 1 -- 172.21.15.160:0/277565199 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc9380f47c0 msgr2=0x7fc9380f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:12:38.552 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:38.549+0000 7fc93d6d1700 1 --2- 172.21.15.160:0/277565199 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc9380f47c0 0x7fc9380f4be0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0x7fc92c009fb0 tx=0x7fc92c009b30).stop 2023-12-25T20:12:38.552 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:38.549+0000 7fc93d6d1700 1 -- 172.21.15.160:0/277565199 shutdown_connections 2023-12-25T20:12:38.552 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:38.549+0000 7fc93d6d1700 1 --2- 172.21.15.160:0/277565199 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc9380f47c0 0x7fc9380f4be0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:12:38.552 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:38.549+0000 7fc93d6d1700 1 -- 172.21.15.160:0/277565199 >> 172.21.15.160:0/277565199 conn(0x7fc9380f02e0 msgr2=0x7fc9380f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:12:38.552 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:38.549+0000 7fc93d6d1700 1 -- 172.21.15.160:0/277565199 shutdown_connections 2023-12-25T20:12:38.552 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:38.549+0000 7fc93d6d1700 1 -- 172.21.15.160:0/277565199 wait complete. 2023-12-25T20:12:38.552 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:38.550+0000 7fc93d6d1700 1 Processor -- start 2023-12-25T20:12:38.552 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:38.550+0000 7fc93d6d1700 1 -- start start 2023-12-25T20:12:38.552 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:38.550+0000 7fc93d6d1700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc9380f47c0 0x7fc938106700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:12:38.552 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:38.550+0000 7fc93d6d1700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc938106bf0 con 0x7fc9380f47c0 2023-12-25T20:12:38.552 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:38.550+0000 7fc936ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc9380f47c0 0x7fc938106700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:12:38.552 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:38.550+0000 7fc936ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc9380f47c0 0x7fc938106700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:40526/0 (socket says 172.21.15.160:40526) 2023-12-25T20:12:38.552 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:38.550+0000 7fc936ffd700 1 -- 172.21.15.160:0/1809248329 learned_addr learned my addr 172.21.15.160:0/1809248329 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:12:38.553 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:38.550+0000 7fc936ffd700 1 -- 172.21.15.160:0/1809248329 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc92c005040 con 0x7fc9380f47c0 2023-12-25T20:12:38.553 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:38.550+0000 7fc936ffd700 1 --2- 172.21.15.160:0/1809248329 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc9380f47c0 0x7fc938106700 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7fc92c013510 tx=0x7fc92c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:12:38.553 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:38.550+0000 7fc91f7fe700 1 -- 172.21.15.160:0/1809248329 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc92c009320 con 0x7fc9380f47c0 2023-12-25T20:12:38.553 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:38.551+0000 7fc91f7fe700 1 -- 172.21.15.160:0/1809248329 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc92c013f00 con 0x7fc9380f47c0 2023-12-25T20:12:38.553 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:38.551+0000 7fc93d6d1700 1 -- 172.21.15.160:0/1809248329 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc938106df0 con 0x7fc9380f47c0 2023-12-25T20:12:38.553 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:38.551+0000 7fc91f7fe700 1 -- 172.21.15.160:0/1809248329 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc92c014d90 con 0x7fc9380f47c0 2023-12-25T20:12:38.553 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:38.551+0000 7fc93d6d1700 1 -- 172.21.15.160:0/1809248329 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc938107280 con 0x7fc9380f47c0 2023-12-25T20:12:38.554 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:38.551+0000 7fc91f7fe700 1 -- 172.21.15.160:0/1809248329 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc92c014420 con 0x7fc9380f47c0 2023-12-25T20:12:38.554 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:38.551+0000 7fc91f7fe700 1 --2- 172.21.15.160:0/1809248329 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fc92002ff60 0x7fc920032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:12:38.554 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:38.551+0000 7fc91f7fe700 1 -- 172.21.15.160:0/1809248329 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc92c0410c0 con 0x7fc9380f47c0 2023-12-25T20:12:38.554 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:38.551+0000 7fc93d6d1700 1 -- 172.21.15.160:0/1809248329 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc938045ad0 con 0x7fc9380f47c0 2023-12-25T20:12:38.554 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:38.552+0000 7fc9367fc700 1 --2- 172.21.15.160:0/1809248329 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fc92002ff60 0x7fc920032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:12:38.554 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:38.552+0000 7fc9367fc700 1 --2- 172.21.15.160:0/1809248329 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fc92002ff60 0x7fc920032420 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7fc9280096a0 tx=0x7fc928006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:12:38.557 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:38.556+0000 7fc91f7fe700 1 -- 172.21.15.160:0/1809248329 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc92c0095f0 con 0x7fc9380f47c0 2023-12-25T20:12:38.718 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:38 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:12:36.674540+0000 mgr.smithi050.nbzhun (mgr.14162) 166 : cluster [DBG] pgmap v155: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:12:38.763 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:38.763+0000 7fc93d6d1700 1 -- 172.21.15.160:0/1809248329 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc938108040 con 0x7fc9380f47c0 2023-12-25T20:12:38.764 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:38.763+0000 7fc91f7fe700 1 -- 172.21.15.160:0/1809248329 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc92c01db70 con 0x7fc9380f47c0 2023-12-25T20:12:38.764 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:12:38.764 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:12:38.766 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:38.765+0000 7fc93d6d1700 1 -- 172.21.15.160:0/1809248329 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fc92002ff60 msgr2=0x7fc920032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:12:38.766 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:38.765+0000 7fc93d6d1700 1 --2- 172.21.15.160:0/1809248329 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fc92002ff60 0x7fc920032420 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7fc9280096a0 tx=0x7fc928006b40).stop 2023-12-25T20:12:38.766 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:38.765+0000 7fc93d6d1700 1 -- 172.21.15.160:0/1809248329 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc9380f47c0 msgr2=0x7fc938106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:12:38.766 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:38.765+0000 7fc93d6d1700 1 --2- 172.21.15.160:0/1809248329 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc9380f47c0 0x7fc938106700 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7fc92c013510 tx=0x7fc92c00b040).stop 2023-12-25T20:12:38.766 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:38.765+0000 7fc93d6d1700 1 -- 172.21.15.160:0/1809248329 shutdown_connections 2023-12-25T20:12:38.766 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:38.765+0000 7fc93d6d1700 1 --2- 172.21.15.160:0/1809248329 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fc92002ff60 0x7fc920032420 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:12:38.766 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:38.765+0000 7fc93d6d1700 1 --2- 172.21.15.160:0/1809248329 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc9380f47c0 0x7fc938106700 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:12:38.766 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:38.765+0000 7fc93d6d1700 1 -- 172.21.15.160:0/1809248329 >> 172.21.15.160:0/1809248329 conn(0x7fc9380f02e0 msgr2=0x7fc9380f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:12:38.766 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:38.765+0000 7fc93d6d1700 1 -- 172.21.15.160:0/1809248329 shutdown_connections 2023-12-25T20:12:38.766 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:38.765+0000 7fc93d6d1700 1 -- 172.21.15.160:0/1809248329 wait complete. 2023-12-25T20:12:38.767 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:12:39.718 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:39 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:12:38.765094+0000 mon.smithi050 (mon.0) 238 : audit [DBG] from='client.? 172.21.15.160:0/1809248329' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:12:40.175 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:12:40.175 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:12:40.718 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:40 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:12:38.674763+0000 mgr.smithi050.nbzhun (mgr.14162) 167 : cluster [DBG] pgmap v156: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:12:41.396 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:41.393+0000 7fcfadc94700 1 -- 172.21.15.160:0/2799666340 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcfa80f6510 msgr2=0x7fcfa80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:12:41.397 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:41.393+0000 7fcfadc94700 1 --2- 172.21.15.160:0/2799666340 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcfa80f6510 0x7fcfa80f6930 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 rx=0x7fcf98009fb0 tx=0x7fcf98009b30).stop 2023-12-25T20:12:41.397 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:41.393+0000 7fcfadc94700 1 -- 172.21.15.160:0/2799666340 shutdown_connections 2023-12-25T20:12:41.397 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:41.393+0000 7fcfadc94700 1 --2- 172.21.15.160:0/2799666340 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcfa80f6510 0x7fcfa80f6930 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:12:41.397 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:41.393+0000 7fcfadc94700 1 -- 172.21.15.160:0/2799666340 >> 172.21.15.160:0/2799666340 conn(0x7fcfa80f1c70 msgr2=0x7fcfa80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:12:41.397 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:41.393+0000 7fcfadc94700 1 -- 172.21.15.160:0/2799666340 shutdown_connections 2023-12-25T20:12:41.397 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:41.393+0000 7fcfadc94700 1 -- 172.21.15.160:0/2799666340 wait complete. 2023-12-25T20:12:41.397 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:41.393+0000 7fcfadc94700 1 Processor -- start 2023-12-25T20:12:41.397 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:41.393+0000 7fcfadc94700 1 -- start start 2023-12-25T20:12:41.397 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:41.394+0000 7fcfadc94700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcfa80f6510 0x7fcfa8106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:12:41.397 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:41.394+0000 7fcfadc94700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa8106c20 con 0x7fcfa80f6510 2023-12-25T20:12:41.397 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:41.394+0000 7fcfa77fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcfa80f6510 0x7fcfa8106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:12:41.397 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:41.394+0000 7fcfa77fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcfa80f6510 0x7fcfa8106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:40552/0 (socket says 172.21.15.160:40552) 2023-12-25T20:12:41.397 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:41.394+0000 7fcfa77fe700 1 -- 172.21.15.160:0/731556429 learned_addr learned my addr 172.21.15.160:0/731556429 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:12:41.398 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:41.394+0000 7fcfa77fe700 1 -- 172.21.15.160:0/731556429 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf98005040 con 0x7fcfa80f6510 2023-12-25T20:12:41.398 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:41.394+0000 7fcfa77fe700 1 --2- 172.21.15.160:0/731556429 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcfa80f6510 0x7fcfa8106730 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7fcf98013510 tx=0x7fcf9800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:12:41.398 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:41.394+0000 7fcf8ffff700 1 -- 172.21.15.160:0/731556429 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcf98009320 con 0x7fcfa80f6510 2023-12-25T20:12:41.398 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:41.394+0000 7fcfadc94700 1 -- 172.21.15.160:0/731556429 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfa8106e20 con 0x7fcfa80f6510 2023-12-25T20:12:41.398 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:41.394+0000 7fcf8ffff700 1 -- 172.21.15.160:0/731556429 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcf98013f00 con 0x7fcfa80f6510 2023-12-25T20:12:41.398 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:41.394+0000 7fcf8ffff700 1 -- 172.21.15.160:0/731556429 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcf98014d90 con 0x7fcfa80f6510 2023-12-25T20:12:41.398 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:41.394+0000 7fcfadc94700 1 -- 172.21.15.160:0/731556429 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfa81072b0 con 0x7fcfa80f6510 2023-12-25T20:12:41.398 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:41.395+0000 7fcf8ffff700 1 -- 172.21.15.160:0/731556429 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcf98014420 con 0x7fcfa80f6510 2023-12-25T20:12:41.398 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:41.395+0000 7fcfadc94700 1 -- 172.21.15.160:0/731556429 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfa8045ad0 con 0x7fcfa80f6510 2023-12-25T20:12:41.398 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:41.395+0000 7fcf8ffff700 1 --2- 172.21.15.160:0/731556429 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fcf9002ff60 0x7fcf90032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:12:41.398 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:41.395+0000 7fcf8ffff700 1 -- 172.21.15.160:0/731556429 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcf98041080 con 0x7fcfa80f6510 2023-12-25T20:12:41.398 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:41.395+0000 7fcfa6ffd700 1 --2- 172.21.15.160:0/731556429 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fcf9002ff60 0x7fcf90032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:12:41.398 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:41.396+0000 7fcfa6ffd700 1 --2- 172.21.15.160:0/731556429 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fcf9002ff60 0x7fcf90032420 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7fcf9c0096a0 tx=0x7fcf9c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:12:41.401 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:41.400+0000 7fcf8ffff700 1 -- 172.21.15.160:0/731556429 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcf9801dd50 con 0x7fcfa80f6510 2023-12-25T20:12:41.606 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:41.605+0000 7fcfadc94700 1 -- 172.21.15.160:0/731556429 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcfa80fbf60 con 0x7fcfa80f6510 2023-12-25T20:12:41.607 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:41.606+0000 7fcf8ffff700 1 -- 172.21.15.160:0/731556429 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fcf9801db70 con 0x7fcfa80f6510 2023-12-25T20:12:41.607 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:12:41.607 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:12:41.609 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:41.607+0000 7fcfadc94700 1 -- 172.21.15.160:0/731556429 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fcf9002ff60 msgr2=0x7fcf90032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:12:41.609 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:41.607+0000 7fcfadc94700 1 --2- 172.21.15.160:0/731556429 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fcf9002ff60 0x7fcf90032420 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7fcf9c0096a0 tx=0x7fcf9c006b40).stop 2023-12-25T20:12:41.609 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:41.607+0000 7fcfadc94700 1 -- 172.21.15.160:0/731556429 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcfa80f6510 msgr2=0x7fcfa8106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:12:41.609 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:41.607+0000 7fcfadc94700 1 --2- 172.21.15.160:0/731556429 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcfa80f6510 0x7fcfa8106730 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7fcf98013510 tx=0x7fcf9800b040).stop 2023-12-25T20:12:41.609 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:41.608+0000 7fcfadc94700 1 -- 172.21.15.160:0/731556429 shutdown_connections 2023-12-25T20:12:41.610 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:41.608+0000 7fcfadc94700 1 --2- 172.21.15.160:0/731556429 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fcf9002ff60 0x7fcf90032420 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:12:41.610 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:41.608+0000 7fcfadc94700 1 --2- 172.21.15.160:0/731556429 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcfa80f6510 0x7fcfa8106730 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:12:41.610 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:41.608+0000 7fcfadc94700 1 -- 172.21.15.160:0/731556429 >> 172.21.15.160:0/731556429 conn(0x7fcfa80f1c70 msgr2=0x7fcfa80f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:12:41.610 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:41.608+0000 7fcfadc94700 1 -- 172.21.15.160:0/731556429 shutdown_connections 2023-12-25T20:12:41.610 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:41.608+0000 7fcfadc94700 1 -- 172.21.15.160:0/731556429 wait complete. 2023-12-25T20:12:41.610 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:12:42.718 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:42 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:12:40.674984+0000 mgr.smithi050.nbzhun (mgr.14162) 168 : cluster [DBG] pgmap v157: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:12:42.719 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:42 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:12:41.607625+0000 mon.smithi050 (mon.0) 239 : audit [DBG] from='client.? 172.21.15.160:0/731556429' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:12:42.993 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:12:42.994 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:12:44.225 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:44.222+0000 7fe2fe944700 1 -- 172.21.15.160:0/2439459361 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe2f80f5410 msgr2=0x7fe2f80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:12:44.225 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:44.222+0000 7fe2fe944700 1 --2- 172.21.15.160:0/2439459361 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe2f80f5410 0x7fe2f80f5830 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0x7fe2e8009fb0 tx=0x7fe2e8009b30).stop 2023-12-25T20:12:44.225 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:44.222+0000 7fe2fe944700 1 -- 172.21.15.160:0/2439459361 shutdown_connections 2023-12-25T20:12:44.225 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:44.222+0000 7fe2fe944700 1 --2- 172.21.15.160:0/2439459361 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe2f80f5410 0x7fe2f80f5830 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:12:44.225 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:44.222+0000 7fe2fe944700 1 -- 172.21.15.160:0/2439459361 >> 172.21.15.160:0/2439459361 conn(0x7fe2f80f0b70 msgr2=0x7fe2f80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:12:44.225 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:44.222+0000 7fe2fe944700 1 -- 172.21.15.160:0/2439459361 shutdown_connections 2023-12-25T20:12:44.225 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:44.222+0000 7fe2fe944700 1 -- 172.21.15.160:0/2439459361 wait complete. 2023-12-25T20:12:44.225 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:44.223+0000 7fe2fe944700 1 Processor -- start 2023-12-25T20:12:44.225 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:44.223+0000 7fe2fe944700 1 -- start start 2023-12-25T20:12:44.226 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:44.223+0000 7fe2fe944700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe2f80f5410 0x7fe2f8105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:12:44.226 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:44.223+0000 7fe2fe944700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2f8105b20 con 0x7fe2f80f5410 2023-12-25T20:12:44.226 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:44.223+0000 7fe2f7fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe2f80f5410 0x7fe2f8105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:12:44.226 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:44.223+0000 7fe2f7fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe2f80f5410 0x7fe2f8105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:40562/0 (socket says 172.21.15.160:40562) 2023-12-25T20:12:44.226 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:44.223+0000 7fe2f7fff700 1 -- 172.21.15.160:0/297289755 learned_addr learned my addr 172.21.15.160:0/297289755 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:12:44.226 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:44.224+0000 7fe2f7fff700 1 -- 172.21.15.160:0/297289755 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2e8005040 con 0x7fe2f80f5410 2023-12-25T20:12:44.226 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:44.224+0000 7fe2f7fff700 1 --2- 172.21.15.160:0/297289755 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe2f80f5410 0x7fe2f8105630 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7fe2e8013510 tx=0x7fe2e800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:12:44.226 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:44.224+0000 7fe2f4ff9700 1 -- 172.21.15.160:0/297289755 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe2e8009320 con 0x7fe2f80f5410 2023-12-25T20:12:44.226 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:44.224+0000 7fe2f4ff9700 1 -- 172.21.15.160:0/297289755 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe2e8013f00 con 0x7fe2f80f5410 2023-12-25T20:12:44.226 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:44.224+0000 7fe2fe944700 1 -- 172.21.15.160:0/297289755 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2f8105d20 con 0x7fe2f80f5410 2023-12-25T20:12:44.226 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:44.224+0000 7fe2f4ff9700 1 -- 172.21.15.160:0/297289755 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe2e8014d90 con 0x7fe2f80f5410 2023-12-25T20:12:44.226 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:44.224+0000 7fe2fe944700 1 -- 172.21.15.160:0/297289755 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2f81061b0 con 0x7fe2f80f5410 2023-12-25T20:12:44.227 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:44.225+0000 7fe2f4ff9700 1 -- 172.21.15.160:0/297289755 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe2e80263a0 con 0x7fe2f80f5410 2023-12-25T20:12:44.227 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:44.225+0000 7fe2f4ff9700 1 --2- 172.21.15.160:0/297289755 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fe2e002ff60 0x7fe2e0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:12:44.227 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:44.225+0000 7fe2fe944700 1 -- 172.21.15.160:0/297289755 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2f8045ad0 con 0x7fe2f80f5410 2023-12-25T20:12:44.227 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:44.225+0000 7fe2f4ff9700 1 -- 172.21.15.160:0/297289755 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe2e8041370 con 0x7fe2f80f5410 2023-12-25T20:12:44.227 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:44.225+0000 7fe2f77fe700 1 --2- 172.21.15.160:0/297289755 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fe2e002ff60 0x7fe2e0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:12:44.227 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:44.225+0000 7fe2f77fe700 1 --2- 172.21.15.160:0/297289755 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fe2e002ff60 0x7fe2e0032420 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7fe2ec0096a0 tx=0x7fe2ec006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:12:44.230 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:44.230+0000 7fe2f4ff9700 1 -- 172.21.15.160:0/297289755 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe2e801cb10 con 0x7fe2f80f5410 2023-12-25T20:12:44.435 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:44.435+0000 7fe2fe944700 1 -- 172.21.15.160:0/297289755 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe2f80faf50 con 0x7fe2f80f5410 2023-12-25T20:12:44.436 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:44.435+0000 7fe2f4ff9700 1 -- 172.21.15.160:0/297289755 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe2e802b030 con 0x7fe2f80f5410 2023-12-25T20:12:44.436 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:12:44.436 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:12:44.438 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:44.437+0000 7fe2fe944700 1 -- 172.21.15.160:0/297289755 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fe2e002ff60 msgr2=0x7fe2e0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:12:44.438 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:44.437+0000 7fe2fe944700 1 --2- 172.21.15.160:0/297289755 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fe2e002ff60 0x7fe2e0032420 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7fe2ec0096a0 tx=0x7fe2ec006b40).stop 2023-12-25T20:12:44.438 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:44.437+0000 7fe2fe944700 1 -- 172.21.15.160:0/297289755 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe2f80f5410 msgr2=0x7fe2f8105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:12:44.438 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:44.437+0000 7fe2fe944700 1 --2- 172.21.15.160:0/297289755 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe2f80f5410 0x7fe2f8105630 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7fe2e8013510 tx=0x7fe2e800b040).stop 2023-12-25T20:12:44.438 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:44.437+0000 7fe2fe944700 1 -- 172.21.15.160:0/297289755 shutdown_connections 2023-12-25T20:12:44.438 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:44.437+0000 7fe2fe944700 1 --2- 172.21.15.160:0/297289755 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fe2e002ff60 0x7fe2e0032420 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:12:44.438 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:44.437+0000 7fe2fe944700 1 --2- 172.21.15.160:0/297289755 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe2f80f5410 0x7fe2f8105630 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:12:44.438 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:44.437+0000 7fe2fe944700 1 -- 172.21.15.160:0/297289755 >> 172.21.15.160:0/297289755 conn(0x7fe2f80f0b70 msgr2=0x7fe2f80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:12:44.438 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:44.437+0000 7fe2fe944700 1 -- 172.21.15.160:0/297289755 shutdown_connections 2023-12-25T20:12:44.438 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:44.437+0000 7fe2fe944700 1 -- 172.21.15.160:0/297289755 wait complete. 2023-12-25T20:12:44.438 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:12:44.718 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:44 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:12:42.675292+0000 mgr.smithi050.nbzhun (mgr.14162) 169 : cluster [DBG] pgmap v158: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:12:44.719 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:44 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:12:44.437191+0000 mon.smithi050 (mon.0) 240 : audit [DBG] from='client.? 172.21.15.160:0/297289755' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:12:45.832 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:12:45.833 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:12:46.718 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:46 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:12:44.675520+0000 mgr.smithi050.nbzhun (mgr.14162) 170 : cluster [DBG] pgmap v159: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:12:47.049 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:47.046+0000 7f46888c3700 1 -- 172.21.15.160:0/1031302054 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4684060160 msgr2=0x7f46840f8fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:12:47.049 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:47.046+0000 7f46888c3700 1 --2- 172.21.15.160:0/1031302054 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4684060160 0x7f46840f8fa0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 rx=0x7f4678009fb0 tx=0x7f4678009b30).stop 2023-12-25T20:12:47.049 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:47.046+0000 7f46888c3700 1 -- 172.21.15.160:0/1031302054 shutdown_connections 2023-12-25T20:12:47.049 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:47.046+0000 7f46888c3700 1 --2- 172.21.15.160:0/1031302054 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4684060160 0x7f46840f8fa0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:12:47.049 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:47.046+0000 7f46888c3700 1 -- 172.21.15.160:0/1031302054 >> 172.21.15.160:0/1031302054 conn(0x7f46840f09a0 msgr2=0x7f46840f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:12:47.049 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:47.046+0000 7f46888c3700 1 -- 172.21.15.160:0/1031302054 shutdown_connections 2023-12-25T20:12:47.049 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:47.046+0000 7f46888c3700 1 -- 172.21.15.160:0/1031302054 wait complete. 2023-12-25T20:12:47.049 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:47.047+0000 7f46888c3700 1 Processor -- start 2023-12-25T20:12:47.049 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:47.047+0000 7f46888c3700 1 -- start start 2023-12-25T20:12:47.049 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:47.047+0000 7f46888c3700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4684060160 0x7f46840f6a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:12:47.050 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:47.047+0000 7f46888c3700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46840f4ea0 con 0x7f4684060160 2023-12-25T20:12:47.050 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:47.047+0000 7f4682d9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4684060160 0x7f46840f6a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:12:47.050 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:47.047+0000 7f4682d9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4684060160 0x7f46840f6a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:40586/0 (socket says 172.21.15.160:40586) 2023-12-25T20:12:47.050 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:47.047+0000 7f4682d9d700 1 -- 172.21.15.160:0/3118512677 learned_addr learned my addr 172.21.15.160:0/3118512677 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:12:47.050 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:47.048+0000 7f4682d9d700 1 -- 172.21.15.160:0/3118512677 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4678005040 con 0x7f4684060160 2023-12-25T20:12:47.050 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:47.048+0000 7f4682d9d700 1 --2- 172.21.15.160:0/3118512677 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4684060160 0x7f46840f6a20 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7f4678013510 tx=0x7f467800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:12:47.050 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:47.048+0000 7f466b7fe700 1 -- 172.21.15.160:0/3118512677 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4678009320 con 0x7f4684060160 2023-12-25T20:12:47.050 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:47.048+0000 7f46888c3700 1 -- 172.21.15.160:0/3118512677 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46840f5040 con 0x7f4684060160 2023-12-25T20:12:47.050 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:47.048+0000 7f466b7fe700 1 -- 172.21.15.160:0/3118512677 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4678013f00 con 0x7f4684060160 2023-12-25T20:12:47.050 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:47.048+0000 7f466b7fe700 1 -- 172.21.15.160:0/3118512677 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4678014d90 con 0x7f4684060160 2023-12-25T20:12:47.050 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:47.048+0000 7f46888c3700 1 -- 172.21.15.160:0/3118512677 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46840f54d0 con 0x7f4684060160 2023-12-25T20:12:47.051 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:47.049+0000 7f46888c3700 1 -- 172.21.15.160:0/3118512677 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4684059100 con 0x7f4684060160 2023-12-25T20:12:47.051 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:47.049+0000 7f466b7fe700 1 -- 172.21.15.160:0/3118512677 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f46780095f0 con 0x7f4684060160 2023-12-25T20:12:47.051 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:47.049+0000 7f466b7fe700 1 --2- 172.21.15.160:0/3118512677 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f466c02ff60 0x7f466c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:12:47.052 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:47.049+0000 7f466b7fe700 1 -- 172.21.15.160:0/3118512677 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4678041000 con 0x7f4684060160 2023-12-25T20:12:47.052 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:47.049+0000 7f468259c700 1 --2- 172.21.15.160:0/3118512677 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f466c02ff60 0x7f466c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:12:47.052 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:47.050+0000 7f468259c700 1 --2- 172.21.15.160:0/3118512677 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f466c02ff60 0x7f466c032420 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f46740096a0 tx=0x7f4674006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:12:47.055 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:47.054+0000 7f466b7fe700 1 -- 172.21.15.160:0/3118512677 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4678014420 con 0x7f4684060160 2023-12-25T20:12:47.260 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:47.260+0000 7f46888c3700 1 -- 172.21.15.160:0/3118512677 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f46840fbce0 con 0x7f4684060160 2023-12-25T20:12:47.261 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:47.261+0000 7f466b7fe700 1 -- 172.21.15.160:0/3118512677 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4678010230 con 0x7f4684060160 2023-12-25T20:12:47.262 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:12:47.262 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:12:47.266 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:47.262+0000 7f46888c3700 1 -- 172.21.15.160:0/3118512677 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f466c02ff60 msgr2=0x7f466c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:12:47.266 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:47.262+0000 7f46888c3700 1 --2- 172.21.15.160:0/3118512677 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f466c02ff60 0x7f466c032420 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f46740096a0 tx=0x7f4674006b40).stop 2023-12-25T20:12:47.266 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:47.262+0000 7f46888c3700 1 -- 172.21.15.160:0/3118512677 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4684060160 msgr2=0x7f46840f6a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:12:47.266 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:47.262+0000 7f46888c3700 1 --2- 172.21.15.160:0/3118512677 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4684060160 0x7f46840f6a20 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7f4678013510 tx=0x7f467800b040).stop 2023-12-25T20:12:47.266 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:47.262+0000 7f46888c3700 1 -- 172.21.15.160:0/3118512677 shutdown_connections 2023-12-25T20:12:47.266 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:47.262+0000 7f46888c3700 1 --2- 172.21.15.160:0/3118512677 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f466c02ff60 0x7f466c032420 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:12:47.266 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:47.262+0000 7f46888c3700 1 --2- 172.21.15.160:0/3118512677 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4684060160 0x7f46840f6a20 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:12:47.266 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:47.262+0000 7f46888c3700 1 -- 172.21.15.160:0/3118512677 >> 172.21.15.160:0/3118512677 conn(0x7f46840f09a0 msgr2=0x7f46840f14a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:12:47.266 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:47.262+0000 7f46888c3700 1 -- 172.21.15.160:0/3118512677 shutdown_connections 2023-12-25T20:12:47.266 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:47.262+0000 7f46888c3700 1 -- 172.21.15.160:0/3118512677 wait complete. 2023-12-25T20:12:47.266 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:12:47.719 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:47 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:12:47.262279+0000 mon.smithi050 (mon.0) 241 : audit [DBG] from='client.? 172.21.15.160:0/3118512677' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:12:48.549 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:48 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:12:46.675847+0000 mgr.smithi050.nbzhun (mgr.14162) 171 : cluster [DBG] pgmap v160: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:12:48.654 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:12:48.654 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:12:49.859 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:49.857+0000 7f242dc95700 1 -- 172.21.15.160:0/3793636409 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2428060220 msgr2=0x7f24280fb550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:12:49.859 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:49.857+0000 7f242dc95700 1 --2- 172.21.15.160:0/3793636409 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2428060220 0x7f24280fb550 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 rx=0x7f2418009fb0 tx=0x7f2418009b30).stop 2023-12-25T20:12:49.859 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:49.857+0000 7f242dc95700 1 -- 172.21.15.160:0/3793636409 shutdown_connections 2023-12-25T20:12:49.859 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:49.857+0000 7f242dc95700 1 --2- 172.21.15.160:0/3793636409 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2428060220 0x7f24280fb550 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:12:49.859 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:49.857+0000 7f242dc95700 1 -- 172.21.15.160:0/3793636409 >> 172.21.15.160:0/3793636409 conn(0x7f24280f0b70 msgr2=0x7f24280f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:12:49.859 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:49.857+0000 7f242dc95700 1 -- 172.21.15.160:0/3793636409 shutdown_connections 2023-12-25T20:12:49.859 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:49.857+0000 7f242dc95700 1 -- 172.21.15.160:0/3793636409 wait complete. 2023-12-25T20:12:49.859 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:49.858+0000 7f242dc95700 1 Processor -- start 2023-12-25T20:12:49.860 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:49.858+0000 7f242dc95700 1 -- start start 2023-12-25T20:12:49.860 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:49.858+0000 7f242dc95700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2428060220 0x7f24281088c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:12:49.860 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:49.858+0000 7f242dc95700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2428108db0 con 0x7f2428060220 2023-12-25T20:12:49.860 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:49.858+0000 7f24277fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2428060220 0x7f24281088c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:12:49.860 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:49.858+0000 7f24277fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2428060220 0x7f24281088c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:49036/0 (socket says 172.21.15.160:49036) 2023-12-25T20:12:49.860 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:49.858+0000 7f24277fe700 1 -- 172.21.15.160:0/1673263464 learned_addr learned my addr 172.21.15.160:0/1673263464 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:12:49.861 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:49.859+0000 7f24277fe700 1 -- 172.21.15.160:0/1673263464 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2418005040 con 0x7f2428060220 2023-12-25T20:12:49.861 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:49.859+0000 7f24277fe700 1 --2- 172.21.15.160:0/1673263464 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2428060220 0x7f24281088c0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7f2418013510 tx=0x7f241800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:12:49.861 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:49.859+0000 7f240ffff700 1 -- 172.21.15.160:0/1673263464 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2418009320 con 0x7f2428060220 2023-12-25T20:12:49.861 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:49.859+0000 7f240ffff700 1 -- 172.21.15.160:0/1673263464 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2418013f00 con 0x7f2428060220 2023-12-25T20:12:49.862 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:49.859+0000 7f240ffff700 1 -- 172.21.15.160:0/1673263464 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2418014dc0 con 0x7f2428060220 2023-12-25T20:12:49.862 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:49.859+0000 7f242dc95700 1 -- 172.21.15.160:0/1673263464 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2428108fb0 con 0x7f2428060220 2023-12-25T20:12:49.862 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:49.859+0000 7f242dc95700 1 -- 172.21.15.160:0/1673263464 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2428109440 con 0x7f2428060220 2023-12-25T20:12:49.862 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:49.860+0000 7f240ffff700 1 -- 172.21.15.160:0/1673263464 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f24180263a0 con 0x7f2428060220 2023-12-25T20:12:49.862 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:49.860+0000 7f240ffff700 1 --2- 172.21.15.160:0/1673263464 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f241002ffb0 0x7f2410032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:12:49.862 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:49.860+0000 7f240ffff700 1 -- 172.21.15.160:0/1673263464 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2418041ee0 con 0x7f2428060220 2023-12-25T20:12:49.862 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:49.860+0000 7f242dc95700 1 -- 172.21.15.160:0/1673263464 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2428059100 con 0x7f2428060220 2023-12-25T20:12:49.862 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:49.860+0000 7f2426ffd700 1 --2- 172.21.15.160:0/1673263464 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f241002ffb0 0x7f2410032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:12:49.862 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:49.860+0000 7f2426ffd700 1 --2- 172.21.15.160:0/1673263464 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f241002ffb0 0x7f2410032470 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f241c0096a0 tx=0x7f241c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:12:49.865 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:49.865+0000 7f240ffff700 1 -- 172.21.15.160:0/1673263464 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2418012920 con 0x7f2428060220 2023-12-25T20:12:50.069 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:50.069+0000 7f242dc95700 1 -- 172.21.15.160:0/1673263464 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f24280fe150 con 0x7f2428060220 2023-12-25T20:12:50.070 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:50.069+0000 7f240ffff700 1 -- 172.21.15.160:0/1673263464 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f241802b020 con 0x7f2428060220 2023-12-25T20:12:50.070 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:12:50.070 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:12:50.072 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:50.071+0000 7f242dc95700 1 -- 172.21.15.160:0/1673263464 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f241002ffb0 msgr2=0x7f2410032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:12:50.072 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:50.071+0000 7f242dc95700 1 --2- 172.21.15.160:0/1673263464 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f241002ffb0 0x7f2410032470 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f241c0096a0 tx=0x7f241c006b40).stop 2023-12-25T20:12:50.072 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:50.071+0000 7f242dc95700 1 -- 172.21.15.160:0/1673263464 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2428060220 msgr2=0x7f24281088c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:12:50.072 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:50.071+0000 7f242dc95700 1 --2- 172.21.15.160:0/1673263464 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2428060220 0x7f24281088c0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7f2418013510 tx=0x7f241800b040).stop 2023-12-25T20:12:50.072 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:50.071+0000 7f242dc95700 1 -- 172.21.15.160:0/1673263464 shutdown_connections 2023-12-25T20:12:50.072 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:50.071+0000 7f242dc95700 1 --2- 172.21.15.160:0/1673263464 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f241002ffb0 0x7f2410032470 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:12:50.072 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:50.071+0000 7f242dc95700 1 --2- 172.21.15.160:0/1673263464 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2428060220 0x7f24281088c0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:12:50.073 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:50.071+0000 7f242dc95700 1 -- 172.21.15.160:0/1673263464 >> 172.21.15.160:0/1673263464 conn(0x7f24280f0b70 msgr2=0x7f24280f1670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:12:50.073 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:50.071+0000 7f242dc95700 1 -- 172.21.15.160:0/1673263464 shutdown_connections 2023-12-25T20:12:50.073 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:50.071+0000 7f242dc95700 1 -- 172.21.15.160:0/1673263464 wait complete. 2023-12-25T20:12:50.073 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:12:50.469 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:50 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:12:48.676100+0000 mgr.smithi050.nbzhun (mgr.14162) 172 : cluster [DBG] pgmap v161: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:12:50.469 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:50 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:12:49.082469+0000 mon.smithi050 (mon.0) 242 : audit [INF] from='mgr.14162 172.21.15.50:0/3178367813' entity='mgr.smithi050.nbzhun' 2023-12-25T20:12:50.469 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:50 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cephadm 2023-12-25T20:12:49.083878+0000 mgr.smithi050.nbzhun (mgr.14162) 173 : cephadm [INF] Deploying daemon node-exporter.smithi050 on smithi050 2023-12-25T20:12:50.469 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:50 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:12:50.071123+0000 mon.smithi050 (mon.0) 243 : audit [DBG] from='client.? 172.21.15.160:0/1673263464' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:12:51.461 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:12:51.461 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:12:52.468 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:52 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:12:50.676426+0000 mgr.smithi050.nbzhun (mgr.14162) 174 : cluster [DBG] pgmap v162: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:12:52.679 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:52.678+0000 7f3d6d0d6700 1 -- 172.21.15.160:0/2289305522 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3d680f47c0 msgr2=0x7f3d680f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:12:52.680 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:52.678+0000 7f3d6d0d6700 1 --2- 172.21.15.160:0/2289305522 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3d680f47c0 0x7f3d680f4be0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 rx=0x7f3d5c009fb0 tx=0x7f3d5c009b30).stop 2023-12-25T20:12:52.680 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:52.678+0000 7f3d6d0d6700 1 -- 172.21.15.160:0/2289305522 shutdown_connections 2023-12-25T20:12:52.680 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:52.678+0000 7f3d6d0d6700 1 --2- 172.21.15.160:0/2289305522 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3d680f47c0 0x7f3d680f4be0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:12:52.680 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:52.678+0000 7f3d6d0d6700 1 -- 172.21.15.160:0/2289305522 >> 172.21.15.160:0/2289305522 conn(0x7f3d680f02e0 msgr2=0x7f3d680f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:12:52.680 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:52.678+0000 7f3d6d0d6700 1 -- 172.21.15.160:0/2289305522 shutdown_connections 2023-12-25T20:12:52.680 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:52.678+0000 7f3d6d0d6700 1 -- 172.21.15.160:0/2289305522 wait complete. 2023-12-25T20:12:52.680 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:52.679+0000 7f3d6d0d6700 1 Processor -- start 2023-12-25T20:12:52.680 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:52.679+0000 7f3d6d0d6700 1 -- start start 2023-12-25T20:12:52.680 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:52.679+0000 7f3d6d0d6700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3d680f47c0 0x7f3d68106700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:12:52.680 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:52.679+0000 7f3d6d0d6700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d68106bf0 con 0x7f3d680f47c0 2023-12-25T20:12:52.680 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:52.679+0000 7f3d66d9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3d680f47c0 0x7f3d68106700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:12:52.680 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:52.679+0000 7f3d66d9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3d680f47c0 0x7f3d68106700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:49052/0 (socket says 172.21.15.160:49052) 2023-12-25T20:12:52.680 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:52.679+0000 7f3d66d9d700 1 -- 172.21.15.160:0/3624790000 learned_addr learned my addr 172.21.15.160:0/3624790000 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:12:52.682 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:52.679+0000 7f3d66d9d700 1 -- 172.21.15.160:0/3624790000 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d5c005040 con 0x7f3d680f47c0 2023-12-25T20:12:52.682 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:52.680+0000 7f3d66d9d700 1 --2- 172.21.15.160:0/3624790000 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3d680f47c0 0x7f3d68106700 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7f3d5c013510 tx=0x7f3d5c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:12:52.682 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:52.680+0000 7f3d4f7fe700 1 -- 172.21.15.160:0/3624790000 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3d5c009320 con 0x7f3d680f47c0 2023-12-25T20:12:52.682 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:52.680+0000 7f3d6d0d6700 1 -- 172.21.15.160:0/3624790000 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d68106df0 con 0x7f3d680f47c0 2023-12-25T20:12:52.682 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:52.680+0000 7f3d4f7fe700 1 -- 172.21.15.160:0/3624790000 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3d5c013f00 con 0x7f3d680f47c0 2023-12-25T20:12:52.682 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:52.680+0000 7f3d4f7fe700 1 -- 172.21.15.160:0/3624790000 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3d5c014d90 con 0x7f3d680f47c0 2023-12-25T20:12:52.682 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:52.680+0000 7f3d6d0d6700 1 -- 172.21.15.160:0/3624790000 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d68107280 con 0x7f3d680f47c0 2023-12-25T20:12:52.682 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:52.681+0000 7f3d4f7fe700 1 -- 172.21.15.160:0/3624790000 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3d5c0095f0 con 0x7f3d680f47c0 2023-12-25T20:12:52.682 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:52.681+0000 7f3d6d0d6700 1 -- 172.21.15.160:0/3624790000 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d68045ad0 con 0x7f3d680f47c0 2023-12-25T20:12:52.683 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:52.681+0000 7f3d4f7fe700 1 --2- 172.21.15.160:0/3624790000 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f3d5002ff60 0x7f3d50032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:12:52.683 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:52.681+0000 7f3d4f7fe700 1 -- 172.21.15.160:0/3624790000 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3d5c040fe0 con 0x7f3d680f47c0 2023-12-25T20:12:52.683 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:52.681+0000 7f3d6659c700 1 --2- 172.21.15.160:0/3624790000 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f3d5002ff60 0x7f3d50032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:12:52.683 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:52.681+0000 7f3d6659c700 1 --2- 172.21.15.160:0/3624790000 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f3d5002ff60 0x7f3d50032420 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f3d580096a0 tx=0x7f3d58006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:12:52.686 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:52.685+0000 7f3d4f7fe700 1 -- 172.21.15.160:0/3624790000 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3d5c014420 con 0x7f3d680f47c0 2023-12-25T20:12:52.891 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:52.891+0000 7f3d6d0d6700 1 -- 172.21.15.160:0/3624790000 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3d681080d0 con 0x7f3d680f47c0 2023-12-25T20:12:52.892 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:52.892+0000 7f3d4f7fe700 1 -- 172.21.15.160:0/3624790000 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3d5c010230 con 0x7f3d680f47c0 2023-12-25T20:12:52.892 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:12:52.892 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:12:52.894 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:52.893+0000 7f3d6d0d6700 1 -- 172.21.15.160:0/3624790000 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f3d5002ff60 msgr2=0x7f3d50032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:12:52.894 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:52.893+0000 7f3d6d0d6700 1 --2- 172.21.15.160:0/3624790000 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f3d5002ff60 0x7f3d50032420 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f3d580096a0 tx=0x7f3d58006b40).stop 2023-12-25T20:12:52.894 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:52.893+0000 7f3d6d0d6700 1 -- 172.21.15.160:0/3624790000 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3d680f47c0 msgr2=0x7f3d68106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:12:52.894 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:52.893+0000 7f3d6d0d6700 1 --2- 172.21.15.160:0/3624790000 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3d680f47c0 0x7f3d68106700 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7f3d5c013510 tx=0x7f3d5c00b040).stop 2023-12-25T20:12:52.894 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:52.893+0000 7f3d6d0d6700 1 -- 172.21.15.160:0/3624790000 shutdown_connections 2023-12-25T20:12:52.895 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:52.893+0000 7f3d6d0d6700 1 --2- 172.21.15.160:0/3624790000 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f3d5002ff60 0x7f3d50032420 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:12:52.895 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:52.893+0000 7f3d6d0d6700 1 --2- 172.21.15.160:0/3624790000 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3d680f47c0 0x7f3d68106700 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:12:52.895 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:52.893+0000 7f3d6d0d6700 1 -- 172.21.15.160:0/3624790000 >> 172.21.15.160:0/3624790000 conn(0x7f3d680f02e0 msgr2=0x7f3d680f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:12:52.895 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:52.893+0000 7f3d6d0d6700 1 -- 172.21.15.160:0/3624790000 shutdown_connections 2023-12-25T20:12:52.895 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:52.893+0000 7f3d6d0d6700 1 -- 172.21.15.160:0/3624790000 wait complete. 2023-12-25T20:12:52.895 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:12:53.468 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:53 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:12:52.893249+0000 mon.smithi050 (mon.0) 244 : audit [DBG] from='client.? 172.21.15.160:0/3624790000' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:12:54.312 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:12:54.312 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:12:54.468 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:54 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:12:52.676781+0000 mgr.smithi050.nbzhun (mgr.14162) 175 : cluster [DBG] pgmap v163: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:12:55.528 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:55.527+0000 7fd646475700 1 -- 172.21.15.160:0/1929244790 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd6400f5410 msgr2=0x7fd6400f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:12:55.528 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:55.527+0000 7fd646475700 1 --2- 172.21.15.160:0/1929244790 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd6400f5410 0x7fd6400f5830 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 rx=0x7fd628009fb0 tx=0x7fd628009b30).stop 2023-12-25T20:12:55.528 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:55.527+0000 7fd646475700 1 -- 172.21.15.160:0/1929244790 shutdown_connections 2023-12-25T20:12:55.528 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:55.527+0000 7fd646475700 1 --2- 172.21.15.160:0/1929244790 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd6400f5410 0x7fd6400f5830 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:12:55.528 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:55.527+0000 7fd646475700 1 -- 172.21.15.160:0/1929244790 >> 172.21.15.160:0/1929244790 conn(0x7fd6400f0b70 msgr2=0x7fd6400f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:12:55.528 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:55.527+0000 7fd646475700 1 -- 172.21.15.160:0/1929244790 shutdown_connections 2023-12-25T20:12:55.528 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:55.527+0000 7fd646475700 1 -- 172.21.15.160:0/1929244790 wait complete. 2023-12-25T20:12:55.529 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:55.527+0000 7fd646475700 1 Processor -- start 2023-12-25T20:12:55.530 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:55.528+0000 7fd646475700 1 -- start start 2023-12-25T20:12:55.530 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:55.528+0000 7fd646475700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd6400f5410 0x7fd640109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:12:55.530 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:55.528+0000 7fd646475700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd640060240 con 0x7fd6400f5410 2023-12-25T20:12:55.530 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:55.528+0000 7fd63ffff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd6400f5410 0x7fd640109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:12:55.530 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:55.528+0000 7fd63ffff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd6400f5410 0x7fd640109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:49072/0 (socket says 172.21.15.160:49072) 2023-12-25T20:12:55.530 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:55.528+0000 7fd63ffff700 1 -- 172.21.15.160:0/569545776 learned_addr learned my addr 172.21.15.160:0/569545776 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:12:55.530 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:55.528+0000 7fd63ffff700 1 -- 172.21.15.160:0/569545776 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd628005040 con 0x7fd6400f5410 2023-12-25T20:12:55.530 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:55.529+0000 7fd63ffff700 1 --2- 172.21.15.160:0/569545776 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd6400f5410 0x7fd640109430 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7fd628013510 tx=0x7fd62800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:12:55.530 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:55.529+0000 7fd63cff9700 1 -- 172.21.15.160:0/569545776 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd628009320 con 0x7fd6400f5410 2023-12-25T20:12:55.530 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:55.529+0000 7fd646475700 1 -- 172.21.15.160:0/569545776 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd640060440 con 0x7fd6400f5410 2023-12-25T20:12:55.530 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:55.529+0000 7fd63cff9700 1 -- 172.21.15.160:0/569545776 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd628013f00 con 0x7fd6400f5410 2023-12-25T20:12:55.530 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:55.529+0000 7fd63cff9700 1 -- 172.21.15.160:0/569545776 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd628014d90 con 0x7fd6400f5410 2023-12-25T20:12:55.530 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:55.529+0000 7fd646475700 1 -- 172.21.15.160:0/569545776 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd640109ae0 con 0x7fd6400f5410 2023-12-25T20:12:55.531 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:55.530+0000 7fd646475700 1 -- 172.21.15.160:0/569545776 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd640045ad0 con 0x7fd6400f5410 2023-12-25T20:12:55.531 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:55.530+0000 7fd63cff9700 1 -- 172.21.15.160:0/569545776 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd6280095f0 con 0x7fd6400f5410 2023-12-25T20:12:55.531 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:55.530+0000 7fd63cff9700 1 --2- 172.21.15.160:0/569545776 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fd62c02ff60 0x7fd62c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:12:55.531 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:55.530+0000 7fd63cff9700 1 -- 172.21.15.160:0/569545776 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd628041090 con 0x7fd6400f5410 2023-12-25T20:12:55.531 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:55.530+0000 7fd63f7fe700 1 --2- 172.21.15.160:0/569545776 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fd62c02ff60 0x7fd62c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:12:55.532 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:55.530+0000 7fd63f7fe700 1 --2- 172.21.15.160:0/569545776 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fd62c02ff60 0x7fd62c032420 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7fd6300096a0 tx=0x7fd630006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:12:55.535 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:55.534+0000 7fd63cff9700 1 -- 172.21.15.160:0/569545776 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd628014420 con 0x7fd6400f5410 2023-12-25T20:12:55.740 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:55.739+0000 7fd646475700 1 -- 172.21.15.160:0/569545776 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd6400fae60 con 0x7fd6400f5410 2023-12-25T20:12:55.740 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:55.739+0000 7fd63cff9700 1 -- 172.21.15.160:0/569545776 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd62801d7f0 con 0x7fd6400f5410 2023-12-25T20:12:55.740 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:12:55.741 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:12:55.742 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:55.741+0000 7fd646475700 1 -- 172.21.15.160:0/569545776 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fd62c02ff60 msgr2=0x7fd62c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:12:55.742 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:55.741+0000 7fd646475700 1 --2- 172.21.15.160:0/569545776 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fd62c02ff60 0x7fd62c032420 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7fd6300096a0 tx=0x7fd630006b40).stop 2023-12-25T20:12:55.742 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:55.741+0000 7fd646475700 1 -- 172.21.15.160:0/569545776 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd6400f5410 msgr2=0x7fd640109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:12:55.742 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:55.741+0000 7fd646475700 1 --2- 172.21.15.160:0/569545776 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd6400f5410 0x7fd640109430 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7fd628013510 tx=0x7fd62800b040).stop 2023-12-25T20:12:55.742 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:55.741+0000 7fd646475700 1 -- 172.21.15.160:0/569545776 shutdown_connections 2023-12-25T20:12:55.742 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:55.741+0000 7fd646475700 1 --2- 172.21.15.160:0/569545776 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fd62c02ff60 0x7fd62c032420 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:12:55.743 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:55.741+0000 7fd646475700 1 --2- 172.21.15.160:0/569545776 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd6400f5410 0x7fd640109430 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:12:55.743 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:55.741+0000 7fd646475700 1 -- 172.21.15.160:0/569545776 >> 172.21.15.160:0/569545776 conn(0x7fd6400f0b70 msgr2=0x7fd6400f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:12:55.743 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:55.741+0000 7fd646475700 1 -- 172.21.15.160:0/569545776 shutdown_connections 2023-12-25T20:12:55.743 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:55.741+0000 7fd646475700 1 -- 172.21.15.160:0/569545776 wait complete. 2023-12-25T20:12:55.743 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:12:56.468 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:56 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:12:54.677098+0000 mgr.smithi050.nbzhun (mgr.14162) 176 : cluster [DBG] pgmap v164: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:12:56.469 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:56 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:12:55.741243+0000 mon.smithi050 (mon.0) 245 : audit [DBG] from='client.? 172.21.15.160:0/569545776' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:12:57.113 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:12:57.113 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:12:58.356 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:58.354+0000 7f3a272ce700 1 -- 172.21.15.160:0/2758266942 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3a200f5410 msgr2=0x7f3a200f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:12:58.356 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:58.354+0000 7f3a272ce700 1 --2- 172.21.15.160:0/2758266942 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3a200f5410 0x7f3a200f5830 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 rx=0x7f3a10009fb0 tx=0x7f3a10009b30).stop 2023-12-25T20:12:58.356 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:58.354+0000 7f3a272ce700 1 -- 172.21.15.160:0/2758266942 shutdown_connections 2023-12-25T20:12:58.356 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:58.354+0000 7f3a272ce700 1 --2- 172.21.15.160:0/2758266942 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3a200f5410 0x7f3a200f5830 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:12:58.356 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:58.354+0000 7f3a272ce700 1 -- 172.21.15.160:0/2758266942 >> 172.21.15.160:0/2758266942 conn(0x7f3a200f0b70 msgr2=0x7f3a200f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:12:58.356 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:58.354+0000 7f3a272ce700 1 -- 172.21.15.160:0/2758266942 shutdown_connections 2023-12-25T20:12:58.356 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:58.354+0000 7f3a272ce700 1 -- 172.21.15.160:0/2758266942 wait complete. 2023-12-25T20:12:58.356 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:58.355+0000 7f3a272ce700 1 Processor -- start 2023-12-25T20:12:58.356 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:58.355+0000 7f3a272ce700 1 -- start start 2023-12-25T20:12:58.356 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:58.355+0000 7f3a272ce700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3a200f5410 0x7f3a20106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:12:58.357 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:58.355+0000 7f3a272ce700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a20106c10 con 0x7f3a200f5410 2023-12-25T20:12:58.357 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:58.355+0000 7f3a2506a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3a200f5410 0x7f3a20106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:12:58.357 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:58.355+0000 7f3a2506a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3a200f5410 0x7f3a20106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:37200/0 (socket says 172.21.15.160:37200) 2023-12-25T20:12:58.357 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:58.355+0000 7f3a2506a700 1 -- 172.21.15.160:0/2406431037 learned_addr learned my addr 172.21.15.160:0/2406431037 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:12:58.357 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:58.355+0000 7f3a2506a700 1 -- 172.21.15.160:0/2406431037 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a10005040 con 0x7f3a200f5410 2023-12-25T20:12:58.358 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:58.356+0000 7f3a2506a700 1 --2- 172.21.15.160:0/2406431037 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3a200f5410 0x7f3a20106720 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7f3a10014000 tx=0x7f3a1000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:12:58.358 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:58.356+0000 7f3a15ffb700 1 -- 172.21.15.160:0/2406431037 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3a100095f0 con 0x7f3a200f5410 2023-12-25T20:12:58.358 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:58.356+0000 7f3a272ce700 1 -- 172.21.15.160:0/2406431037 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a20106e10 con 0x7f3a200f5410 2023-12-25T20:12:58.358 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:58.356+0000 7f3a15ffb700 1 -- 172.21.15.160:0/2406431037 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3a10007e70 con 0x7f3a200f5410 2023-12-25T20:12:58.358 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:58.356+0000 7f3a15ffb700 1 -- 172.21.15.160:0/2406431037 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3a10014ea0 con 0x7f3a200f5410 2023-12-25T20:12:58.358 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:58.356+0000 7f3a272ce700 1 -- 172.21.15.160:0/2406431037 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a201072a0 con 0x7f3a200f5410 2023-12-25T20:12:58.359 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:58.357+0000 7f3a15ffb700 1 -- 172.21.15.160:0/2406431037 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3a10025070 con 0x7f3a200f5410 2023-12-25T20:12:58.359 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:58.357+0000 7f3a272ce700 1 -- 172.21.15.160:0/2406431037 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a20045ad0 con 0x7f3a200f5410 2023-12-25T20:12:58.359 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:58.357+0000 7f3a15ffb700 1 --2- 172.21.15.160:0/2406431037 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f3a0c02ff60 0x7f3a0c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:12:58.359 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:58.357+0000 7f3a15ffb700 1 -- 172.21.15.160:0/2406431037 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3a10040970 con 0x7f3a200f5410 2023-12-25T20:12:58.359 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:58.357+0000 7f3a24869700 1 --2- 172.21.15.160:0/2406431037 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f3a0c02ff60 0x7f3a0c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:12:58.359 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:58.358+0000 7f3a24869700 1 --2- 172.21.15.160:0/2406431037 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f3a0c02ff60 0x7f3a0c032420 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f3a1c0096a0 tx=0x7f3a1c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:12:58.362 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:58.362+0000 7f3a15ffb700 1 -- 172.21.15.160:0/2406431037 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3a100079c0 con 0x7f3a200f5410 2023-12-25T20:12:58.468 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:58 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:12:56.677451+0000 mgr.smithi050.nbzhun (mgr.14162) 177 : cluster [DBG] pgmap v165: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:12:58.567 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:58.566+0000 7f3a272ce700 1 -- 172.21.15.160:0/2406431037 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3a20108150 con 0x7f3a200f5410 2023-12-25T20:12:58.567 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:58.567+0000 7f3a15ffb700 1 -- 172.21.15.160:0/2406431037 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3a1000f300 con 0x7f3a200f5410 2023-12-25T20:12:58.568 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:12:58.568 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:12:58.570 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:58.568+0000 7f3a272ce700 1 -- 172.21.15.160:0/2406431037 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f3a0c02ff60 msgr2=0x7f3a0c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:12:58.570 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:58.568+0000 7f3a272ce700 1 --2- 172.21.15.160:0/2406431037 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f3a0c02ff60 0x7f3a0c032420 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f3a1c0096a0 tx=0x7f3a1c006b40).stop 2023-12-25T20:12:58.570 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:58.568+0000 7f3a272ce700 1 -- 172.21.15.160:0/2406431037 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3a200f5410 msgr2=0x7f3a20106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:12:58.570 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:58.568+0000 7f3a272ce700 1 --2- 172.21.15.160:0/2406431037 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3a200f5410 0x7f3a20106720 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7f3a10014000 tx=0x7f3a1000b040).stop 2023-12-25T20:12:58.570 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:58.569+0000 7f3a272ce700 1 -- 172.21.15.160:0/2406431037 shutdown_connections 2023-12-25T20:12:58.570 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:58.569+0000 7f3a272ce700 1 --2- 172.21.15.160:0/2406431037 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f3a0c02ff60 0x7f3a0c032420 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:12:58.570 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:58.569+0000 7f3a272ce700 1 --2- 172.21.15.160:0/2406431037 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3a200f5410 0x7f3a20106720 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:12:58.570 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:58.569+0000 7f3a272ce700 1 -- 172.21.15.160:0/2406431037 >> 172.21.15.160:0/2406431037 conn(0x7f3a200f0b70 msgr2=0x7f3a2005fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:12:58.570 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:58.569+0000 7f3a272ce700 1 -- 172.21.15.160:0/2406431037 shutdown_connections 2023-12-25T20:12:58.570 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:12:58.569+0000 7f3a272ce700 1 -- 172.21.15.160:0/2406431037 wait complete. 2023-12-25T20:12:58.570 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:12:59.468 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:12:59 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:12:58.568622+0000 mon.smithi050 (mon.0) 246 : audit [DBG] from='client.? 172.21.15.160:0/2406431037' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:12:59.967 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:12:59.967 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:13:00.468 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:13:00 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:12:58.677776+0000 mgr.smithi050.nbzhun (mgr.14162) 178 : cluster [DBG] pgmap v166: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:13:01.272 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:01.270+0000 7f2e10fd5700 1 -- 172.21.15.160:0/2318999540 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2e0c0f5410 msgr2=0x7f2e0c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:13:01.272 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:01.270+0000 7f2e10fd5700 1 --2- 172.21.15.160:0/2318999540 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2e0c0f5410 0x7f2e0c0f5830 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 rx=0x7f2dfc009fb0 tx=0x7f2dfc009b30).stop 2023-12-25T20:13:01.272 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:01.270+0000 7f2e10fd5700 1 -- 172.21.15.160:0/2318999540 shutdown_connections 2023-12-25T20:13:01.272 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:01.270+0000 7f2e10fd5700 1 --2- 172.21.15.160:0/2318999540 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2e0c0f5410 0x7f2e0c0f5830 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:13:01.272 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:01.270+0000 7f2e10fd5700 1 -- 172.21.15.160:0/2318999540 >> 172.21.15.160:0/2318999540 conn(0x7f2e0c0f0b70 msgr2=0x7f2e0c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:13:01.273 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:01.270+0000 7f2e10fd5700 1 -- 172.21.15.160:0/2318999540 shutdown_connections 2023-12-25T20:13:01.273 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:01.270+0000 7f2e10fd5700 1 -- 172.21.15.160:0/2318999540 wait complete. 2023-12-25T20:13:01.273 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:01.271+0000 7f2e10fd5700 1 Processor -- start 2023-12-25T20:13:01.273 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:01.271+0000 7f2e10fd5700 1 -- start start 2023-12-25T20:13:01.273 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:01.271+0000 7f2e10fd5700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2e0c0f5410 0x7f2e0c105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:13:01.273 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:01.271+0000 7f2e10fd5700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e0c105b20 con 0x7f2e0c0f5410 2023-12-25T20:13:01.273 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:01.271+0000 7f2e0a59c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2e0c0f5410 0x7f2e0c105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:13:01.273 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:01.271+0000 7f2e0a59c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2e0c0f5410 0x7f2e0c105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:37214/0 (socket says 172.21.15.160:37214) 2023-12-25T20:13:01.273 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:01.271+0000 7f2e0a59c700 1 -- 172.21.15.160:0/697970588 learned_addr learned my addr 172.21.15.160:0/697970588 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:13:01.273 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:01.272+0000 7f2e0a59c700 1 -- 172.21.15.160:0/697970588 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dfc005040 con 0x7f2e0c0f5410 2023-12-25T20:13:01.273 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:01.272+0000 7f2e0a59c700 1 --2- 172.21.15.160:0/697970588 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2e0c0f5410 0x7f2e0c105630 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7f2dfc013510 tx=0x7f2dfc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:13:01.274 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:01.272+0000 7f2e02ffd700 1 -- 172.21.15.160:0/697970588 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2dfc009320 con 0x7f2e0c0f5410 2023-12-25T20:13:01.274 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:01.272+0000 7f2e02ffd700 1 -- 172.21.15.160:0/697970588 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2dfc013f00 con 0x7f2e0c0f5410 2023-12-25T20:13:01.275 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:01.272+0000 7f2e10fd5700 1 -- 172.21.15.160:0/697970588 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e0c105d20 con 0x7f2e0c0f5410 2023-12-25T20:13:01.275 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:01.272+0000 7f2e02ffd700 1 -- 172.21.15.160:0/697970588 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2dfc014d90 con 0x7f2e0c0f5410 2023-12-25T20:13:01.275 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:01.272+0000 7f2e10fd5700 1 -- 172.21.15.160:0/697970588 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e0c1061b0 con 0x7f2e0c0f5410 2023-12-25T20:13:01.275 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:01.273+0000 7f2e02ffd700 1 -- 172.21.15.160:0/697970588 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2dfc0095f0 con 0x7f2e0c0f5410 2023-12-25T20:13:01.275 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:01.273+0000 7f2e10fd5700 1 -- 172.21.15.160:0/697970588 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e0c045ad0 con 0x7f2e0c0f5410 2023-12-25T20:13:01.275 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:01.273+0000 7f2e02ffd700 1 --2- 172.21.15.160:0/697970588 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f2df802ffb0 0x7f2df8032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:13:01.275 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:01.273+0000 7f2e02ffd700 1 -- 172.21.15.160:0/697970588 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2dfc042dc0 con 0x7f2e0c0f5410 2023-12-25T20:13:01.275 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:01.273+0000 7f2e09d9b700 1 --2- 172.21.15.160:0/697970588 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f2df802ffb0 0x7f2df8032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:13:01.275 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:01.274+0000 7f2e09d9b700 1 --2- 172.21.15.160:0/697970588 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f2df802ffb0 0x7f2df8032470 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f2df40096a0 tx=0x7f2df4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:13:01.279 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:01.278+0000 7f2e02ffd700 1 -- 172.21.15.160:0/697970588 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2dfc014420 con 0x7f2e0c0f5410 2023-12-25T20:13:01.482 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:01.481+0000 7f2e10fd5700 1 -- 172.21.15.160:0/697970588 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2e0c0fae60 con 0x7f2e0c0f5410 2023-12-25T20:13:01.482 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:01.482+0000 7f2e02ffd700 1 -- 172.21.15.160:0/697970588 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2dfc011290 con 0x7f2e0c0f5410 2023-12-25T20:13:01.482 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:13:01.483 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:13:01.484 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:01.483+0000 7f2e10fd5700 1 -- 172.21.15.160:0/697970588 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f2df802ffb0 msgr2=0x7f2df8032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:13:01.484 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:01.483+0000 7f2e10fd5700 1 --2- 172.21.15.160:0/697970588 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f2df802ffb0 0x7f2df8032470 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f2df40096a0 tx=0x7f2df4006b40).stop 2023-12-25T20:13:01.484 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:01.483+0000 7f2e10fd5700 1 -- 172.21.15.160:0/697970588 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2e0c0f5410 msgr2=0x7f2e0c105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:13:01.484 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:01.483+0000 7f2e10fd5700 1 --2- 172.21.15.160:0/697970588 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2e0c0f5410 0x7f2e0c105630 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7f2dfc013510 tx=0x7f2dfc00b040).stop 2023-12-25T20:13:01.484 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:01.483+0000 7f2e10fd5700 1 -- 172.21.15.160:0/697970588 shutdown_connections 2023-12-25T20:13:01.484 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:01.483+0000 7f2e10fd5700 1 --2- 172.21.15.160:0/697970588 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f2df802ffb0 0x7f2df8032470 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:13:01.484 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:01.483+0000 7f2e10fd5700 1 --2- 172.21.15.160:0/697970588 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2e0c0f5410 0x7f2e0c105630 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:13:01.484 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:01.483+0000 7f2e10fd5700 1 -- 172.21.15.160:0/697970588 >> 172.21.15.160:0/697970588 conn(0x7f2e0c0f0b70 msgr2=0x7f2e0c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:13:01.484 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:01.483+0000 7f2e10fd5700 1 -- 172.21.15.160:0/697970588 shutdown_connections 2023-12-25T20:13:01.484 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:01.483+0000 7f2e10fd5700 1 -- 172.21.15.160:0/697970588 wait complete. 2023-12-25T20:13:01.485 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:13:02.468 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:13:02 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:13:00.678033+0000 mgr.smithi050.nbzhun (mgr.14162) 179 : cluster [DBG] pgmap v167: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:13:02.469 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:13:02 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:13:01.483427+0000 mon.smithi050 (mon.0) 247 : audit [DBG] from='client.? 172.21.15.160:0/697970588' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:13:02.887 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:13:02.887 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:13:04.118 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:04.115+0000 7fd81a079700 1 -- 172.21.15.160:0/1655376062 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd8140601e0 msgr2=0x7fd8140f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:13:04.118 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:04.115+0000 7fd81a079700 1 --2- 172.21.15.160:0/1655376062 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd8140601e0 0x7fd8140f8f20 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 rx=0x7fd810009fb0 tx=0x7fd810009b30).stop 2023-12-25T20:13:04.118 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:04.115+0000 7fd81a079700 1 -- 172.21.15.160:0/1655376062 shutdown_connections 2023-12-25T20:13:04.118 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:04.115+0000 7fd81a079700 1 --2- 172.21.15.160:0/1655376062 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd8140601e0 0x7fd8140f8f20 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:13:04.118 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:04.115+0000 7fd81a079700 1 -- 172.21.15.160:0/1655376062 >> 172.21.15.160:0/1655376062 conn(0x7fd8140f09a0 msgr2=0x7fd8140f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:13:04.118 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:04.115+0000 7fd81a079700 1 -- 172.21.15.160:0/1655376062 shutdown_connections 2023-12-25T20:13:04.118 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:04.115+0000 7fd81a079700 1 -- 172.21.15.160:0/1655376062 wait complete. 2023-12-25T20:13:04.118 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:04.115+0000 7fd81a079700 1 Processor -- start 2023-12-25T20:13:04.118 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:04.115+0000 7fd81a079700 1 -- start start 2023-12-25T20:13:04.119 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:04.115+0000 7fd81a079700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd8140601e0 0x7fd814106500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:13:04.119 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:04.115+0000 7fd81a079700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8141069f0 con 0x7fd8140601e0 2023-12-25T20:13:04.119 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:04.116+0000 7fd819077700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd8140601e0 0x7fd814106500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:13:04.119 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:04.116+0000 7fd819077700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd8140601e0 0x7fd814106500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:37236/0 (socket says 172.21.15.160:37236) 2023-12-25T20:13:04.119 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:04.116+0000 7fd819077700 1 -- 172.21.15.160:0/2513081276 learned_addr learned my addr 172.21.15.160:0/2513081276 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:13:04.119 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:04.116+0000 7fd819077700 1 -- 172.21.15.160:0/2513081276 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd810005040 con 0x7fd8140601e0 2023-12-25T20:13:04.119 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:04.116+0000 7fd819077700 1 --2- 172.21.15.160:0/2513081276 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd8140601e0 0x7fd814106500 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7fd810013510 tx=0x7fd81000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:13:04.119 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:04.116+0000 7fd809ffb700 1 -- 172.21.15.160:0/2513081276 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd810009320 con 0x7fd8140601e0 2023-12-25T20:13:04.119 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:04.116+0000 7fd81a079700 1 -- 172.21.15.160:0/2513081276 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd814106bf0 con 0x7fd8140601e0 2023-12-25T20:13:04.119 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:04.116+0000 7fd809ffb700 1 -- 172.21.15.160:0/2513081276 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd810013f00 con 0x7fd8140601e0 2023-12-25T20:13:04.119 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:04.116+0000 7fd809ffb700 1 -- 172.21.15.160:0/2513081276 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd810014d90 con 0x7fd8140601e0 2023-12-25T20:13:04.119 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:04.117+0000 7fd81a079700 1 -- 172.21.15.160:0/2513081276 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd814107080 con 0x7fd8140601e0 2023-12-25T20:13:04.120 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:04.117+0000 7fd809ffb700 1 -- 172.21.15.160:0/2513081276 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd8100095f0 con 0x7fd8140601e0 2023-12-25T20:13:04.120 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:04.117+0000 7fd81a079700 1 -- 172.21.15.160:0/2513081276 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd814045ad0 con 0x7fd8140601e0 2023-12-25T20:13:04.120 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:04.118+0000 7fd809ffb700 1 --2- 172.21.15.160:0/2513081276 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fd80002ff60 0x7fd800032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:13:04.120 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:04.118+0000 7fd809ffb700 1 -- 172.21.15.160:0/2513081276 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd810040fe0 con 0x7fd8140601e0 2023-12-25T20:13:04.120 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:04.118+0000 7fd818876700 1 --2- 172.21.15.160:0/2513081276 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fd80002ff60 0x7fd800032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:13:04.121 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:04.118+0000 7fd818876700 1 --2- 172.21.15.160:0/2513081276 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fd80002ff60 0x7fd800032420 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7fd8040096a0 tx=0x7fd804006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:13:04.123 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:04.123+0000 7fd809ffb700 1 -- 172.21.15.160:0/2513081276 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd810014420 con 0x7fd8140601e0 2023-12-25T20:13:04.328 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:04.327+0000 7fd81a079700 1 -- 172.21.15.160:0/2513081276 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd814107ed0 con 0x7fd8140601e0 2023-12-25T20:13:04.328 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:04.328+0000 7fd809ffb700 1 -- 172.21.15.160:0/2513081276 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd810010230 con 0x7fd8140601e0 2023-12-25T20:13:04.329 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:13:04.329 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:13:04.330 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:04.330+0000 7fd81a079700 1 -- 172.21.15.160:0/2513081276 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fd80002ff60 msgr2=0x7fd800032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:13:04.330 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:04.330+0000 7fd81a079700 1 --2- 172.21.15.160:0/2513081276 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fd80002ff60 0x7fd800032420 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7fd8040096a0 tx=0x7fd804006b40).stop 2023-12-25T20:13:04.331 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:04.330+0000 7fd81a079700 1 -- 172.21.15.160:0/2513081276 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd8140601e0 msgr2=0x7fd814106500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:13:04.331 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:04.330+0000 7fd81a079700 1 --2- 172.21.15.160:0/2513081276 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd8140601e0 0x7fd814106500 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7fd810013510 tx=0x7fd81000b040).stop 2023-12-25T20:13:04.331 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:04.330+0000 7fd81a079700 1 -- 172.21.15.160:0/2513081276 shutdown_connections 2023-12-25T20:13:04.331 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:04.330+0000 7fd81a079700 1 --2- 172.21.15.160:0/2513081276 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fd80002ff60 0x7fd800032420 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:13:04.331 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:04.330+0000 7fd81a079700 1 --2- 172.21.15.160:0/2513081276 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd8140601e0 0x7fd814106500 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:13:04.331 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:04.330+0000 7fd81a079700 1 -- 172.21.15.160:0/2513081276 >> 172.21.15.160:0/2513081276 conn(0x7fd8140f09a0 msgr2=0x7fd8140f1420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:13:04.331 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:04.330+0000 7fd81a079700 1 -- 172.21.15.160:0/2513081276 shutdown_connections 2023-12-25T20:13:04.331 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:04.330+0000 7fd81a079700 1 -- 172.21.15.160:0/2513081276 wait complete. 2023-12-25T20:13:04.331 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:13:04.468 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:13:04 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:13:02.678378+0000 mgr.smithi050.nbzhun (mgr.14162) 180 : cluster [DBG] pgmap v168: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:13:05.468 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:13:05 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:13:04.329653+0000 mon.smithi050 (mon.0) 248 : audit [DBG] from='client.? 172.21.15.160:0/2513081276' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:13:05.745 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:13:05.745 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:13:06.468 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:13:06 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:13:04.678718+0000 mgr.smithi050.nbzhun (mgr.14162) 181 : cluster [DBG] pgmap v169: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:13:06.910 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:06.909+0000 7f06e52a4700 1 -- 172.21.15.160:0/2691688827 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f06e00f62a0 msgr2=0x7f06e00f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:13:06.910 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:06.909+0000 7f06e52a4700 1 --2- 172.21.15.160:0/2691688827 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f06e00f62a0 0x7f06e00f66c0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 rx=0x7f06d0009fb0 tx=0x7f06d0009b30).stop 2023-12-25T20:13:06.910 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:06.909+0000 7f06e52a4700 1 -- 172.21.15.160:0/2691688827 shutdown_connections 2023-12-25T20:13:06.910 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:06.909+0000 7f06e52a4700 1 --2- 172.21.15.160:0/2691688827 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f06e00f62a0 0x7f06e00f66c0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:13:06.910 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:06.909+0000 7f06e52a4700 1 -- 172.21.15.160:0/2691688827 >> 172.21.15.160:0/2691688827 conn(0x7f06e00f1aa0 msgr2=0x7f06e00f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:13:06.910 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:06.909+0000 7f06e52a4700 1 -- 172.21.15.160:0/2691688827 shutdown_connections 2023-12-25T20:13:06.911 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:06.909+0000 7f06e52a4700 1 -- 172.21.15.160:0/2691688827 wait complete. 2023-12-25T20:13:06.911 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:06.909+0000 7f06e52a4700 1 Processor -- start 2023-12-25T20:13:06.911 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:06.909+0000 7f06e52a4700 1 -- start start 2023-12-25T20:13:06.911 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:06.910+0000 7f06e52a4700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f06e00f62a0 0x7f06e01064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:13:06.911 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:06.910+0000 7f06e52a4700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06e01069a0 con 0x7f06e00f62a0 2023-12-25T20:13:06.913 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:06.910+0000 7f06dffff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f06e00f62a0 0x7f06e01064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:13:06.913 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:06.910+0000 7f06dffff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f06e00f62a0 0x7f06e01064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:37262/0 (socket says 172.21.15.160:37262) 2023-12-25T20:13:06.914 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:06.910+0000 7f06dffff700 1 -- 172.21.15.160:0/1756755078 learned_addr learned my addr 172.21.15.160:0/1756755078 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:13:06.914 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:06.910+0000 7f06dffff700 1 -- 172.21.15.160:0/1756755078 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06d0005040 con 0x7f06e00f62a0 2023-12-25T20:13:06.914 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:06.910+0000 7f06dffff700 1 --2- 172.21.15.160:0/1756755078 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f06e00f62a0 0x7f06e01064b0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7f06d0013510 tx=0x7f06d000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:13:06.914 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:06.910+0000 7f06dcff9700 1 -- 172.21.15.160:0/1756755078 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f06d0009320 con 0x7f06e00f62a0 2023-12-25T20:13:06.914 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:06.910+0000 7f06dcff9700 1 -- 172.21.15.160:0/1756755078 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f06d0013f00 con 0x7f06e00f62a0 2023-12-25T20:13:06.914 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:06.910+0000 7f06e52a4700 1 -- 172.21.15.160:0/1756755078 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06e0106ba0 con 0x7f06e00f62a0 2023-12-25T20:13:06.915 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:06.911+0000 7f06dcff9700 1 -- 172.21.15.160:0/1756755078 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f06d0014dc0 con 0x7f06e00f62a0 2023-12-25T20:13:06.915 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:06.911+0000 7f06e52a4700 1 -- 172.21.15.160:0/1756755078 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06e0107030 con 0x7f06e00f62a0 2023-12-25T20:13:06.915 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:06.911+0000 7f06dcff9700 1 -- 172.21.15.160:0/1756755078 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f06d00095f0 con 0x7f06e00f62a0 2023-12-25T20:13:06.915 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:06.911+0000 7f06e52a4700 1 -- 172.21.15.160:0/1756755078 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06e0045ad0 con 0x7f06e00f62a0 2023-12-25T20:13:06.915 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:06.911+0000 7f06dcff9700 1 --2- 172.21.15.160:0/1756755078 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f06c802ff60 0x7f06c8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:13:06.915 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:06.912+0000 7f06dcff9700 1 -- 172.21.15.160:0/1756755078 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f06d00410c0 con 0x7f06e00f62a0 2023-12-25T20:13:06.915 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:06.912+0000 7f06df7fe700 1 --2- 172.21.15.160:0/1756755078 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f06c802ff60 0x7f06c8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:13:06.915 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:06.912+0000 7f06df7fe700 1 --2- 172.21.15.160:0/1756755078 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f06c802ff60 0x7f06c8032420 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f06d40096a0 tx=0x7f06d4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:13:06.917 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:06.917+0000 7f06dcff9700 1 -- 172.21.15.160:0/1756755078 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f06d0014420 con 0x7f06e00f62a0 2023-12-25T20:13:07.125 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:07.124+0000 7f06e52a4700 1 -- 172.21.15.160:0/1756755078 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f06e0107ee0 con 0x7f06e00f62a0 2023-12-25T20:13:07.126 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:07.125+0000 7f06dcff9700 1 -- 172.21.15.160:0/1756755078 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f06d0010230 con 0x7f06e00f62a0 2023-12-25T20:13:07.126 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:13:07.126 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:13:07.128 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:07.127+0000 7f06e52a4700 1 -- 172.21.15.160:0/1756755078 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f06c802ff60 msgr2=0x7f06c8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:13:07.128 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:07.127+0000 7f06e52a4700 1 --2- 172.21.15.160:0/1756755078 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f06c802ff60 0x7f06c8032420 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f06d40096a0 tx=0x7f06d4006b40).stop 2023-12-25T20:13:07.128 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:07.127+0000 7f06e52a4700 1 -- 172.21.15.160:0/1756755078 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f06e00f62a0 msgr2=0x7f06e01064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:13:07.128 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:07.127+0000 7f06e52a4700 1 --2- 172.21.15.160:0/1756755078 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f06e00f62a0 0x7f06e01064b0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7f06d0013510 tx=0x7f06d000b040).stop 2023-12-25T20:13:07.128 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:07.127+0000 7f06e52a4700 1 -- 172.21.15.160:0/1756755078 shutdown_connections 2023-12-25T20:13:07.128 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:07.127+0000 7f06e52a4700 1 --2- 172.21.15.160:0/1756755078 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f06c802ff60 0x7f06c8032420 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:13:07.128 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:07.127+0000 7f06e52a4700 1 --2- 172.21.15.160:0/1756755078 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f06e00f62a0 0x7f06e01064b0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:13:07.128 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:07.127+0000 7f06e52a4700 1 -- 172.21.15.160:0/1756755078 >> 172.21.15.160:0/1756755078 conn(0x7f06e00f1aa0 msgr2=0x7f06e00f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:13:07.128 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:07.127+0000 7f06e52a4700 1 -- 172.21.15.160:0/1756755078 shutdown_connections 2023-12-25T20:13:07.128 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:07.127+0000 7f06e52a4700 1 -- 172.21.15.160:0/1756755078 wait complete. 2023-12-25T20:13:07.129 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:13:08.468 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:13:08 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:13:06.679039+0000 mgr.smithi050.nbzhun (mgr.14162) 182 : cluster [DBG] pgmap v170: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:13:08.468 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:13:08 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:13:07.127014+0000 mon.smithi050 (mon.0) 249 : audit [DBG] from='client.? 172.21.15.160:0/1756755078' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:13:08.534 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:13:08.534 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:13:09.750 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:09.748+0000 7f6d3ae09700 1 -- 172.21.15.160:0/1295831956 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6d340f5410 msgr2=0x7f6d340f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:13:09.750 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:09.748+0000 7f6d3ae09700 1 --2- 172.21.15.160:0/1295831956 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6d340f5410 0x7f6d340f5830 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 rx=0x7f6d28009fb0 tx=0x7f6d28009b30).stop 2023-12-25T20:13:09.751 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:09.748+0000 7f6d3ae09700 1 -- 172.21.15.160:0/1295831956 shutdown_connections 2023-12-25T20:13:09.751 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:09.748+0000 7f6d3ae09700 1 --2- 172.21.15.160:0/1295831956 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6d340f5410 0x7f6d340f5830 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:13:09.751 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:09.748+0000 7f6d3ae09700 1 -- 172.21.15.160:0/1295831956 >> 172.21.15.160:0/1295831956 conn(0x7f6d340f0b70 msgr2=0x7f6d340f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:13:09.751 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:09.748+0000 7f6d3ae09700 1 -- 172.21.15.160:0/1295831956 shutdown_connections 2023-12-25T20:13:09.751 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:09.748+0000 7f6d3ae09700 1 -- 172.21.15.160:0/1295831956 wait complete. 2023-12-25T20:13:09.751 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:09.749+0000 7f6d3ae09700 1 Processor -- start 2023-12-25T20:13:09.751 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:09.749+0000 7f6d3ae09700 1 -- start start 2023-12-25T20:13:09.751 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:09.749+0000 7f6d3ae09700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6d340f5410 0x7f6d341093d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:13:09.751 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:09.749+0000 7f6d3ae09700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d34060240 con 0x7f6d340f5410 2023-12-25T20:13:09.753 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:09.750+0000 7f6d38ba5700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6d340f5410 0x7f6d341093d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:13:09.753 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:09.750+0000 7f6d38ba5700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6d340f5410 0x7f6d341093d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:35346/0 (socket says 172.21.15.160:35346) 2023-12-25T20:13:09.753 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:09.750+0000 7f6d38ba5700 1 -- 172.21.15.160:0/2341547866 learned_addr learned my addr 172.21.15.160:0/2341547866 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:13:09.753 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:09.750+0000 7f6d38ba5700 1 -- 172.21.15.160:0/2341547866 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d28005040 con 0x7f6d340f5410 2023-12-25T20:13:09.753 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:09.750+0000 7f6d38ba5700 1 --2- 172.21.15.160:0/2341547866 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6d340f5410 0x7f6d341093d0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7f6d28013510 tx=0x7f6d2800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:13:09.753 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:09.750+0000 7f6d317fa700 1 -- 172.21.15.160:0/2341547866 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6d28009320 con 0x7f6d340f5410 2023-12-25T20:13:09.753 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:09.750+0000 7f6d317fa700 1 -- 172.21.15.160:0/2341547866 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6d28013f00 con 0x7f6d340f5410 2023-12-25T20:13:09.753 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:09.750+0000 7f6d3ae09700 1 -- 172.21.15.160:0/2341547866 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d34060440 con 0x7f6d340f5410 2023-12-25T20:13:09.753 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:09.750+0000 7f6d317fa700 1 -- 172.21.15.160:0/2341547866 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6d28014d90 con 0x7f6d340f5410 2023-12-25T20:13:09.753 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:09.750+0000 7f6d3ae09700 1 -- 172.21.15.160:0/2341547866 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d34109a80 con 0x7f6d340f5410 2023-12-25T20:13:09.754 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:09.751+0000 7f6d317fa700 1 -- 172.21.15.160:0/2341547866 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6d280095f0 con 0x7f6d340f5410 2023-12-25T20:13:09.754 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:09.751+0000 7f6d3ae09700 1 -- 172.21.15.160:0/2341547866 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d34045ad0 con 0x7f6d340f5410 2023-12-25T20:13:09.754 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:09.751+0000 7f6d317fa700 1 --2- 172.21.15.160:0/2341547866 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f6d2402ff60 0x7f6d24032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:13:09.754 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:09.751+0000 7f6d317fa700 1 -- 172.21.15.160:0/2341547866 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6d280410c0 con 0x7f6d340f5410 2023-12-25T20:13:09.754 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:09.752+0000 7f6d33fff700 1 --2- 172.21.15.160:0/2341547866 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f6d2402ff60 0x7f6d24032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:13:09.754 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:09.752+0000 7f6d33fff700 1 --2- 172.21.15.160:0/2341547866 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f6d2402ff60 0x7f6d24032420 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f6d200096a0 tx=0x7f6d20006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:13:09.757 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:09.757+0000 7f6d317fa700 1 -- 172.21.15.160:0/2341547866 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6d2801d7f0 con 0x7f6d340f5410 2023-12-25T20:13:09.966 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:09.965+0000 7f6d3ae09700 1 -- 172.21.15.160:0/2341547866 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6d3410a2a0 con 0x7f6d340f5410 2023-12-25T20:13:09.966 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:09.965+0000 7f6d317fa700 1 -- 172.21.15.160:0/2341547866 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6d28010230 con 0x7f6d340f5410 2023-12-25T20:13:09.966 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:13:09.966 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:13:09.968 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:09.967+0000 7f6d3ae09700 1 -- 172.21.15.160:0/2341547866 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f6d2402ff60 msgr2=0x7f6d24032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:13:09.968 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:09.967+0000 7f6d3ae09700 1 --2- 172.21.15.160:0/2341547866 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f6d2402ff60 0x7f6d24032420 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f6d200096a0 tx=0x7f6d20006b40).stop 2023-12-25T20:13:09.968 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:09.967+0000 7f6d3ae09700 1 -- 172.21.15.160:0/2341547866 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6d340f5410 msgr2=0x7f6d341093d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:13:09.968 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:09.967+0000 7f6d3ae09700 1 --2- 172.21.15.160:0/2341547866 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6d340f5410 0x7f6d341093d0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7f6d28013510 tx=0x7f6d2800b040).stop 2023-12-25T20:13:09.968 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:09.967+0000 7f6d3ae09700 1 -- 172.21.15.160:0/2341547866 shutdown_connections 2023-12-25T20:13:09.968 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:09.967+0000 7f6d3ae09700 1 --2- 172.21.15.160:0/2341547866 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f6d2402ff60 0x7f6d24032420 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:13:09.968 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:09.967+0000 7f6d3ae09700 1 --2- 172.21.15.160:0/2341547866 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6d340f5410 0x7f6d341093d0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:13:09.968 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:09.967+0000 7f6d3ae09700 1 -- 172.21.15.160:0/2341547866 >> 172.21.15.160:0/2341547866 conn(0x7f6d340f0b70 msgr2=0x7f6d340f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:13:09.968 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:09.967+0000 7f6d3ae09700 1 -- 172.21.15.160:0/2341547866 shutdown_connections 2023-12-25T20:13:09.968 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:09.967+0000 7f6d3ae09700 1 -- 172.21.15.160:0/2341547866 wait complete. 2023-12-25T20:13:09.969 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:13:10.468 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:13:10 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:13:08.679354+0000 mgr.smithi050.nbzhun (mgr.14162) 183 : cluster [DBG] pgmap v171: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:13:10.468 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:13:10 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:13:09.967303+0000 mon.smithi050 (mon.0) 250 : audit [DBG] from='client.? 172.21.15.160:0/2341547866' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:13:11.354 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:13:11.354 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:13:12.468 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:13:12 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:13:10.679665+0000 mgr.smithi050.nbzhun (mgr.14162) 184 : cluster [DBG] pgmap v172: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:13:12.589 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:12.588+0000 7f5e47cab700 1 -- 172.21.15.160:0/106165030 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5e400f6990 msgr2=0x7f5e400f6db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:13:12.592 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:12.588+0000 7f5e47cab700 1 --2- 172.21.15.160:0/106165030 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5e400f6990 0x7f5e400f6db0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 rx=0x7f5e3c009fb0 tx=0x7f5e3c009b30).stop 2023-12-25T20:13:12.593 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:12.588+0000 7f5e47cab700 1 -- 172.21.15.160:0/106165030 shutdown_connections 2023-12-25T20:13:12.593 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:12.588+0000 7f5e47cab700 1 --2- 172.21.15.160:0/106165030 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5e400f6990 0x7f5e400f6db0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:13:12.593 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:12.588+0000 7f5e47cab700 1 -- 172.21.15.160:0/106165030 >> 172.21.15.160:0/106165030 conn(0x7f5e400f0260 msgr2=0x7f5e400f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:13:12.593 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:12.588+0000 7f5e47cab700 1 -- 172.21.15.160:0/106165030 shutdown_connections 2023-12-25T20:13:12.593 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:12.588+0000 7f5e47cab700 1 -- 172.21.15.160:0/106165030 wait complete. 2023-12-25T20:13:12.593 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:12.589+0000 7f5e47cab700 1 Processor -- start 2023-12-25T20:13:12.593 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:12.589+0000 7f5e47cab700 1 -- start start 2023-12-25T20:13:12.593 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:12.589+0000 7f5e47cab700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5e400f6990 0x7f5e40109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:13:12.594 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:12.589+0000 7f5e47cab700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e400f52d0 con 0x7f5e400f6990 2023-12-25T20:13:12.594 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:12.589+0000 7f5e45a47700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5e400f6990 0x7f5e40109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:13:12.594 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:12.589+0000 7f5e45a47700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5e400f6990 0x7f5e40109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:35376/0 (socket says 172.21.15.160:35376) 2023-12-25T20:13:12.594 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:12.589+0000 7f5e45a47700 1 -- 172.21.15.160:0/2550748281 learned_addr learned my addr 172.21.15.160:0/2550748281 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:13:12.595 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:12.589+0000 7f5e45a47700 1 -- 172.21.15.160:0/2550748281 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e3c005040 con 0x7f5e400f6990 2023-12-25T20:13:12.595 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:12.589+0000 7f5e45a47700 1 --2- 172.21.15.160:0/2550748281 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5e400f6990 0x7f5e40109430 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7f5e3c013510 tx=0x7f5e3c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:13:12.595 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:12.590+0000 7f5e367fc700 1 -- 172.21.15.160:0/2550748281 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5e3c009320 con 0x7f5e400f6990 2023-12-25T20:13:12.595 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:12.590+0000 7f5e47cab700 1 -- 172.21.15.160:0/2550748281 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e400f54d0 con 0x7f5e400f6990 2023-12-25T20:13:12.595 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:12.590+0000 7f5e367fc700 1 -- 172.21.15.160:0/2550748281 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5e3c013f00 con 0x7f5e400f6990 2023-12-25T20:13:12.595 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:12.590+0000 7f5e367fc700 1 -- 172.21.15.160:0/2550748281 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5e3c014d90 con 0x7f5e400f6990 2023-12-25T20:13:12.596 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:12.590+0000 7f5e47cab700 1 -- 172.21.15.160:0/2550748281 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e40109ae0 con 0x7f5e400f6990 2023-12-25T20:13:12.596 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:12.590+0000 7f5e367fc700 1 -- 172.21.15.160:0/2550748281 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5e3c0095f0 con 0x7f5e400f6990 2023-12-25T20:13:12.596 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:12.590+0000 7f5e47cab700 1 -- 172.21.15.160:0/2550748281 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e40045ad0 con 0x7f5e400f6990 2023-12-25T20:13:12.596 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:12.591+0000 7f5e367fc700 1 --2- 172.21.15.160:0/2550748281 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f5e2c02ff60 0x7f5e2c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:13:12.596 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:12.591+0000 7f5e367fc700 1 -- 172.21.15.160:0/2550748281 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5e3c041000 con 0x7f5e400f6990 2023-12-25T20:13:12.596 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:12.591+0000 7f5e45246700 1 --2- 172.21.15.160:0/2550748281 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f5e2c02ff60 0x7f5e2c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:13:12.596 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:12.591+0000 7f5e45246700 1 --2- 172.21.15.160:0/2550748281 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f5e2c02ff60 0x7f5e2c032420 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f5e300096a0 tx=0x7f5e30006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:13:12.596 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:12.596+0000 7f5e367fc700 1 -- 172.21.15.160:0/2550748281 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5e3c014420 con 0x7f5e400f6990 2023-12-25T20:13:12.800 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:12.799+0000 7f5e47cab700 1 -- 172.21.15.160:0/2550748281 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5e400fae60 con 0x7f5e400f6990 2023-12-25T20:13:12.800 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:12.800+0000 7f5e367fc700 1 -- 172.21.15.160:0/2550748281 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5e3c01d7f0 con 0x7f5e400f6990 2023-12-25T20:13:12.801 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:13:12.801 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:13:12.802 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:12.801+0000 7f5e47cab700 1 -- 172.21.15.160:0/2550748281 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f5e2c02ff60 msgr2=0x7f5e2c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:13:12.803 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:12.801+0000 7f5e47cab700 1 --2- 172.21.15.160:0/2550748281 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f5e2c02ff60 0x7f5e2c032420 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f5e300096a0 tx=0x7f5e30006b40).stop 2023-12-25T20:13:12.803 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:12.801+0000 7f5e47cab700 1 -- 172.21.15.160:0/2550748281 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5e400f6990 msgr2=0x7f5e40109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:13:12.803 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:12.801+0000 7f5e47cab700 1 --2- 172.21.15.160:0/2550748281 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5e400f6990 0x7f5e40109430 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7f5e3c013510 tx=0x7f5e3c00b040).stop 2023-12-25T20:13:12.803 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:12.801+0000 7f5e47cab700 1 -- 172.21.15.160:0/2550748281 shutdown_connections 2023-12-25T20:13:12.803 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:12.801+0000 7f5e47cab700 1 --2- 172.21.15.160:0/2550748281 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f5e2c02ff60 0x7f5e2c032420 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:13:12.803 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:12.801+0000 7f5e47cab700 1 --2- 172.21.15.160:0/2550748281 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5e400f6990 0x7f5e40109430 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:13:12.803 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:12.801+0000 7f5e47cab700 1 -- 172.21.15.160:0/2550748281 >> 172.21.15.160:0/2550748281 conn(0x7f5e400f0260 msgr2=0x7f5e400f0d50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:13:12.803 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:12.801+0000 7f5e47cab700 1 -- 172.21.15.160:0/2550748281 shutdown_connections 2023-12-25T20:13:12.803 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:12.801+0000 7f5e47cab700 1 -- 172.21.15.160:0/2550748281 wait complete. 2023-12-25T20:13:12.804 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:13:13.468 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:13:13 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:13:12.801529+0000 mon.smithi050 (mon.0) 251 : audit [DBG] from='client.? 172.21.15.160:0/2550748281' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:13:14.200 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:13:14.201 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:13:14.468 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:13:14 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:13:12.679991+0000 mgr.smithi050.nbzhun (mgr.14162) 185 : cluster [DBG] pgmap v173: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:13:15.402 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:15.401+0000 7fb22a33a700 1 -- 172.21.15.160:0/797289557 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb2240f62a0 msgr2=0x7fb2240f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:13:15.402 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:15.401+0000 7fb22a33a700 1 --2- 172.21.15.160:0/797289557 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb2240f62a0 0x7fb2240f66c0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 rx=0x7fb220009fb0 tx=0x7fb220009b30).stop 2023-12-25T20:13:15.402 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:15.401+0000 7fb22a33a700 1 -- 172.21.15.160:0/797289557 shutdown_connections 2023-12-25T20:13:15.402 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:15.401+0000 7fb22a33a700 1 --2- 172.21.15.160:0/797289557 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb2240f62a0 0x7fb2240f66c0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:13:15.403 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:15.401+0000 7fb22a33a700 1 -- 172.21.15.160:0/797289557 >> 172.21.15.160:0/797289557 conn(0x7fb2240f1aa0 msgr2=0x7fb2240f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:13:15.403 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:15.401+0000 7fb22a33a700 1 -- 172.21.15.160:0/797289557 shutdown_connections 2023-12-25T20:13:15.403 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:15.401+0000 7fb22a33a700 1 -- 172.21.15.160:0/797289557 wait complete. 2023-12-25T20:13:15.403 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:15.402+0000 7fb22a33a700 1 Processor -- start 2023-12-25T20:13:15.403 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:15.402+0000 7fb22a33a700 1 -- start start 2023-12-25T20:13:15.403 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:15.402+0000 7fb22a33a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb2240f62a0 0x7fb2241064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:13:15.403 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:15.402+0000 7fb22a33a700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2241069a0 con 0x7fb2240f62a0 2023-12-25T20:13:15.403 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:15.403+0000 7fb229338700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb2240f62a0 0x7fb2241064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:13:15.403 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:15.403+0000 7fb229338700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb2240f62a0 0x7fb2241064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:35392/0 (socket says 172.21.15.160:35392) 2023-12-25T20:13:15.403 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:15.403+0000 7fb229338700 1 -- 172.21.15.160:0/78653825 learned_addr learned my addr 172.21.15.160:0/78653825 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:13:15.404 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:15.403+0000 7fb229338700 1 -- 172.21.15.160:0/78653825 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb220005040 con 0x7fb2240f62a0 2023-12-25T20:13:15.405 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:15.403+0000 7fb229338700 1 --2- 172.21.15.160:0/78653825 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb2240f62a0 0x7fb2241064b0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7fb220009830 tx=0x7fb22000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:13:15.405 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:15.403+0000 7fb219ffb700 1 -- 172.21.15.160:0/78653825 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb220009320 con 0x7fb2240f62a0 2023-12-25T20:13:15.405 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:15.403+0000 7fb219ffb700 1 -- 172.21.15.160:0/78653825 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb2200136c0 con 0x7fb2240f62a0 2023-12-25T20:13:15.405 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:15.403+0000 7fb22a33a700 1 -- 172.21.15.160:0/78653825 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb224106ba0 con 0x7fb2240f62a0 2023-12-25T20:13:15.405 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:15.403+0000 7fb219ffb700 1 -- 172.21.15.160:0/78653825 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb22001ee90 con 0x7fb2240f62a0 2023-12-25T20:13:15.405 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:15.404+0000 7fb22a33a700 1 -- 172.21.15.160:0/78653825 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb224107030 con 0x7fb2240f62a0 2023-12-25T20:13:15.405 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:15.405+0000 7fb22a33a700 1 -- 172.21.15.160:0/78653825 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb224045ad0 con 0x7fb2240f62a0 2023-12-25T20:13:15.405 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:15.405+0000 7fb219ffb700 1 -- 172.21.15.160:0/78653825 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb220014420 con 0x7fb2240f62a0 2023-12-25T20:13:15.405 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:15.405+0000 7fb219ffb700 1 --2- 172.21.15.160:0/78653825 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fb21002ff10 0x7fb2100323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:13:15.405 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:15.405+0000 7fb219ffb700 1 -- 172.21.15.160:0/78653825 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb220040ef0 con 0x7fb2240f62a0 2023-12-25T20:13:15.405 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:15.405+0000 7fb228b37700 1 --2- 172.21.15.160:0/78653825 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fb21002ff10 0x7fb2100323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:13:15.406 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:15.406+0000 7fb228b37700 1 --2- 172.21.15.160:0/78653825 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fb21002ff10 0x7fb2100323d0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7fb2140096a0 tx=0x7fb214006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:13:15.409 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:15.410+0000 7fb219ffb700 1 -- 172.21.15.160:0/78653825 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb22001c6e0 con 0x7fb2240f62a0 2023-12-25T20:13:15.468 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:13:15 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:13:14.767576+0000 mon.smithi050 (mon.0) 252 : audit [INF] from='mgr.14162 172.21.15.50:0/3178367813' entity='mgr.smithi050.nbzhun' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi050.nbzhun/mirror_snapshot_schedule"}]: dispatch 2023-12-25T20:13:15.469 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:13:15 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:13:14.768005+0000 mon.smithi050 (mon.0) 253 : audit [INF] from='mgr.14162 172.21.15.50:0/3178367813' entity='mgr.smithi050.nbzhun' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi050.nbzhun/trash_purge_schedule"}]: dispatch 2023-12-25T20:13:15.616 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:15.616+0000 7fb22a33a700 1 -- 172.21.15.160:0/78653825 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb224107e80 con 0x7fb2240f62a0 2023-12-25T20:13:15.616 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:15.616+0000 7fb219ffb700 1 -- 172.21.15.160:0/78653825 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb22002c2d0 con 0x7fb2240f62a0 2023-12-25T20:13:15.616 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:13:15.617 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:13:15.618 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:15.618+0000 7fb22a33a700 1 -- 172.21.15.160:0/78653825 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fb21002ff10 msgr2=0x7fb2100323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:13:15.618 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:15.618+0000 7fb22a33a700 1 --2- 172.21.15.160:0/78653825 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fb21002ff10 0x7fb2100323d0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7fb2140096a0 tx=0x7fb214006b40).stop 2023-12-25T20:13:15.618 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:15.618+0000 7fb22a33a700 1 -- 172.21.15.160:0/78653825 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb2240f62a0 msgr2=0x7fb2241064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:13:15.618 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:15.618+0000 7fb22a33a700 1 --2- 172.21.15.160:0/78653825 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb2240f62a0 0x7fb2241064b0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7fb220009830 tx=0x7fb22000b040).stop 2023-12-25T20:13:15.618 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:15.618+0000 7fb22a33a700 1 -- 172.21.15.160:0/78653825 shutdown_connections 2023-12-25T20:13:15.618 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:15.618+0000 7fb22a33a700 1 --2- 172.21.15.160:0/78653825 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fb21002ff10 0x7fb2100323d0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:13:15.618 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:15.618+0000 7fb22a33a700 1 --2- 172.21.15.160:0/78653825 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb2240f62a0 0x7fb2241064b0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:13:15.618 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:15.618+0000 7fb22a33a700 1 -- 172.21.15.160:0/78653825 >> 172.21.15.160:0/78653825 conn(0x7fb2240f1aa0 msgr2=0x7fb2240f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:13:15.618 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:15.618+0000 7fb22a33a700 1 -- 172.21.15.160:0/78653825 shutdown_connections 2023-12-25T20:13:15.618 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:15.618+0000 7fb22a33a700 1 -- 172.21.15.160:0/78653825 wait complete. 2023-12-25T20:13:15.619 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:13:16.468 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:13:16 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:13:14.680386+0000 mgr.smithi050.nbzhun (mgr.14162) 186 : cluster [DBG] pgmap v174: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:13:16.469 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:13:16 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:13:15.617274+0000 mon.smithi050 (mon.0) 254 : audit [DBG] from='client.? 172.21.15.160:0/78653825' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:13:17.025 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:13:17.025 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:13:18.274 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:18.274+0000 7f7dda9dd700 1 -- 172.21.15.160:0/3145898827 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7dd40f5410 msgr2=0x7f7dd40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:13:18.274 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:18.274+0000 7f7dda9dd700 1 --2- 172.21.15.160:0/3145898827 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7dd40f5410 0x7f7dd40f5830 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 rx=0x7f7dc4009fb0 tx=0x7f7dc4009b30).stop 2023-12-25T20:13:18.274 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:18.274+0000 7f7dda9dd700 1 -- 172.21.15.160:0/3145898827 shutdown_connections 2023-12-25T20:13:18.274 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:18.274+0000 7f7dda9dd700 1 --2- 172.21.15.160:0/3145898827 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7dd40f5410 0x7f7dd40f5830 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:13:18.274 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:18.274+0000 7f7dda9dd700 1 -- 172.21.15.160:0/3145898827 >> 172.21.15.160:0/3145898827 conn(0x7f7dd40f0b70 msgr2=0x7f7dd40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:13:18.275 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:18.274+0000 7f7dda9dd700 1 -- 172.21.15.160:0/3145898827 shutdown_connections 2023-12-25T20:13:18.275 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:18.274+0000 7f7dda9dd700 1 -- 172.21.15.160:0/3145898827 wait complete. 2023-12-25T20:13:18.275 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:18.274+0000 7f7dda9dd700 1 Processor -- start 2023-12-25T20:13:18.275 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:18.274+0000 7f7dda9dd700 1 -- start start 2023-12-25T20:13:18.275 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:18.275+0000 7f7dda9dd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7dd40f5410 0x7f7dd4105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:13:18.275 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:18.275+0000 7f7dda9dd700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dd4105b20 con 0x7f7dd40f5410 2023-12-25T20:13:18.277 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:18.275+0000 7f7dd3fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7dd40f5410 0x7f7dd4105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:13:18.277 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:18.275+0000 7f7dd3fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7dd40f5410 0x7f7dd4105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:47718/0 (socket says 172.21.15.160:47718) 2023-12-25T20:13:18.277 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:18.275+0000 7f7dd3fff700 1 -- 172.21.15.160:0/1957004636 learned_addr learned my addr 172.21.15.160:0/1957004636 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:13:18.277 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:18.275+0000 7f7dd3fff700 1 -- 172.21.15.160:0/1957004636 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dc4005040 con 0x7f7dd40f5410 2023-12-25T20:13:18.277 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:18.275+0000 7f7dd3fff700 1 --2- 172.21.15.160:0/1957004636 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7dd40f5410 0x7f7dd4105630 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7f7dc4013510 tx=0x7f7dc400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:13:18.277 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:18.276+0000 7f7dd17fa700 1 -- 172.21.15.160:0/1957004636 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7dc4009320 con 0x7f7dd40f5410 2023-12-25T20:13:18.277 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:18.276+0000 7f7dda9dd700 1 -- 172.21.15.160:0/1957004636 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7dd4105d20 con 0x7f7dd40f5410 2023-12-25T20:13:18.277 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:18.276+0000 7f7dda9dd700 1 -- 172.21.15.160:0/1957004636 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7dd41061b0 con 0x7f7dd40f5410 2023-12-25T20:13:18.277 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:18.276+0000 7f7dd17fa700 1 -- 172.21.15.160:0/1957004636 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7dc4013f00 con 0x7f7dd40f5410 2023-12-25T20:13:18.277 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:18.276+0000 7f7dd17fa700 1 -- 172.21.15.160:0/1957004636 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7dc4014d60 con 0x7f7dd40f5410 2023-12-25T20:13:18.277 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:18.276+0000 7f7dd17fa700 1 -- 172.21.15.160:0/1957004636 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7dc40095f0 con 0x7f7dd40f5410 2023-12-25T20:13:18.278 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:18.276+0000 7f7dda9dd700 1 -- 172.21.15.160:0/1957004636 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7dd4045ad0 con 0x7f7dd40f5410 2023-12-25T20:13:18.278 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:18.277+0000 7f7dd17fa700 1 --2- 172.21.15.160:0/1957004636 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f7dc002ffb0 0x7f7dc0032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:13:18.278 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:18.277+0000 7f7dd17fa700 1 -- 172.21.15.160:0/1957004636 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7dc4040f60 con 0x7f7dd40f5410 2023-12-25T20:13:18.278 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:18.277+0000 7f7dcbfff700 1 --2- 172.21.15.160:0/1957004636 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f7dc002ffb0 0x7f7dc0032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:13:18.278 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:18.277+0000 7f7dcbfff700 1 --2- 172.21.15.160:0/1957004636 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f7dc002ffb0 0x7f7dc0032470 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f7dbc0096a0 tx=0x7f7dbc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:13:18.281 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:18.282+0000 7f7dd17fa700 1 -- 172.21.15.160:0/1957004636 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7dc4014420 con 0x7f7dd40f5410 2023-12-25T20:13:18.468 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:13:18 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:13:16.680799+0000 mgr.smithi050.nbzhun (mgr.14162) 187 : cluster [DBG] pgmap v175: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:13:18.484 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:18.484+0000 7f7dda9dd700 1 -- 172.21.15.160:0/1957004636 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7dd40fae60 con 0x7f7dd40f5410 2023-12-25T20:13:18.485 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:18.485+0000 7f7dd17fa700 1 -- 172.21.15.160:0/1957004636 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7dc4010230 con 0x7f7dd40f5410 2023-12-25T20:13:18.485 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:13:18.485 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:13:18.487 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:18.487+0000 7f7dda9dd700 1 -- 172.21.15.160:0/1957004636 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f7dc002ffb0 msgr2=0x7f7dc0032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:13:18.487 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:18.487+0000 7f7dda9dd700 1 --2- 172.21.15.160:0/1957004636 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f7dc002ffb0 0x7f7dc0032470 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f7dbc0096a0 tx=0x7f7dbc006b40).stop 2023-12-25T20:13:18.487 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:18.487+0000 7f7dda9dd700 1 -- 172.21.15.160:0/1957004636 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7dd40f5410 msgr2=0x7f7dd4105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:13:18.487 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:18.487+0000 7f7dda9dd700 1 --2- 172.21.15.160:0/1957004636 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7dd40f5410 0x7f7dd4105630 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7f7dc4013510 tx=0x7f7dc400b040).stop 2023-12-25T20:13:18.487 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:18.487+0000 7f7dda9dd700 1 -- 172.21.15.160:0/1957004636 shutdown_connections 2023-12-25T20:13:18.487 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:18.487+0000 7f7dda9dd700 1 --2- 172.21.15.160:0/1957004636 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f7dc002ffb0 0x7f7dc0032470 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:13:18.487 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:18.487+0000 7f7dda9dd700 1 --2- 172.21.15.160:0/1957004636 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7dd40f5410 0x7f7dd4105630 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:13:18.487 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:18.487+0000 7f7dda9dd700 1 -- 172.21.15.160:0/1957004636 >> 172.21.15.160:0/1957004636 conn(0x7f7dd40f0b70 msgr2=0x7f7dd40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:13:18.487 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:18.487+0000 7f7dda9dd700 1 -- 172.21.15.160:0/1957004636 shutdown_connections 2023-12-25T20:13:18.487 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:18.487+0000 7f7dda9dd700 1 -- 172.21.15.160:0/1957004636 wait complete. 2023-12-25T20:13:18.488 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:13:19.468 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:13:19 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:13:18.485816+0000 mon.smithi050 (mon.0) 255 : audit [DBG] from='client.? 172.21.15.160:0/1957004636' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:13:19.914 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:13:19.914 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:13:20.468 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:13:20 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:13:18.681147+0000 mgr.smithi050.nbzhun (mgr.14162) 188 : cluster [DBG] pgmap v176: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:13:21.122 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:21.121+0000 7f7d86c3d700 1 -- 172.21.15.160:0/3935171372 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7d800f6510 msgr2=0x7f7d800f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:13:21.122 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:21.121+0000 7f7d86c3d700 1 --2- 172.21.15.160:0/3935171372 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7d800f6510 0x7f7d800f6930 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 rx=0x7f7d7c009fb0 tx=0x7f7d7c009b30).stop 2023-12-25T20:13:21.122 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:21.121+0000 7f7d86c3d700 1 -- 172.21.15.160:0/3935171372 shutdown_connections 2023-12-25T20:13:21.122 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:21.121+0000 7f7d86c3d700 1 --2- 172.21.15.160:0/3935171372 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7d800f6510 0x7f7d800f6930 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:13:21.122 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:21.121+0000 7f7d86c3d700 1 -- 172.21.15.160:0/3935171372 >> 172.21.15.160:0/3935171372 conn(0x7f7d800f1c70 msgr2=0x7f7d800f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:13:21.122 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:21.121+0000 7f7d86c3d700 1 -- 172.21.15.160:0/3935171372 shutdown_connections 2023-12-25T20:13:21.122 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:21.121+0000 7f7d86c3d700 1 -- 172.21.15.160:0/3935171372 wait complete. 2023-12-25T20:13:21.122 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:21.122+0000 7f7d86c3d700 1 Processor -- start 2023-12-25T20:13:21.122 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:21.122+0000 7f7d86c3d700 1 -- start start 2023-12-25T20:13:21.122 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:21.122+0000 7f7d86c3d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7d800f6510 0x7f7d8006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:13:21.123 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:21.122+0000 7f7d86c3d700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d80069e20 con 0x7f7d800f6510 2023-12-25T20:13:21.123 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:21.122+0000 7f7d849d9700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7d800f6510 0x7f7d8006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:13:21.123 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:21.122+0000 7f7d849d9700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7d800f6510 0x7f7d8006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:47730/0 (socket says 172.21.15.160:47730) 2023-12-25T20:13:21.123 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:21.122+0000 7f7d849d9700 1 -- 172.21.15.160:0/3028616950 learned_addr learned my addr 172.21.15.160:0/3028616950 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:13:21.124 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:21.123+0000 7f7d849d9700 1 -- 172.21.15.160:0/3028616950 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d7c005040 con 0x7f7d800f6510 2023-12-25T20:13:21.124 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:21.123+0000 7f7d849d9700 1 --2- 172.21.15.160:0/3028616950 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7d800f6510 0x7f7d8006b9a0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7f7d7c013510 tx=0x7f7d7c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:13:21.124 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:21.123+0000 7f7d757fa700 1 -- 172.21.15.160:0/3028616950 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7d7c009320 con 0x7f7d800f6510 2023-12-25T20:13:21.124 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:21.123+0000 7f7d86c3d700 1 -- 172.21.15.160:0/3028616950 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d80069fc0 con 0x7f7d800f6510 2023-12-25T20:13:21.124 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:21.123+0000 7f7d757fa700 1 -- 172.21.15.160:0/3028616950 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7d7c013f00 con 0x7f7d800f6510 2023-12-25T20:13:21.124 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:21.123+0000 7f7d757fa700 1 -- 172.21.15.160:0/3028616950 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7d7c014d90 con 0x7f7d800f6510 2023-12-25T20:13:21.125 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:21.123+0000 7f7d86c3d700 1 -- 172.21.15.160:0/3028616950 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d8006a450 con 0x7f7d800f6510 2023-12-25T20:13:21.125 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:21.124+0000 7f7d757fa700 1 -- 172.21.15.160:0/3028616950 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7d7c0263a0 con 0x7f7d800f6510 2023-12-25T20:13:21.125 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:21.124+0000 7f7d757fa700 1 --2- 172.21.15.160:0/3028616950 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f7d6802ff60 0x7f7d68032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:13:21.125 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:21.124+0000 7f7d757fa700 1 -- 172.21.15.160:0/3028616950 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7d7c041f30 con 0x7f7d800f6510 2023-12-25T20:13:21.125 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:21.124+0000 7f7d86c3d700 1 -- 172.21.15.160:0/3028616950 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d80059100 con 0x7f7d800f6510 2023-12-25T20:13:21.125 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:21.124+0000 7f7d77fff700 1 --2- 172.21.15.160:0/3028616950 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f7d6802ff60 0x7f7d68032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:13:21.125 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:21.125+0000 7f7d77fff700 1 --2- 172.21.15.160:0/3028616950 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f7d6802ff60 0x7f7d68032420 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f7d700096a0 tx=0x7f7d70006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:13:21.128 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:21.129+0000 7f7d757fa700 1 -- 172.21.15.160:0/3028616950 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7d7c012920 con 0x7f7d800f6510 2023-12-25T20:13:21.334 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:21.334+0000 7f7d86c3d700 1 -- 172.21.15.160:0/3028616950 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7d8006ab50 con 0x7f7d800f6510 2023-12-25T20:13:21.334 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:21.335+0000 7f7d757fa700 1 -- 172.21.15.160:0/3028616950 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7d7c02b020 con 0x7f7d800f6510 2023-12-25T20:13:21.335 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:13:21.335 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:13:21.336 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:21.336+0000 7f7d86c3d700 1 -- 172.21.15.160:0/3028616950 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f7d6802ff60 msgr2=0x7f7d68032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:13:21.336 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:21.336+0000 7f7d86c3d700 1 --2- 172.21.15.160:0/3028616950 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f7d6802ff60 0x7f7d68032420 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f7d700096a0 tx=0x7f7d70006b40).stop 2023-12-25T20:13:21.336 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:21.336+0000 7f7d86c3d700 1 -- 172.21.15.160:0/3028616950 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7d800f6510 msgr2=0x7f7d8006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:13:21.336 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:21.336+0000 7f7d86c3d700 1 --2- 172.21.15.160:0/3028616950 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7d800f6510 0x7f7d8006b9a0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7f7d7c013510 tx=0x7f7d7c00b040).stop 2023-12-25T20:13:21.336 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:21.336+0000 7f7d86c3d700 1 -- 172.21.15.160:0/3028616950 shutdown_connections 2023-12-25T20:13:21.336 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:21.336+0000 7f7d86c3d700 1 --2- 172.21.15.160:0/3028616950 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f7d6802ff60 0x7f7d68032420 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:13:21.336 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:21.336+0000 7f7d86c3d700 1 --2- 172.21.15.160:0/3028616950 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7d800f6510 0x7f7d8006b9a0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:13:21.337 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:21.336+0000 7f7d86c3d700 1 -- 172.21.15.160:0/3028616950 >> 172.21.15.160:0/3028616950 conn(0x7f7d800f1c70 msgr2=0x7f7d800f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:13:21.337 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:21.336+0000 7f7d86c3d700 1 -- 172.21.15.160:0/3028616950 shutdown_connections 2023-12-25T20:13:21.337 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:21.336+0000 7f7d86c3d700 1 -- 172.21.15.160:0/3028616950 wait complete. 2023-12-25T20:13:21.337 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:13:22.468 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:13:22 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:13:20.681507+0000 mgr.smithi050.nbzhun (mgr.14162) 189 : cluster [DBG] pgmap v177: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:13:22.469 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:13:22 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:13:21.335519+0000 mon.smithi050 (mon.0) 256 : audit [DBG] from='client.? 172.21.15.160:0/3028616950' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:13:22.730 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:13:22.730 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:13:23.895 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:23.894+0000 7f7299cf0700 1 -- 172.21.15.160:0/2416246530 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7294060220 msgr2=0x7f72940f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:13:23.895 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:23.894+0000 7f7299cf0700 1 --2- 172.21.15.160:0/2416246530 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7294060220 0x7f72940f9150 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 rx=0x7f7284009fb0 tx=0x7f7284009b30).stop 2023-12-25T20:13:23.895 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:23.895+0000 7f7299cf0700 1 -- 172.21.15.160:0/2416246530 shutdown_connections 2023-12-25T20:13:23.895 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:23.895+0000 7f7299cf0700 1 --2- 172.21.15.160:0/2416246530 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7294060220 0x7f72940f9150 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:13:23.895 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:23.895+0000 7f7299cf0700 1 -- 172.21.15.160:0/2416246530 >> 172.21.15.160:0/2416246530 conn(0x7f72940f0b70 msgr2=0x7f72940f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:13:23.895 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:23.895+0000 7f7299cf0700 1 -- 172.21.15.160:0/2416246530 shutdown_connections 2023-12-25T20:13:23.895 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:23.895+0000 7f7299cf0700 1 -- 172.21.15.160:0/2416246530 wait complete. 2023-12-25T20:13:23.895 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:23.895+0000 7f7299cf0700 1 Processor -- start 2023-12-25T20:13:23.895 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:23.895+0000 7f7299cf0700 1 -- start start 2023-12-25T20:13:23.897 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:23.896+0000 7f7299cf0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7294060220 0x7f72941066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:13:23.897 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:23.896+0000 7f7299cf0700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7294106bd0 con 0x7f7294060220 2023-12-25T20:13:23.897 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:23.896+0000 7f72937fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7294060220 0x7f72941066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:13:23.898 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:23.896+0000 7f72937fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7294060220 0x7f72941066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:47744/0 (socket says 172.21.15.160:47744) 2023-12-25T20:13:23.898 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:23.896+0000 7f72937fe700 1 -- 172.21.15.160:0/3251384230 learned_addr learned my addr 172.21.15.160:0/3251384230 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:13:23.898 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:23.896+0000 7f72937fe700 1 -- 172.21.15.160:0/3251384230 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7284005040 con 0x7f7294060220 2023-12-25T20:13:23.898 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:23.897+0000 7f72937fe700 1 --2- 172.21.15.160:0/3251384230 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7294060220 0x7f72941066e0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7f7284013510 tx=0x7f728400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:13:23.898 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:23.897+0000 7f728bfff700 1 -- 172.21.15.160:0/3251384230 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7284009320 con 0x7f7294060220 2023-12-25T20:13:23.898 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:23.897+0000 7f7299cf0700 1 -- 172.21.15.160:0/3251384230 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7294106dd0 con 0x7f7294060220 2023-12-25T20:13:23.898 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:23.897+0000 7f728bfff700 1 -- 172.21.15.160:0/3251384230 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7284013f00 con 0x7f7294060220 2023-12-25T20:13:23.898 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:23.897+0000 7f728bfff700 1 -- 172.21.15.160:0/3251384230 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7284014dc0 con 0x7f7294060220 2023-12-25T20:13:23.898 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:23.897+0000 7f7299cf0700 1 -- 172.21.15.160:0/3251384230 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7294107260 con 0x7f7294060220 2023-12-25T20:13:23.898 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:23.897+0000 7f728bfff700 1 -- 172.21.15.160:0/3251384230 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7284014420 con 0x7f7294060220 2023-12-25T20:13:23.898 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:23.898+0000 7f7299cf0700 1 -- 172.21.15.160:0/3251384230 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7294045ad0 con 0x7f7294060220 2023-12-25T20:13:23.898 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:23.898+0000 7f728bfff700 1 --2- 172.21.15.160:0/3251384230 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f728002ff60 0x7f7280032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:13:23.898 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:23.898+0000 7f728bfff700 1 -- 172.21.15.160:0/3251384230 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7284041050 con 0x7f7294060220 2023-12-25T20:13:23.898 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:23.898+0000 7f7292ffd700 1 --2- 172.21.15.160:0/3251384230 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f728002ff60 0x7f7280032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:13:23.899 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:23.898+0000 7f7292ffd700 1 --2- 172.21.15.160:0/3251384230 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f728002ff60 0x7f7280032420 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f727c0096a0 tx=0x7f727c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:13:23.902 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:23.903+0000 7f728bfff700 1 -- 172.21.15.160:0/3251384230 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f728401cb10 con 0x7f7294060220 2023-12-25T20:13:24.107 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:24.107+0000 7f7299cf0700 1 -- 172.21.15.160:0/3251384230 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7294108110 con 0x7f7294060220 2023-12-25T20:13:24.108 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:24.108+0000 7f728bfff700 1 -- 172.21.15.160:0/3251384230 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f728400f750 con 0x7f7294060220 2023-12-25T20:13:24.108 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:13:24.108 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:13:24.109 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:24.110+0000 7f7299cf0700 1 -- 172.21.15.160:0/3251384230 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f728002ff60 msgr2=0x7f7280032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:13:24.110 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:24.110+0000 7f7299cf0700 1 --2- 172.21.15.160:0/3251384230 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f728002ff60 0x7f7280032420 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f727c0096a0 tx=0x7f727c006b40).stop 2023-12-25T20:13:24.110 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:24.110+0000 7f7299cf0700 1 -- 172.21.15.160:0/3251384230 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7294060220 msgr2=0x7f72941066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:13:24.110 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:24.110+0000 7f7299cf0700 1 --2- 172.21.15.160:0/3251384230 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7294060220 0x7f72941066e0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7f7284013510 tx=0x7f728400b040).stop 2023-12-25T20:13:24.110 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:24.110+0000 7f7299cf0700 1 -- 172.21.15.160:0/3251384230 shutdown_connections 2023-12-25T20:13:24.110 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:24.110+0000 7f7299cf0700 1 --2- 172.21.15.160:0/3251384230 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f728002ff60 0x7f7280032420 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:13:24.110 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:24.110+0000 7f7299cf0700 1 --2- 172.21.15.160:0/3251384230 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7294060220 0x7f72941066e0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:13:24.110 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:24.110+0000 7f7299cf0700 1 -- 172.21.15.160:0/3251384230 >> 172.21.15.160:0/3251384230 conn(0x7f72940f0b70 msgr2=0x7f72940f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:13:24.110 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:24.110+0000 7f7299cf0700 1 -- 172.21.15.160:0/3251384230 shutdown_connections 2023-12-25T20:13:24.110 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:24.110+0000 7f7299cf0700 1 -- 172.21.15.160:0/3251384230 wait complete. 2023-12-25T20:13:24.110 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:13:24.468 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:13:24 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:13:22.681809+0000 mgr.smithi050.nbzhun (mgr.14162) 190 : cluster [DBG] pgmap v178: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:13:24.469 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:13:24 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:13:24.108965+0000 mon.smithi050 (mon.0) 257 : audit [DBG] from='client.? 172.21.15.160:0/3251384230' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:13:25.515 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:13:25.515 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:13:26.469 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:13:26 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:13:24.682170+0000 mgr.smithi050.nbzhun (mgr.14162) 191 : cluster [DBG] pgmap v179: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:13:26.657 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:26.649+0000 7f74b31b2700 1 -- 172.21.15.160:0/1413299390 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f74ac0f6510 msgr2=0x7f74ac0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:13:26.657 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:26.649+0000 7f74b31b2700 1 --2- 172.21.15.160:0/1413299390 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f74ac0f6510 0x7f74ac0f6930 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 rx=0x7f74a0009fb0 tx=0x7f74a0009b30).stop 2023-12-25T20:13:26.657 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:26.649+0000 7f74b31b2700 1 -- 172.21.15.160:0/1413299390 shutdown_connections 2023-12-25T20:13:26.657 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:26.649+0000 7f74b31b2700 1 --2- 172.21.15.160:0/1413299390 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f74ac0f6510 0x7f74ac0f6930 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:13:26.657 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:26.649+0000 7f74b31b2700 1 -- 172.21.15.160:0/1413299390 >> 172.21.15.160:0/1413299390 conn(0x7f74ac0f1c70 msgr2=0x7f74ac0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:13:26.657 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:26.649+0000 7f74b31b2700 1 -- 172.21.15.160:0/1413299390 shutdown_connections 2023-12-25T20:13:26.657 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:26.649+0000 7f74b31b2700 1 -- 172.21.15.160:0/1413299390 wait complete. 2023-12-25T20:13:26.657 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:26.650+0000 7f74b31b2700 1 Processor -- start 2023-12-25T20:13:26.657 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:26.650+0000 7f74b31b2700 1 -- start start 2023-12-25T20:13:26.658 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:26.650+0000 7f74b31b2700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f74ac0f6510 0x7f74ac106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:13:26.658 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:26.650+0000 7f74b31b2700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74ac106c20 con 0x7f74ac0f6510 2023-12-25T20:13:26.659 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:26.650+0000 7f74b0f4e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f74ac0f6510 0x7f74ac106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:13:26.659 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:26.650+0000 7f74b0f4e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f74ac0f6510 0x7f74ac106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:47758/0 (socket says 172.21.15.160:47758) 2023-12-25T20:13:26.659 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:26.650+0000 7f74b0f4e700 1 -- 172.21.15.160:0/3066141241 learned_addr learned my addr 172.21.15.160:0/3066141241 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:13:26.659 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:26.650+0000 7f74b0f4e700 1 -- 172.21.15.160:0/3066141241 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74a0005040 con 0x7f74ac0f6510 2023-12-25T20:13:26.659 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:26.650+0000 7f74b0f4e700 1 --2- 172.21.15.160:0/3066141241 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f74ac0f6510 0x7f74ac106730 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7f74a0013510 tx=0x7f74a000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:13:26.659 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:26.651+0000 7f74a97fa700 1 -- 172.21.15.160:0/3066141241 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f74a0009320 con 0x7f74ac0f6510 2023-12-25T20:13:26.659 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:26.651+0000 7f74b31b2700 1 -- 172.21.15.160:0/3066141241 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74ac106e20 con 0x7f74ac0f6510 2023-12-25T20:13:26.659 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:26.651+0000 7f74a97fa700 1 -- 172.21.15.160:0/3066141241 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f74a0013f00 con 0x7f74ac0f6510 2023-12-25T20:13:26.659 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:26.651+0000 7f74a97fa700 1 -- 172.21.15.160:0/3066141241 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f74a0014d90 con 0x7f74ac0f6510 2023-12-25T20:13:26.662 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:26.651+0000 7f74b31b2700 1 -- 172.21.15.160:0/3066141241 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74ac1072b0 con 0x7f74ac0f6510 2023-12-25T20:13:26.662 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:26.652+0000 7f74a97fa700 1 -- 172.21.15.160:0/3066141241 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f74a00095f0 con 0x7f74ac0f6510 2023-12-25T20:13:26.662 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:26.652+0000 7f74b31b2700 1 -- 172.21.15.160:0/3066141241 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74ac045ad0 con 0x7f74ac0f6510 2023-12-25T20:13:26.662 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:26.652+0000 7f74a97fa700 1 --2- 172.21.15.160:0/3066141241 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f749c02ff60 0x7f749c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:13:26.663 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:26.652+0000 7f74a97fa700 1 -- 172.21.15.160:0/3066141241 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f74a0041000 con 0x7f74ac0f6510 2023-12-25T20:13:26.663 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:26.652+0000 7f74abfff700 1 --2- 172.21.15.160:0/3066141241 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f749c02ff60 0x7f749c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:13:26.663 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:26.653+0000 7f74abfff700 1 --2- 172.21.15.160:0/3066141241 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f749c02ff60 0x7f749c032420 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f74980096a0 tx=0x7f7498006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:13:26.663 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:26.657+0000 7f74a97fa700 1 -- 172.21.15.160:0/3066141241 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f74a0014420 con 0x7f74ac0f6510 2023-12-25T20:13:26.864 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:26.862+0000 7f74b31b2700 1 -- 172.21.15.160:0/3066141241 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f74ac0fbf60 con 0x7f74ac0f6510 2023-12-25T20:13:26.865 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:26.863+0000 7f74a97fa700 1 -- 172.21.15.160:0/3066141241 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f74a0010230 con 0x7f74ac0f6510 2023-12-25T20:13:26.865 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:13:26.865 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:13:26.866 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:26.864+0000 7f74b31b2700 1 -- 172.21.15.160:0/3066141241 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f749c02ff60 msgr2=0x7f749c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:13:26.866 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:26.864+0000 7f74b31b2700 1 --2- 172.21.15.160:0/3066141241 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f749c02ff60 0x7f749c032420 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f74980096a0 tx=0x7f7498006b40).stop 2023-12-25T20:13:26.866 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:26.864+0000 7f74b31b2700 1 -- 172.21.15.160:0/3066141241 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f74ac0f6510 msgr2=0x7f74ac106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:13:26.866 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:26.864+0000 7f74b31b2700 1 --2- 172.21.15.160:0/3066141241 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f74ac0f6510 0x7f74ac106730 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7f74a0013510 tx=0x7f74a000b040).stop 2023-12-25T20:13:26.866 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:26.865+0000 7f74b31b2700 1 -- 172.21.15.160:0/3066141241 shutdown_connections 2023-12-25T20:13:26.866 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:26.865+0000 7f74b31b2700 1 --2- 172.21.15.160:0/3066141241 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f749c02ff60 0x7f749c032420 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:13:26.866 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:26.865+0000 7f74b31b2700 1 --2- 172.21.15.160:0/3066141241 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f74ac0f6510 0x7f74ac106730 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:13:26.867 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:26.865+0000 7f74b31b2700 1 -- 172.21.15.160:0/3066141241 >> 172.21.15.160:0/3066141241 conn(0x7f74ac0f1c70 msgr2=0x7f74ac0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:13:26.867 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:26.865+0000 7f74b31b2700 1 -- 172.21.15.160:0/3066141241 shutdown_connections 2023-12-25T20:13:26.867 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:26.865+0000 7f74b31b2700 1 -- 172.21.15.160:0/3066141241 wait complete. 2023-12-25T20:13:26.867 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:13:27.468 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:13:27 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:13:26.863790+0000 mon.smithi050 (mon.0) 258 : audit [DBG] from='client.? 172.21.15.160:0/3066141241' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:13:28.260 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:13:28.261 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:13:28.469 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:13:28 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:13:26.682417+0000 mgr.smithi050.nbzhun (mgr.14162) 192 : cluster [DBG] pgmap v180: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:13:29.506 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:29.506+0000 7fb38359e700 1 -- 172.21.15.160:0/3577143905 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb3840f51a0 msgr2=0x7fb3840f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:13:29.506 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:29.506+0000 7fb38359e700 1 --2- 172.21.15.160:0/3577143905 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb3840f51a0 0x7fb3840f55c0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 rx=0x7fb36c009fb0 tx=0x7fb36c009b30).stop 2023-12-25T20:13:29.506 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:29.506+0000 7fb38359e700 1 -- 172.21.15.160:0/3577143905 shutdown_connections 2023-12-25T20:13:29.506 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:29.506+0000 7fb38359e700 1 --2- 172.21.15.160:0/3577143905 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb3840f51a0 0x7fb3840f55c0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:13:29.507 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:29.506+0000 7fb38359e700 1 -- 172.21.15.160:0/3577143905 >> 172.21.15.160:0/3577143905 conn(0x7fb3840f09a0 msgr2=0x7fb3840f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:13:29.507 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:29.506+0000 7fb38359e700 1 -- 172.21.15.160:0/3577143905 shutdown_connections 2023-12-25T20:13:29.507 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:29.506+0000 7fb38359e700 1 -- 172.21.15.160:0/3577143905 wait complete. 2023-12-25T20:13:29.507 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:29.506+0000 7fb38359e700 1 Processor -- start 2023-12-25T20:13:29.507 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:29.506+0000 7fb38359e700 1 -- start start 2023-12-25T20:13:29.507 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:29.507+0000 7fb38359e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb3840f51a0 0x7fb384109170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:13:29.507 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:29.507+0000 7fb38359e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb384060280 con 0x7fb3840f51a0 2023-12-25T20:13:29.509 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:29.507+0000 7fb38259c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb3840f51a0 0x7fb384109170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:13:29.509 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:29.507+0000 7fb38259c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb3840f51a0 0x7fb384109170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:54328/0 (socket says 172.21.15.160:54328) 2023-12-25T20:13:29.509 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:29.507+0000 7fb38259c700 1 -- 172.21.15.160:0/3406525567 learned_addr learned my addr 172.21.15.160:0/3406525567 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:13:29.509 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:29.507+0000 7fb38259c700 1 -- 172.21.15.160:0/3406525567 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb36c005040 con 0x7fb3840f51a0 2023-12-25T20:13:29.509 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:29.507+0000 7fb38259c700 1 --2- 172.21.15.160:0/3406525567 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb3840f51a0 0x7fb384109170 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7fb36c013510 tx=0x7fb36c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:13:29.509 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:29.507+0000 7fb37affd700 1 -- 172.21.15.160:0/3406525567 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb36c009320 con 0x7fb3840f51a0 2023-12-25T20:13:29.509 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:29.508+0000 7fb38359e700 1 -- 172.21.15.160:0/3406525567 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb384060480 con 0x7fb3840f51a0 2023-12-25T20:13:29.509 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:29.508+0000 7fb37affd700 1 -- 172.21.15.160:0/3406525567 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb36c013f00 con 0x7fb3840f51a0 2023-12-25T20:13:29.509 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:29.508+0000 7fb37affd700 1 -- 172.21.15.160:0/3406525567 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb36c014dc0 con 0x7fb3840f51a0 2023-12-25T20:13:29.509 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:29.508+0000 7fb38359e700 1 -- 172.21.15.160:0/3406525567 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3841098a0 con 0x7fb3840f51a0 2023-12-25T20:13:29.510 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:29.508+0000 7fb37affd700 1 -- 172.21.15.160:0/3406525567 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb36c0095f0 con 0x7fb3840f51a0 2023-12-25T20:13:29.510 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:29.508+0000 7fb38359e700 1 -- 172.21.15.160:0/3406525567 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb384045ad0 con 0x7fb3840f51a0 2023-12-25T20:13:29.510 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:29.508+0000 7fb37affd700 1 --2- 172.21.15.160:0/3406525567 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fb37002ff60 0x7fb370032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:13:29.510 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:29.509+0000 7fb37affd700 1 -- 172.21.15.160:0/3406525567 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb36c041000 con 0x7fb3840f51a0 2023-12-25T20:13:29.510 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:29.509+0000 7fb381d9b700 1 --2- 172.21.15.160:0/3406525567 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fb37002ff60 0x7fb370032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:13:29.510 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:29.509+0000 7fb381d9b700 1 --2- 172.21.15.160:0/3406525567 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fb37002ff60 0x7fb370032420 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7fb3740096a0 tx=0x7fb374006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:13:29.513 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:29.513+0000 7fb37affd700 1 -- 172.21.15.160:0/3406525567 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb36c01c790 con 0x7fb3840f51a0 2023-12-25T20:13:29.717 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:29.717+0000 7fb38359e700 1 -- 172.21.15.160:0/3406525567 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb3840fabe0 con 0x7fb3840f51a0 2023-12-25T20:13:29.717 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:29.718+0000 7fb37affd700 1 -- 172.21.15.160:0/3406525567 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb36c010230 con 0x7fb3840f51a0 2023-12-25T20:13:29.718 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:13:29.718 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:13:29.719 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:29.719+0000 7fb38359e700 1 -- 172.21.15.160:0/3406525567 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fb37002ff60 msgr2=0x7fb370032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:13:29.719 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:29.719+0000 7fb38359e700 1 --2- 172.21.15.160:0/3406525567 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fb37002ff60 0x7fb370032420 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7fb3740096a0 tx=0x7fb374006b40).stop 2023-12-25T20:13:29.719 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:29.720+0000 7fb38359e700 1 -- 172.21.15.160:0/3406525567 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb3840f51a0 msgr2=0x7fb384109170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:13:29.720 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:29.720+0000 7fb38359e700 1 --2- 172.21.15.160:0/3406525567 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb3840f51a0 0x7fb384109170 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7fb36c013510 tx=0x7fb36c00b040).stop 2023-12-25T20:13:29.720 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:29.720+0000 7fb38359e700 1 -- 172.21.15.160:0/3406525567 shutdown_connections 2023-12-25T20:13:29.720 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:29.720+0000 7fb38359e700 1 --2- 172.21.15.160:0/3406525567 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fb37002ff60 0x7fb370032420 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:13:29.720 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:29.720+0000 7fb38359e700 1 --2- 172.21.15.160:0/3406525567 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb3840f51a0 0x7fb384109170 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:13:29.720 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:29.720+0000 7fb38359e700 1 -- 172.21.15.160:0/3406525567 >> 172.21.15.160:0/3406525567 conn(0x7fb3840f09a0 msgr2=0x7fb3840f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:13:29.720 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:29.720+0000 7fb38359e700 1 -- 172.21.15.160:0/3406525567 shutdown_connections 2023-12-25T20:13:29.720 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:29.720+0000 7fb38359e700 1 -- 172.21.15.160:0/3406525567 wait complete. 2023-12-25T20:13:29.720 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:13:30.468 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:13:30 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:13:28.682774+0000 mgr.smithi050.nbzhun (mgr.14162) 193 : cluster [DBG] pgmap v181: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:13:30.468 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:13:30 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:13:29.718556+0000 mon.smithi050 (mon.0) 259 : audit [DBG] from='client.? 172.21.15.160:0/3406525567' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:13:31.123 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:13:31.123 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:13:32.320 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:32.319+0000 7f140faf3700 1 -- 172.21.15.160:0/3491447586 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f14080f4740 msgr2=0x7f14080f4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:13:32.320 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:32.319+0000 7f140faf3700 1 --2- 172.21.15.160:0/3491447586 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f14080f4740 0x7f14080f4b60 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 rx=0x7f13f8009fb0 tx=0x7f13f8009b30).stop 2023-12-25T20:13:32.320 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:32.319+0000 7f140faf3700 1 -- 172.21.15.160:0/3491447586 shutdown_connections 2023-12-25T20:13:32.320 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:32.319+0000 7f140faf3700 1 --2- 172.21.15.160:0/3491447586 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f14080f4740 0x7f14080f4b60 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:13:32.321 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:32.319+0000 7f140faf3700 1 -- 172.21.15.160:0/3491447586 >> 172.21.15.160:0/3491447586 conn(0x7f14080f0260 msgr2=0x7f14080f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:13:32.321 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:32.319+0000 7f140faf3700 1 -- 172.21.15.160:0/3491447586 shutdown_connections 2023-12-25T20:13:32.321 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:32.319+0000 7f140faf3700 1 -- 172.21.15.160:0/3491447586 wait complete. 2023-12-25T20:13:32.321 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:32.320+0000 7f140faf3700 1 Processor -- start 2023-12-25T20:13:32.321 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:32.320+0000 7f140faf3700 1 -- start start 2023-12-25T20:13:32.321 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:32.320+0000 7f140faf3700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f14080f4740 0x7f14080f72f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:13:32.321 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:32.320+0000 7f140faf3700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14080f5770 con 0x7f14080f4740 2023-12-25T20:13:32.321 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:32.320+0000 7f140d88f700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f14080f4740 0x7f14080f72f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:13:32.321 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:32.320+0000 7f140d88f700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f14080f4740 0x7f14080f72f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:54336/0 (socket says 172.21.15.160:54336) 2023-12-25T20:13:32.321 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:32.320+0000 7f140d88f700 1 -- 172.21.15.160:0/500871465 learned_addr learned my addr 172.21.15.160:0/500871465 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:13:32.321 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:32.320+0000 7f140d88f700 1 -- 172.21.15.160:0/500871465 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13f8005040 con 0x7f14080f4740 2023-12-25T20:13:32.321 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:32.320+0000 7f140d88f700 1 --2- 172.21.15.160:0/500871465 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f14080f4740 0x7f14080f72f0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7f13f8013510 tx=0x7f13f800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:13:32.321 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:32.321+0000 7f13fe7fc700 1 -- 172.21.15.160:0/500871465 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f13f8009320 con 0x7f14080f4740 2023-12-25T20:13:32.322 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:32.321+0000 7f140faf3700 1 -- 172.21.15.160:0/500871465 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14080f5910 con 0x7f14080f4740 2023-12-25T20:13:32.322 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:32.321+0000 7f13fe7fc700 1 -- 172.21.15.160:0/500871465 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f13f8013f00 con 0x7f14080f4740 2023-12-25T20:13:32.322 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:32.321+0000 7f13fe7fc700 1 -- 172.21.15.160:0/500871465 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f13f8014d90 con 0x7f14080f4740 2023-12-25T20:13:32.323 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:32.321+0000 7f140faf3700 1 -- 172.21.15.160:0/500871465 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14080f5da0 con 0x7f14080f4740 2023-12-25T20:13:32.323 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:32.321+0000 7f13fe7fc700 1 -- 172.21.15.160:0/500871465 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f13f80263a0 con 0x7f14080f4740 2023-12-25T20:13:32.323 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:32.322+0000 7f13fe7fc700 1 --2- 172.21.15.160:0/500871465 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f13f402ff60 0x7f13f4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:13:32.323 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:32.322+0000 7f13fe7fc700 1 -- 172.21.15.160:0/500871465 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f13f80410d0 con 0x7f14080f4740 2023-12-25T20:13:32.323 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:32.322+0000 7f140faf3700 1 -- 172.21.15.160:0/500871465 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1408059100 con 0x7f14080f4740 2023-12-25T20:13:32.323 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:32.322+0000 7f140d08e700 1 --2- 172.21.15.160:0/500871465 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f13f402ff60 0x7f13f4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:13:32.323 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:32.322+0000 7f140d08e700 1 --2- 172.21.15.160:0/500871465 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f13f402ff60 0x7f13f4032420 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f14040096a0 tx=0x7f1404006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:13:32.326 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:32.327+0000 7f13fe7fc700 1 -- 172.21.15.160:0/500871465 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f13f8012920 con 0x7f14080f4740 2023-12-25T20:13:32.468 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:13:32 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:13:30.683176+0000 mgr.smithi050.nbzhun (mgr.14162) 194 : cluster [DBG] pgmap v182: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:13:32.532 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:32.532+0000 7f140faf3700 1 -- 172.21.15.160:0/500871465 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1408045ad0 con 0x7f14080f4740 2023-12-25T20:13:32.532 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:32.533+0000 7f13fe7fc700 1 -- 172.21.15.160:0/500871465 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f13f802b020 con 0x7f14080f4740 2023-12-25T20:13:32.533 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:13:32.533 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:13:32.534 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:32.534+0000 7f140faf3700 1 -- 172.21.15.160:0/500871465 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f13f402ff60 msgr2=0x7f13f4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:13:32.534 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:32.534+0000 7f140faf3700 1 --2- 172.21.15.160:0/500871465 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f13f402ff60 0x7f13f4032420 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f14040096a0 tx=0x7f1404006b40).stop 2023-12-25T20:13:32.534 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:32.534+0000 7f140faf3700 1 -- 172.21.15.160:0/500871465 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f14080f4740 msgr2=0x7f14080f72f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:13:32.534 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:32.534+0000 7f140faf3700 1 --2- 172.21.15.160:0/500871465 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f14080f4740 0x7f14080f72f0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7f13f8013510 tx=0x7f13f800b040).stop 2023-12-25T20:13:32.534 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:32.534+0000 7f140faf3700 1 -- 172.21.15.160:0/500871465 shutdown_connections 2023-12-25T20:13:32.534 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:32.534+0000 7f140faf3700 1 --2- 172.21.15.160:0/500871465 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f13f402ff60 0x7f13f4032420 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:13:32.535 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:32.534+0000 7f140faf3700 1 --2- 172.21.15.160:0/500871465 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f14080f4740 0x7f14080f72f0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:13:32.535 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:32.534+0000 7f140faf3700 1 -- 172.21.15.160:0/500871465 >> 172.21.15.160:0/500871465 conn(0x7f14080f0260 msgr2=0x7f14080f0d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:13:32.535 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:32.534+0000 7f140faf3700 1 -- 172.21.15.160:0/500871465 shutdown_connections 2023-12-25T20:13:32.535 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:32.534+0000 7f140faf3700 1 -- 172.21.15.160:0/500871465 wait complete. 2023-12-25T20:13:32.535 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:13:33.468 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:13:33 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:13:32.533695+0000 mon.smithi050 (mon.0) 260 : audit [DBG] from='client.? 172.21.15.160:0/500871465' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:13:33.926 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:13:33.926 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:13:34.468 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:13:34 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:13:32.683463+0000 mgr.smithi050.nbzhun (mgr.14162) 195 : cluster [DBG] pgmap v183: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:13:35.131 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:35.131+0000 7fc07e83a700 1 -- 172.21.15.160:0/4035268599 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc0780fcb40 msgr2=0x7fc0780fcf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:13:35.132 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:35.131+0000 7fc07e83a700 1 --2- 172.21.15.160:0/4035268599 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc0780fcb40 0x7fc0780fcf60 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 rx=0x7fc074009fb0 tx=0x7fc074009b30).stop 2023-12-25T20:13:35.132 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:35.131+0000 7fc07e83a700 1 -- 172.21.15.160:0/4035268599 shutdown_connections 2023-12-25T20:13:35.132 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:35.131+0000 7fc07e83a700 1 --2- 172.21.15.160:0/4035268599 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc0780fcb40 0x7fc0780fcf60 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:13:35.132 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:35.131+0000 7fc07e83a700 1 -- 172.21.15.160:0/4035268599 >> 172.21.15.160:0/4035268599 conn(0x7fc07806bad0 msgr2=0x7fc07806def0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:13:35.132 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:35.131+0000 7fc07e83a700 1 -- 172.21.15.160:0/4035268599 shutdown_connections 2023-12-25T20:13:35.132 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:35.131+0000 7fc07e83a700 1 -- 172.21.15.160:0/4035268599 wait complete. 2023-12-25T20:13:35.132 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:35.131+0000 7fc07e83a700 1 Processor -- start 2023-12-25T20:13:35.132 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:35.132+0000 7fc07e83a700 1 -- start start 2023-12-25T20:13:35.132 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:35.132+0000 7fc07e83a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc0780fcb40 0x7fc07810cb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:13:35.132 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:35.132+0000 7fc07e83a700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc07810cff0 con 0x7fc0780fcb40 2023-12-25T20:13:35.132 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:35.132+0000 7fc07d838700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc0780fcb40 0x7fc07810cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:13:35.132 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:35.132+0000 7fc07d838700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc0780fcb40 0x7fc07810cb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:54360/0 (socket says 172.21.15.160:54360) 2023-12-25T20:13:35.132 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:35.132+0000 7fc07d838700 1 -- 172.21.15.160:0/3680348710 learned_addr learned my addr 172.21.15.160:0/3680348710 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:13:35.134 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:35.132+0000 7fc07d838700 1 -- 172.21.15.160:0/3680348710 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc074005040 con 0x7fc0780fcb40 2023-12-25T20:13:35.134 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:35.132+0000 7fc07d838700 1 --2- 172.21.15.160:0/3680348710 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc0780fcb40 0x7fc07810cb00 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7fc074013510 tx=0x7fc07400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:13:35.134 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:35.133+0000 7fc06e7fc700 1 -- 172.21.15.160:0/3680348710 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc074009320 con 0x7fc0780fcb40 2023-12-25T20:13:35.134 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:35.133+0000 7fc06e7fc700 1 -- 172.21.15.160:0/3680348710 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc074013f00 con 0x7fc0780fcb40 2023-12-25T20:13:35.134 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:35.133+0000 7fc07e83a700 1 -- 172.21.15.160:0/3680348710 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc07810d1f0 con 0x7fc0780fcb40 2023-12-25T20:13:35.134 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:35.133+0000 7fc06e7fc700 1 -- 172.21.15.160:0/3680348710 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc074014d90 con 0x7fc0780fcb40 2023-12-25T20:13:35.134 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:35.133+0000 7fc07e83a700 1 -- 172.21.15.160:0/3680348710 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc07810d680 con 0x7fc0780fcb40 2023-12-25T20:13:35.134 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:35.134+0000 7fc06e7fc700 1 -- 172.21.15.160:0/3680348710 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc0740095f0 con 0x7fc0780fcb40 2023-12-25T20:13:35.134 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:35.134+0000 7fc07e83a700 1 -- 172.21.15.160:0/3680348710 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc078045ad0 con 0x7fc0780fcb40 2023-12-25T20:13:35.134 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:35.134+0000 7fc06e7fc700 1 --2- 172.21.15.160:0/3680348710 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fc06402ff60 0x7fc064032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:13:35.135 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:35.134+0000 7fc06e7fc700 1 -- 172.21.15.160:0/3680348710 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc074040fc0 con 0x7fc0780fcb40 2023-12-25T20:13:35.135 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:35.134+0000 7fc07d037700 1 --2- 172.21.15.160:0/3680348710 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fc06402ff60 0x7fc064032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:13:35.135 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:35.134+0000 7fc07d037700 1 --2- 172.21.15.160:0/3680348710 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fc06402ff60 0x7fc064032420 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7fc0680096a0 tx=0x7fc068006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:13:35.138 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:35.139+0000 7fc06e7fc700 1 -- 172.21.15.160:0/3680348710 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc074014420 con 0x7fc0780fcb40 2023-12-25T20:13:35.342 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:35.342+0000 7fc07e83a700 1 -- 172.21.15.160:0/3680348710 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc07810e4d0 con 0x7fc0780fcb40 2023-12-25T20:13:35.342 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:35.343+0000 7fc06e7fc700 1 -- 172.21.15.160:0/3680348710 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc074010230 con 0x7fc0780fcb40 2023-12-25T20:13:35.343 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:13:35.343 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:13:35.344 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:35.344+0000 7fc07e83a700 1 -- 172.21.15.160:0/3680348710 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fc06402ff60 msgr2=0x7fc064032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:13:35.344 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:35.344+0000 7fc07e83a700 1 --2- 172.21.15.160:0/3680348710 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fc06402ff60 0x7fc064032420 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7fc0680096a0 tx=0x7fc068006b40).stop 2023-12-25T20:13:35.344 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:35.344+0000 7fc07e83a700 1 -- 172.21.15.160:0/3680348710 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc0780fcb40 msgr2=0x7fc07810cb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:13:35.344 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:35.344+0000 7fc07e83a700 1 --2- 172.21.15.160:0/3680348710 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc0780fcb40 0x7fc07810cb00 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7fc074013510 tx=0x7fc07400b040).stop 2023-12-25T20:13:35.344 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:35.345+0000 7fc07e83a700 1 -- 172.21.15.160:0/3680348710 shutdown_connections 2023-12-25T20:13:35.345 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:35.345+0000 7fc07e83a700 1 --2- 172.21.15.160:0/3680348710 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fc06402ff60 0x7fc064032420 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:13:35.345 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:35.345+0000 7fc07e83a700 1 --2- 172.21.15.160:0/3680348710 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc0780fcb40 0x7fc07810cb00 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:13:35.345 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:35.345+0000 7fc07e83a700 1 -- 172.21.15.160:0/3680348710 >> 172.21.15.160:0/3680348710 conn(0x7fc07806bad0 msgr2=0x7fc07806c5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:13:35.345 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:35.345+0000 7fc07e83a700 1 -- 172.21.15.160:0/3680348710 shutdown_connections 2023-12-25T20:13:35.345 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:35.345+0000 7fc07e83a700 1 -- 172.21.15.160:0/3680348710 wait complete. 2023-12-25T20:13:35.345 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:13:36.468 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:13:36 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:13:34.683736+0000 mgr.smithi050.nbzhun (mgr.14162) 196 : cluster [DBG] pgmap v184: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:13:36.469 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:13:36 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:13:35.343583+0000 mon.smithi050 (mon.0) 261 : audit [DBG] from='client.? 172.21.15.160:0/3680348710' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:13:36.729 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:13:36.729 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:13:37.963 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:37.961+0000 7fa8ba556700 1 -- 172.21.15.160:0/1759375328 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa8b40f5410 msgr2=0x7fa8b40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:13:37.963 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:37.961+0000 7fa8ba556700 1 --2- 172.21.15.160:0/1759375328 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa8b40f5410 0x7fa8b40f5830 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 rx=0x7fa8a4009fb0 tx=0x7fa8a4009b30).stop 2023-12-25T20:13:37.963 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:37.961+0000 7fa8ba556700 1 -- 172.21.15.160:0/1759375328 shutdown_connections 2023-12-25T20:13:37.963 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:37.961+0000 7fa8ba556700 1 --2- 172.21.15.160:0/1759375328 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa8b40f5410 0x7fa8b40f5830 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:13:37.963 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:37.961+0000 7fa8ba556700 1 -- 172.21.15.160:0/1759375328 >> 172.21.15.160:0/1759375328 conn(0x7fa8b40f0b70 msgr2=0x7fa8b40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:13:37.963 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:37.961+0000 7fa8ba556700 1 -- 172.21.15.160:0/1759375328 shutdown_connections 2023-12-25T20:13:37.963 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:37.961+0000 7fa8ba556700 1 -- 172.21.15.160:0/1759375328 wait complete. 2023-12-25T20:13:37.963 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:37.962+0000 7fa8ba556700 1 Processor -- start 2023-12-25T20:13:37.963 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:37.962+0000 7fa8ba556700 1 -- start start 2023-12-25T20:13:37.963 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:37.962+0000 7fa8ba556700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa8b40f5410 0x7fa8b410b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:13:37.964 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:37.962+0000 7fa8ba556700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8b4107210 con 0x7fa8b40f5410 2023-12-25T20:13:37.964 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:37.962+0000 7fa8b3fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa8b40f5410 0x7fa8b410b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:13:37.964 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:37.962+0000 7fa8b3fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa8b40f5410 0x7fa8b410b820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:58424/0 (socket says 172.21.15.160:58424) 2023-12-25T20:13:37.964 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:37.962+0000 7fa8b3fff700 1 -- 172.21.15.160:0/2810232102 learned_addr learned my addr 172.21.15.160:0/2810232102 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:13:37.964 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:37.963+0000 7fa8b3fff700 1 -- 172.21.15.160:0/2810232102 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8a4005040 con 0x7fa8b40f5410 2023-12-25T20:13:37.964 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:37.963+0000 7fa8b3fff700 1 --2- 172.21.15.160:0/2810232102 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa8b40f5410 0x7fa8b410b820 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7fa8a4013510 tx=0x7fa8a400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:13:37.964 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:37.963+0000 7fa8b0ff9700 1 -- 172.21.15.160:0/2810232102 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa8a4009320 con 0x7fa8b40f5410 2023-12-25T20:13:37.964 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:37.963+0000 7fa8b0ff9700 1 -- 172.21.15.160:0/2810232102 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa8a4013f00 con 0x7fa8b40f5410 2023-12-25T20:13:37.964 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:37.963+0000 7fa8ba556700 1 -- 172.21.15.160:0/2810232102 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8b4060220 con 0x7fa8b40f5410 2023-12-25T20:13:37.965 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:37.963+0000 7fa8b0ff9700 1 -- 172.21.15.160:0/2810232102 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa8a4014d90 con 0x7fa8b40f5410 2023-12-25T20:13:37.965 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:37.963+0000 7fa8ba556700 1 -- 172.21.15.160:0/2810232102 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8b410bd10 con 0x7fa8b40f5410 2023-12-25T20:13:37.966 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:37.964+0000 7fa8b0ff9700 1 -- 172.21.15.160:0/2810232102 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa8a40095f0 con 0x7fa8b40f5410 2023-12-25T20:13:37.966 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:37.964+0000 7fa8ba556700 1 -- 172.21.15.160:0/2810232102 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8b4045ad0 con 0x7fa8b40f5410 2023-12-25T20:13:37.966 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:37.964+0000 7fa8b0ff9700 1 --2- 172.21.15.160:0/2810232102 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fa89c02ff60 0x7fa89c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:13:37.966 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:37.964+0000 7fa8b0ff9700 1 -- 172.21.15.160:0/2810232102 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa8a40410c0 con 0x7fa8b40f5410 2023-12-25T20:13:37.966 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:37.964+0000 7fa8b37fe700 1 --2- 172.21.15.160:0/2810232102 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fa89c02ff60 0x7fa89c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:13:37.966 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:37.965+0000 7fa8b37fe700 1 --2- 172.21.15.160:0/2810232102 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fa89c02ff60 0x7fa89c032420 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7fa8a80096a0 tx=0x7fa8a8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:13:37.968 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:37.969+0000 7fa8b0ff9700 1 -- 172.21.15.160:0/2810232102 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa8a401d7f0 con 0x7fa8b40f5410 2023-12-25T20:13:38.172 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:38.172+0000 7fa8ba556700 1 -- 172.21.15.160:0/2810232102 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa8b40fae60 con 0x7fa8b40f5410 2023-12-25T20:13:38.172 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:38.173+0000 7fa8b0ff9700 1 -- 172.21.15.160:0/2810232102 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa8a4010230 con 0x7fa8b40f5410 2023-12-25T20:13:38.173 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:13:38.173 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:13:38.174 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:38.174+0000 7fa8ba556700 1 -- 172.21.15.160:0/2810232102 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fa89c02ff60 msgr2=0x7fa89c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:13:38.174 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:38.174+0000 7fa8ba556700 1 --2- 172.21.15.160:0/2810232102 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fa89c02ff60 0x7fa89c032420 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7fa8a80096a0 tx=0x7fa8a8006b40).stop 2023-12-25T20:13:38.174 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:38.174+0000 7fa8ba556700 1 -- 172.21.15.160:0/2810232102 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa8b40f5410 msgr2=0x7fa8b410b820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:13:38.174 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:38.174+0000 7fa8ba556700 1 --2- 172.21.15.160:0/2810232102 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa8b40f5410 0x7fa8b410b820 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7fa8a4013510 tx=0x7fa8a400b040).stop 2023-12-25T20:13:38.175 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:38.174+0000 7fa8ba556700 1 -- 172.21.15.160:0/2810232102 shutdown_connections 2023-12-25T20:13:38.175 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:38.174+0000 7fa8ba556700 1 --2- 172.21.15.160:0/2810232102 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fa89c02ff60 0x7fa89c032420 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:13:38.175 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:38.174+0000 7fa8ba556700 1 --2- 172.21.15.160:0/2810232102 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa8b40f5410 0x7fa8b410b820 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:13:38.175 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:38.174+0000 7fa8ba556700 1 -- 172.21.15.160:0/2810232102 >> 172.21.15.160:0/2810232102 conn(0x7fa8b40f0b70 msgr2=0x7fa8b40f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:13:38.175 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:38.175+0000 7fa8ba556700 1 -- 172.21.15.160:0/2810232102 shutdown_connections 2023-12-25T20:13:38.175 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:38.175+0000 7fa8ba556700 1 -- 172.21.15.160:0/2810232102 wait complete. 2023-12-25T20:13:38.175 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:13:38.468 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:13:38 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:13:36.684027+0000 mgr.smithi050.nbzhun (mgr.14162) 197 : cluster [DBG] pgmap v185: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:13:39.468 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:13:39 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:13:38.173786+0000 mon.smithi050 (mon.0) 262 : audit [DBG] from='client.? 172.21.15.160:0/2810232102' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:13:39.563 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:13:39.563 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:13:40.468 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:13:40 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:13:38.684324+0000 mgr.smithi050.nbzhun (mgr.14162) 198 : cluster [DBG] pgmap v186: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:13:40.808 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:40.807+0000 7fe9f13db700 1 -- 172.21.15.160:0/242972659 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe9ec0f5410 msgr2=0x7fe9ec0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:13:40.808 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:40.807+0000 7fe9f13db700 1 --2- 172.21.15.160:0/242972659 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe9ec0f5410 0x7fe9ec0f5830 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 rx=0x7fe9e0009fb0 tx=0x7fe9e0009b30).stop 2023-12-25T20:13:40.808 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:40.807+0000 7fe9f13db700 1 -- 172.21.15.160:0/242972659 shutdown_connections 2023-12-25T20:13:40.808 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:40.807+0000 7fe9f13db700 1 --2- 172.21.15.160:0/242972659 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe9ec0f5410 0x7fe9ec0f5830 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:13:40.808 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:40.807+0000 7fe9f13db700 1 -- 172.21.15.160:0/242972659 >> 172.21.15.160:0/242972659 conn(0x7fe9ec0f0b70 msgr2=0x7fe9ec0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:13:40.808 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:40.807+0000 7fe9f13db700 1 -- 172.21.15.160:0/242972659 shutdown_connections 2023-12-25T20:13:40.808 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:40.807+0000 7fe9f13db700 1 -- 172.21.15.160:0/242972659 wait complete. 2023-12-25T20:13:40.808 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:40.808+0000 7fe9f13db700 1 Processor -- start 2023-12-25T20:13:40.808 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:40.808+0000 7fe9f13db700 1 -- start start 2023-12-25T20:13:40.808 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:40.808+0000 7fe9f13db700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe9ec0f5410 0x7fe9ec1093f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:13:40.808 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:40.808+0000 7fe9f13db700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9ec060280 con 0x7fe9ec0f5410 2023-12-25T20:13:40.808 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:40.808+0000 7fe9eaffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe9ec0f5410 0x7fe9ec1093f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:13:40.809 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:40.808+0000 7fe9eaffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe9ec0f5410 0x7fe9ec1093f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:58454/0 (socket says 172.21.15.160:58454) 2023-12-25T20:13:40.809 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:40.808+0000 7fe9eaffd700 1 -- 172.21.15.160:0/2049126993 learned_addr learned my addr 172.21.15.160:0/2049126993 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:13:40.813 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:40.808+0000 7fe9eaffd700 1 -- 172.21.15.160:0/2049126993 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9e0005040 con 0x7fe9ec0f5410 2023-12-25T20:13:40.813 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:40.809+0000 7fe9eaffd700 1 --2- 172.21.15.160:0/2049126993 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe9ec0f5410 0x7fe9ec1093f0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7fe9e0014000 tx=0x7fe9e000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:13:40.813 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:40.809+0000 7fe9d37fe700 1 -- 172.21.15.160:0/2049126993 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe9e00095f0 con 0x7fe9ec0f5410 2023-12-25T20:13:40.813 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:40.809+0000 7fe9f13db700 1 -- 172.21.15.160:0/2049126993 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9ec060480 con 0x7fe9ec0f5410 2023-12-25T20:13:40.814 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:40.809+0000 7fe9d37fe700 1 -- 172.21.15.160:0/2049126993 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe9e0007e70 con 0x7fe9ec0f5410 2023-12-25T20:13:40.814 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:40.809+0000 7fe9d37fe700 1 -- 172.21.15.160:0/2049126993 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe9e0014ea0 con 0x7fe9ec0f5410 2023-12-25T20:13:40.814 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:40.809+0000 7fe9f13db700 1 -- 172.21.15.160:0/2049126993 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9ec109b20 con 0x7fe9ec0f5410 2023-12-25T20:13:40.814 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:40.809+0000 7fe9d37fe700 1 -- 172.21.15.160:0/2049126993 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe9e0025070 con 0x7fe9ec0f5410 2023-12-25T20:13:40.814 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:40.810+0000 7fe9d37fe700 1 --2- 172.21.15.160:0/2049126993 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fe9d402ff60 0x7fe9d4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:13:40.814 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:40.810+0000 7fe9d37fe700 1 -- 172.21.15.160:0/2049126993 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe9e0040970 con 0x7fe9ec0f5410 2023-12-25T20:13:40.814 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:40.810+0000 7fe9f13db700 1 -- 172.21.15.160:0/2049126993 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9ec045ad0 con 0x7fe9ec0f5410 2023-12-25T20:13:40.814 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:40.810+0000 7fe9ea7fc700 1 --2- 172.21.15.160:0/2049126993 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fe9d402ff60 0x7fe9d4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:13:40.814 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:40.810+0000 7fe9ea7fc700 1 --2- 172.21.15.160:0/2049126993 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fe9d402ff60 0x7fe9d4032420 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7fe9dc0096a0 tx=0x7fe9dc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:13:40.814 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:40.815+0000 7fe9d37fe700 1 -- 172.21.15.160:0/2049126993 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe9e00079c0 con 0x7fe9ec0f5410 2023-12-25T20:13:41.019 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:41.019+0000 7fe9f13db700 1 -- 172.21.15.160:0/2049126993 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe9ec10a340 con 0x7fe9ec0f5410 2023-12-25T20:13:41.019 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:41.020+0000 7fe9d37fe700 1 -- 172.21.15.160:0/2049126993 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe9e002d090 con 0x7fe9ec0f5410 2023-12-25T20:13:41.019 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:13:41.019 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:13:41.021 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:41.021+0000 7fe9f13db700 1 -- 172.21.15.160:0/2049126993 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fe9d402ff60 msgr2=0x7fe9d4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:13:41.021 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:41.021+0000 7fe9f13db700 1 --2- 172.21.15.160:0/2049126993 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fe9d402ff60 0x7fe9d4032420 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7fe9dc0096a0 tx=0x7fe9dc006b40).stop 2023-12-25T20:13:41.021 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:41.021+0000 7fe9f13db700 1 -- 172.21.15.160:0/2049126993 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe9ec0f5410 msgr2=0x7fe9ec1093f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:13:41.021 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:41.021+0000 7fe9f13db700 1 --2- 172.21.15.160:0/2049126993 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe9ec0f5410 0x7fe9ec1093f0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7fe9e0014000 tx=0x7fe9e000b040).stop 2023-12-25T20:13:41.021 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:41.021+0000 7fe9f13db700 1 -- 172.21.15.160:0/2049126993 shutdown_connections 2023-12-25T20:13:41.021 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:41.021+0000 7fe9f13db700 1 --2- 172.21.15.160:0/2049126993 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fe9d402ff60 0x7fe9d4032420 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:13:41.021 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:41.021+0000 7fe9f13db700 1 --2- 172.21.15.160:0/2049126993 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe9ec0f5410 0x7fe9ec1093f0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:13:41.021 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:41.021+0000 7fe9f13db700 1 -- 172.21.15.160:0/2049126993 >> 172.21.15.160:0/2049126993 conn(0x7fe9ec0f0b70 msgr2=0x7fe9ec0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:13:41.021 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:41.021+0000 7fe9f13db700 1 -- 172.21.15.160:0/2049126993 shutdown_connections 2023-12-25T20:13:41.022 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:41.021+0000 7fe9f13db700 1 -- 172.21.15.160:0/2049126993 wait complete. 2023-12-25T20:13:41.022 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:13:42.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:13:41 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:13:40.684681+0000 mgr.smithi050.nbzhun (mgr.14162) 199 : cluster [DBG] pgmap v187: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:13:42.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:13:41 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:13:41.020376+0000 mon.smithi050 (mon.0) 263 : audit [DBG] from='client.? 172.21.15.160:0/2049126993' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:13:42.427 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:13:42.428 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:13:43.616 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:43.615+0000 7f9e14bf8700 1 -- 172.21.15.160:0/1399626649 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9e100f5410 msgr2=0x7f9e100f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:13:43.616 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:43.615+0000 7f9e14bf8700 1 --2- 172.21.15.160:0/1399626649 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9e100f5410 0x7f9e100f5830 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 rx=0x7f9e00009fb0 tx=0x7f9e00009b30).stop 2023-12-25T20:13:43.616 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:43.615+0000 7f9e14bf8700 1 -- 172.21.15.160:0/1399626649 shutdown_connections 2023-12-25T20:13:43.616 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:43.615+0000 7f9e14bf8700 1 --2- 172.21.15.160:0/1399626649 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9e100f5410 0x7f9e100f5830 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:13:43.616 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:43.615+0000 7f9e14bf8700 1 -- 172.21.15.160:0/1399626649 >> 172.21.15.160:0/1399626649 conn(0x7f9e100f0b70 msgr2=0x7f9e100f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:13:43.617 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:43.615+0000 7f9e14bf8700 1 -- 172.21.15.160:0/1399626649 shutdown_connections 2023-12-25T20:13:43.617 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:43.616+0000 7f9e14bf8700 1 -- 172.21.15.160:0/1399626649 wait complete. 2023-12-25T20:13:43.617 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:43.616+0000 7f9e14bf8700 1 Processor -- start 2023-12-25T20:13:43.617 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:43.616+0000 7f9e14bf8700 1 -- start start 2023-12-25T20:13:43.617 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:43.616+0000 7f9e14bf8700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9e100f5410 0x7f9e10109400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:13:43.617 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:43.616+0000 7f9e14bf8700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e10060240 con 0x7f9e100f5410 2023-12-25T20:13:43.619 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:43.617+0000 7f9e0e59c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9e100f5410 0x7f9e10109400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:13:43.619 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:43.617+0000 7f9e0e59c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9e100f5410 0x7f9e10109400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:58478/0 (socket says 172.21.15.160:58478) 2023-12-25T20:13:43.619 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:43.617+0000 7f9e0e59c700 1 -- 172.21.15.160:0/1629384184 learned_addr learned my addr 172.21.15.160:0/1629384184 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:13:43.619 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:43.617+0000 7f9e0e59c700 1 -- 172.21.15.160:0/1629384184 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e00005040 con 0x7f9e100f5410 2023-12-25T20:13:43.619 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:43.617+0000 7f9e0e59c700 1 --2- 172.21.15.160:0/1629384184 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9e100f5410 0x7f9e10109400 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7f9e00013510 tx=0x7f9e0000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:13:43.619 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:43.617+0000 7f9e06ffd700 1 -- 172.21.15.160:0/1629384184 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9e00009320 con 0x7f9e100f5410 2023-12-25T20:13:43.619 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:43.617+0000 7f9e14bf8700 1 -- 172.21.15.160:0/1629384184 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e10060440 con 0x7f9e100f5410 2023-12-25T20:13:43.619 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:43.617+0000 7f9e06ffd700 1 -- 172.21.15.160:0/1629384184 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9e00013f00 con 0x7f9e100f5410 2023-12-25T20:13:43.619 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:43.617+0000 7f9e06ffd700 1 -- 172.21.15.160:0/1629384184 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9e00014d90 con 0x7f9e100f5410 2023-12-25T20:13:43.620 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:43.617+0000 7f9e14bf8700 1 -- 172.21.15.160:0/1629384184 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e10109ab0 con 0x7f9e100f5410 2023-12-25T20:13:43.620 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:43.618+0000 7f9e06ffd700 1 -- 172.21.15.160:0/1629384184 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9e000095f0 con 0x7f9e100f5410 2023-12-25T20:13:43.620 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:43.618+0000 7f9e14bf8700 1 -- 172.21.15.160:0/1629384184 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e10045ad0 con 0x7f9e100f5410 2023-12-25T20:13:43.620 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:43.618+0000 7f9e06ffd700 1 --2- 172.21.15.160:0/1629384184 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f9dfc02ff60 0x7f9dfc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:13:43.620 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:43.618+0000 7f9e06ffd700 1 -- 172.21.15.160:0/1629384184 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9e000410a0 con 0x7f9e100f5410 2023-12-25T20:13:43.620 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:43.619+0000 7f9e0dd9b700 1 --2- 172.21.15.160:0/1629384184 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f9dfc02ff60 0x7f9dfc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:13:43.620 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:43.619+0000 7f9e0dd9b700 1 --2- 172.21.15.160:0/1629384184 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f9dfc02ff60 0x7f9dfc032420 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f9df80096a0 tx=0x7f9df8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:13:43.622 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:43.623+0000 7f9e06ffd700 1 -- 172.21.15.160:0/1629384184 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9e0001d7f0 con 0x7f9e100f5410 2023-12-25T20:13:43.830 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:43.831+0000 7f9e14bf8700 1 -- 172.21.15.160:0/1629384184 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9e100fae60 con 0x7f9e100f5410 2023-12-25T20:13:43.831 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:43.832+0000 7f9e06ffd700 1 -- 172.21.15.160:0/1629384184 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9e00010230 con 0x7f9e100f5410 2023-12-25T20:13:43.832 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:13:43.832 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:13:43.834 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:43.833+0000 7f9e14bf8700 1 -- 172.21.15.160:0/1629384184 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f9dfc02ff60 msgr2=0x7f9dfc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:13:43.834 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:43.833+0000 7f9e14bf8700 1 --2- 172.21.15.160:0/1629384184 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f9dfc02ff60 0x7f9dfc032420 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f9df80096a0 tx=0x7f9df8006b40).stop 2023-12-25T20:13:43.834 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:43.833+0000 7f9e14bf8700 1 -- 172.21.15.160:0/1629384184 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9e100f5410 msgr2=0x7f9e10109400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:13:43.834 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:43.833+0000 7f9e14bf8700 1 --2- 172.21.15.160:0/1629384184 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9e100f5410 0x7f9e10109400 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7f9e00013510 tx=0x7f9e0000b040).stop 2023-12-25T20:13:43.834 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:43.833+0000 7f9e14bf8700 1 -- 172.21.15.160:0/1629384184 shutdown_connections 2023-12-25T20:13:43.834 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:43.833+0000 7f9e14bf8700 1 --2- 172.21.15.160:0/1629384184 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f9dfc02ff60 0x7f9dfc032420 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:13:43.834 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:43.833+0000 7f9e14bf8700 1 --2- 172.21.15.160:0/1629384184 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9e100f5410 0x7f9e10109400 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:13:43.834 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:43.833+0000 7f9e14bf8700 1 -- 172.21.15.160:0/1629384184 >> 172.21.15.160:0/1629384184 conn(0x7f9e100f0b70 msgr2=0x7f9e100f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:13:43.834 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:43.833+0000 7f9e14bf8700 1 -- 172.21.15.160:0/1629384184 shutdown_connections 2023-12-25T20:13:43.835 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:43.833+0000 7f9e14bf8700 1 -- 172.21.15.160:0/1629384184 wait complete. 2023-12-25T20:13:43.835 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:13:44.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:13:43 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:13:42.685054+0000 mgr.smithi050.nbzhun (mgr.14162) 200 : cluster [DBG] pgmap v188: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:13:45.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:13:44 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:13:43.832112+0000 mon.smithi050 (mon.0) 264 : audit [DBG] from='client.? 172.21.15.160:0/1629384184' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:13:45.299 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:13:45.299 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:13:46.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:13:45 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:13:44.685408+0000 mgr.smithi050.nbzhun (mgr.14162) 201 : cluster [DBG] pgmap v189: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:13:46.516 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:46.515+0000 7f47d74af700 1 -- 172.21.15.160:0/2972881552 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f47d0060280 msgr2=0x7f47d00f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:13:46.517 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:46.515+0000 7f47d74af700 1 --2- 172.21.15.160:0/2972881552 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f47d0060280 0x7f47d00f90f0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 rx=0x7f47c0009fb0 tx=0x7f47c0009b30).stop 2023-12-25T20:13:46.517 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:46.515+0000 7f47d74af700 1 -- 172.21.15.160:0/2972881552 shutdown_connections 2023-12-25T20:13:46.517 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:46.515+0000 7f47d74af700 1 --2- 172.21.15.160:0/2972881552 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f47d0060280 0x7f47d00f90f0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:13:46.517 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:46.515+0000 7f47d74af700 1 -- 172.21.15.160:0/2972881552 >> 172.21.15.160:0/2972881552 conn(0x7f47d00f0b70 msgr2=0x7f47d00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:13:46.517 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:46.515+0000 7f47d74af700 1 -- 172.21.15.160:0/2972881552 shutdown_connections 2023-12-25T20:13:46.517 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:46.515+0000 7f47d74af700 1 -- 172.21.15.160:0/2972881552 wait complete. 2023-12-25T20:13:46.517 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:46.516+0000 7f47d74af700 1 Processor -- start 2023-12-25T20:13:46.517 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:46.516+0000 7f47d74af700 1 -- start start 2023-12-25T20:13:46.517 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:46.516+0000 7f47d74af700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f47d0060280 0x7f47d01066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:13:46.518 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:46.516+0000 7f47d74af700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47d0106bd0 con 0x7f47d0060280 2023-12-25T20:13:46.518 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:46.516+0000 7f47d524b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f47d0060280 0x7f47d01066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:13:46.518 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:46.516+0000 7f47d524b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f47d0060280 0x7f47d01066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:58500/0 (socket says 172.21.15.160:58500) 2023-12-25T20:13:46.518 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:46.516+0000 7f47d524b700 1 -- 172.21.15.160:0/886907526 learned_addr learned my addr 172.21.15.160:0/886907526 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:13:46.518 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:46.516+0000 7f47d524b700 1 -- 172.21.15.160:0/886907526 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47c0005040 con 0x7f47d0060280 2023-12-25T20:13:46.518 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:46.516+0000 7f47d524b700 1 --2- 172.21.15.160:0/886907526 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f47d0060280 0x7f47d01066e0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7f47c0013510 tx=0x7f47c000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:13:46.518 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:46.517+0000 7f47cdffb700 1 -- 172.21.15.160:0/886907526 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f47c0009320 con 0x7f47d0060280 2023-12-25T20:13:46.518 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:46.517+0000 7f47d74af700 1 -- 172.21.15.160:0/886907526 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47d0106dd0 con 0x7f47d0060280 2023-12-25T20:13:46.518 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:46.517+0000 7f47cdffb700 1 -- 172.21.15.160:0/886907526 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f47c0013f00 con 0x7f47d0060280 2023-12-25T20:13:46.518 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:46.517+0000 7f47cdffb700 1 -- 172.21.15.160:0/886907526 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f47c0014dc0 con 0x7f47d0060280 2023-12-25T20:13:46.518 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:46.517+0000 7f47d74af700 1 -- 172.21.15.160:0/886907526 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47d0107260 con 0x7f47d0060280 2023-12-25T20:13:46.519 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:46.518+0000 7f47cdffb700 1 -- 172.21.15.160:0/886907526 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f47c001d3d0 con 0x7f47d0060280 2023-12-25T20:13:46.519 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:46.518+0000 7f47cdffb700 1 --2- 172.21.15.160:0/886907526 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f47b802ff60 0x7f47b8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:13:46.519 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:46.518+0000 7f47cdffb700 1 -- 172.21.15.160:0/886907526 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f47c00410c0 con 0x7f47d0060280 2023-12-25T20:13:46.519 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:46.518+0000 7f47d74af700 1 -- 172.21.15.160:0/886907526 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47d0045ad0 con 0x7f47d0060280 2023-12-25T20:13:46.519 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:46.518+0000 7f47d4a4a700 1 --2- 172.21.15.160:0/886907526 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f47b802ff60 0x7f47b8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:13:46.519 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:46.518+0000 7f47d4a4a700 1 --2- 172.21.15.160:0/886907526 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f47b802ff60 0x7f47b8032420 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f47c40089e0 tx=0x7f47c4006ec0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:13:46.522 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:46.523+0000 7f47cdffb700 1 -- 172.21.15.160:0/886907526 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f47c0012920 con 0x7f47d0060280 2023-12-25T20:13:46.727 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:46.727+0000 7f47d74af700 1 -- 172.21.15.160:0/886907526 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f47d01080b0 con 0x7f47d0060280 2023-12-25T20:13:46.728 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:46.728+0000 7f47cdffb700 1 -- 172.21.15.160:0/886907526 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f47c0012330 con 0x7f47d0060280 2023-12-25T20:13:46.728 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:13:46.729 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:13:46.730 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:46.730+0000 7f47d74af700 1 -- 172.21.15.160:0/886907526 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f47b802ff60 msgr2=0x7f47b8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:13:46.730 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:46.730+0000 7f47d74af700 1 --2- 172.21.15.160:0/886907526 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f47b802ff60 0x7f47b8032420 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f47c40089e0 tx=0x7f47c4006ec0).stop 2023-12-25T20:13:46.730 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:46.730+0000 7f47d74af700 1 -- 172.21.15.160:0/886907526 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f47d0060280 msgr2=0x7f47d01066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:13:46.730 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:46.730+0000 7f47d74af700 1 --2- 172.21.15.160:0/886907526 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f47d0060280 0x7f47d01066e0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7f47c0013510 tx=0x7f47c000b040).stop 2023-12-25T20:13:46.730 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:46.730+0000 7f47d74af700 1 -- 172.21.15.160:0/886907526 shutdown_connections 2023-12-25T20:13:46.731 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:46.730+0000 7f47d74af700 1 --2- 172.21.15.160:0/886907526 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f47b802ff60 0x7f47b8032420 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:13:46.731 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:46.730+0000 7f47d74af700 1 --2- 172.21.15.160:0/886907526 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f47d0060280 0x7f47d01066e0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:13:46.731 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:46.730+0000 7f47d74af700 1 -- 172.21.15.160:0/886907526 >> 172.21.15.160:0/886907526 conn(0x7f47d00f0b70 msgr2=0x7f47d00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:13:46.731 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:46.730+0000 7f47d74af700 1 -- 172.21.15.160:0/886907526 shutdown_connections 2023-12-25T20:13:46.731 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:46.730+0000 7f47d74af700 1 -- 172.21.15.160:0/886907526 wait complete. 2023-12-25T20:13:46.731 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:13:47.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:13:46 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:13:46.728857+0000 mon.smithi050 (mon.0) 265 : audit [DBG] from='client.? 172.21.15.160:0/886907526' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:13:48.124 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:13:48.124 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:13:48.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:13:47 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:13:46.685784+0000 mgr.smithi050.nbzhun (mgr.14162) 202 : cluster [DBG] pgmap v190: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:13:49.315 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:49.314+0000 7f6705cf1700 1 -- 172.21.15.160:0/3160169268 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f67000f5410 msgr2=0x7f67000f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:13:49.315 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:49.314+0000 7f6705cf1700 1 --2- 172.21.15.160:0/3160169268 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f67000f5410 0x7f67000f5830 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 rx=0x7f66e8009fb0 tx=0x7f66e8009b30).stop 2023-12-25T20:13:49.315 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:49.314+0000 7f6705cf1700 1 -- 172.21.15.160:0/3160169268 shutdown_connections 2023-12-25T20:13:49.315 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:49.314+0000 7f6705cf1700 1 --2- 172.21.15.160:0/3160169268 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f67000f5410 0x7f67000f5830 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:13:49.316 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:49.314+0000 7f6705cf1700 1 -- 172.21.15.160:0/3160169268 >> 172.21.15.160:0/3160169268 conn(0x7f67000f0b70 msgr2=0x7f67000f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:13:49.316 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:49.315+0000 7f6705cf1700 1 -- 172.21.15.160:0/3160169268 shutdown_connections 2023-12-25T20:13:49.316 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:49.315+0000 7f6705cf1700 1 -- 172.21.15.160:0/3160169268 wait complete. 2023-12-25T20:13:49.318 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:49.315+0000 7f6705cf1700 1 Processor -- start 2023-12-25T20:13:49.318 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:49.315+0000 7f6705cf1700 1 -- start start 2023-12-25T20:13:49.318 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:49.315+0000 7f6705cf1700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f67000f5410 0x7f6700106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:13:49.318 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:49.315+0000 7f6705cf1700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6700106c10 con 0x7f67000f5410 2023-12-25T20:13:49.318 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:49.316+0000 7f66ff7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f67000f5410 0x7f6700106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:13:49.319 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:49.316+0000 7f66ff7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f67000f5410 0x7f6700106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:45036/0 (socket says 172.21.15.160:45036) 2023-12-25T20:13:49.319 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:49.316+0000 7f66ff7fe700 1 -- 172.21.15.160:0/868335816 learned_addr learned my addr 172.21.15.160:0/868335816 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:13:49.319 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:49.316+0000 7f66ff7fe700 1 -- 172.21.15.160:0/868335816 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66e8005040 con 0x7f67000f5410 2023-12-25T20:13:49.319 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:49.316+0000 7f66ff7fe700 1 --2- 172.21.15.160:0/868335816 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f67000f5410 0x7f6700106720 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7f66e8013510 tx=0x7f66e800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:13:49.319 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:49.316+0000 7f66f7fff700 1 -- 172.21.15.160:0/868335816 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f66e8009320 con 0x7f67000f5410 2023-12-25T20:13:49.319 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:49.316+0000 7f6705cf1700 1 -- 172.21.15.160:0/868335816 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6700106e10 con 0x7f67000f5410 2023-12-25T20:13:49.320 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:49.316+0000 7f66f7fff700 1 -- 172.21.15.160:0/868335816 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f66e8013f00 con 0x7f67000f5410 2023-12-25T20:13:49.320 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:49.316+0000 7f6705cf1700 1 -- 172.21.15.160:0/868335816 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67001072a0 con 0x7f67000f5410 2023-12-25T20:13:49.320 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:49.316+0000 7f66f7fff700 1 -- 172.21.15.160:0/868335816 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f66e8014d90 con 0x7f67000f5410 2023-12-25T20:13:49.320 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:49.317+0000 7f6705cf1700 1 -- 172.21.15.160:0/868335816 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6700045ad0 con 0x7f67000f5410 2023-12-25T20:13:49.320 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:49.317+0000 7f66f7fff700 1 -- 172.21.15.160:0/868335816 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f66e80095f0 con 0x7f67000f5410 2023-12-25T20:13:49.320 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:49.317+0000 7f66f7fff700 1 --2- 172.21.15.160:0/868335816 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f66ec02ff60 0x7f66ec032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:13:49.320 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:49.317+0000 7f66f7fff700 1 -- 172.21.15.160:0/868335816 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f66e8040fd0 con 0x7f67000f5410 2023-12-25T20:13:49.320 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:49.317+0000 7f66feffd700 1 --2- 172.21.15.160:0/868335816 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f66ec02ff60 0x7f66ec032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:13:49.321 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:49.318+0000 7f66feffd700 1 --2- 172.21.15.160:0/868335816 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f66ec02ff60 0x7f66ec032420 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f66f00096a0 tx=0x7f66f0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:13:49.322 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:49.322+0000 7f66f7fff700 1 -- 172.21.15.160:0/868335816 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f66e8014420 con 0x7f67000f5410 2023-12-25T20:13:49.528 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:49.527+0000 7f6705cf1700 1 -- 172.21.15.160:0/868335816 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f67001080f0 con 0x7f67000f5410 2023-12-25T20:13:49.529 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:49.528+0000 7f66f7fff700 1 -- 172.21.15.160:0/868335816 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f66e8010230 con 0x7f67000f5410 2023-12-25T20:13:49.529 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:13:49.529 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:13:49.531 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:49.530+0000 7f6705cf1700 1 -- 172.21.15.160:0/868335816 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f66ec02ff60 msgr2=0x7f66ec032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:13:49.531 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:49.530+0000 7f6705cf1700 1 --2- 172.21.15.160:0/868335816 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f66ec02ff60 0x7f66ec032420 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f66f00096a0 tx=0x7f66f0006b40).stop 2023-12-25T20:13:49.531 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:49.530+0000 7f6705cf1700 1 -- 172.21.15.160:0/868335816 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f67000f5410 msgr2=0x7f6700106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:13:49.531 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:49.530+0000 7f6705cf1700 1 --2- 172.21.15.160:0/868335816 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f67000f5410 0x7f6700106720 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7f66e8013510 tx=0x7f66e800b040).stop 2023-12-25T20:13:49.531 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:49.530+0000 7f6705cf1700 1 -- 172.21.15.160:0/868335816 shutdown_connections 2023-12-25T20:13:49.531 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:49.530+0000 7f6705cf1700 1 --2- 172.21.15.160:0/868335816 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f66ec02ff60 0x7f66ec032420 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:13:49.531 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:49.530+0000 7f6705cf1700 1 --2- 172.21.15.160:0/868335816 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f67000f5410 0x7f6700106720 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:13:49.531 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:49.530+0000 7f6705cf1700 1 -- 172.21.15.160:0/868335816 >> 172.21.15.160:0/868335816 conn(0x7f67000f0b70 msgr2=0x7f670005fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:13:49.531 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:49.530+0000 7f6705cf1700 1 -- 172.21.15.160:0/868335816 shutdown_connections 2023-12-25T20:13:49.532 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:49.530+0000 7f6705cf1700 1 -- 172.21.15.160:0/868335816 wait complete. 2023-12-25T20:13:49.532 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:13:50.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:13:49 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:13:48.686145+0000 mgr.smithi050.nbzhun (mgr.14162) 203 : cluster [DBG] pgmap v191: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:13:50.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:13:49 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:13:49.529837+0000 mon.smithi050 (mon.0) 266 : audit [DBG] from='client.? 172.21.15.160:0/868335816' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:13:50.933 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:13:50.934 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:13:52.131 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:52.129+0000 7fda73c3b700 1 -- 172.21.15.160:0/1660901537 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fda6c0f6510 msgr2=0x7fda6c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:13:52.132 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:52.129+0000 7fda73c3b700 1 --2- 172.21.15.160:0/1660901537 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fda6c0f6510 0x7fda6c0f6930 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 rx=0x7fda5c009fb0 tx=0x7fda5c009b30).stop 2023-12-25T20:13:52.132 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:52.130+0000 7fda73c3b700 1 -- 172.21.15.160:0/1660901537 shutdown_connections 2023-12-25T20:13:52.132 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:52.130+0000 7fda73c3b700 1 --2- 172.21.15.160:0/1660901537 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fda6c0f6510 0x7fda6c0f6930 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:13:52.132 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:52.130+0000 7fda73c3b700 1 -- 172.21.15.160:0/1660901537 >> 172.21.15.160:0/1660901537 conn(0x7fda6c0f1c70 msgr2=0x7fda6c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:13:52.132 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:52.130+0000 7fda73c3b700 1 -- 172.21.15.160:0/1660901537 shutdown_connections 2023-12-25T20:13:52.132 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:52.130+0000 7fda73c3b700 1 -- 172.21.15.160:0/1660901537 wait complete. 2023-12-25T20:13:52.132 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:52.130+0000 7fda73c3b700 1 Processor -- start 2023-12-25T20:13:52.132 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:52.130+0000 7fda73c3b700 1 -- start start 2023-12-25T20:13:52.132 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:52.130+0000 7fda73c3b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fda6c0f6510 0x7fda6c106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:13:52.132 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:52.130+0000 7fda73c3b700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda6c106c20 con 0x7fda6c0f6510 2023-12-25T20:13:52.132 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:52.131+0000 7fda719d7700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fda6c0f6510 0x7fda6c106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:13:52.132 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:52.131+0000 7fda719d7700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fda6c0f6510 0x7fda6c106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:45056/0 (socket says 172.21.15.160:45056) 2023-12-25T20:13:52.133 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:52.131+0000 7fda719d7700 1 -- 172.21.15.160:0/3480375543 learned_addr learned my addr 172.21.15.160:0/3480375543 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:13:52.133 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:52.131+0000 7fda719d7700 1 -- 172.21.15.160:0/3480375543 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda5c005040 con 0x7fda6c0f6510 2023-12-25T20:13:52.133 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:52.131+0000 7fda719d7700 1 --2- 172.21.15.160:0/3480375543 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fda6c0f6510 0x7fda6c106730 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7fda5c013510 tx=0x7fda5c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:13:52.134 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:52.131+0000 7fda627fc700 1 -- 172.21.15.160:0/3480375543 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda5c009320 con 0x7fda6c0f6510 2023-12-25T20:13:52.134 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:52.131+0000 7fda73c3b700 1 -- 172.21.15.160:0/3480375543 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda6c106e20 con 0x7fda6c0f6510 2023-12-25T20:13:52.134 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:52.131+0000 7fda627fc700 1 -- 172.21.15.160:0/3480375543 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fda5c013f00 con 0x7fda6c0f6510 2023-12-25T20:13:52.134 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:52.131+0000 7fda627fc700 1 -- 172.21.15.160:0/3480375543 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda5c014d90 con 0x7fda6c0f6510 2023-12-25T20:13:52.134 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:52.131+0000 7fda73c3b700 1 -- 172.21.15.160:0/3480375543 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda6c1072b0 con 0x7fda6c0f6510 2023-12-25T20:13:52.134 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:52.132+0000 7fda627fc700 1 -- 172.21.15.160:0/3480375543 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fda5c0095f0 con 0x7fda6c0f6510 2023-12-25T20:13:52.134 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:52.132+0000 7fda73c3b700 1 -- 172.21.15.160:0/3480375543 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda6c045ad0 con 0x7fda6c0f6510 2023-12-25T20:13:52.135 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:52.132+0000 7fda627fc700 1 --2- 172.21.15.160:0/3480375543 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fda5802ff60 0x7fda58032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:13:52.135 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:52.132+0000 7fda627fc700 1 -- 172.21.15.160:0/3480375543 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fda5c0410c0 con 0x7fda6c0f6510 2023-12-25T20:13:52.135 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:52.132+0000 7fda711d6700 1 --2- 172.21.15.160:0/3480375543 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fda5802ff60 0x7fda58032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:13:52.135 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:52.133+0000 7fda711d6700 1 --2- 172.21.15.160:0/3480375543 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fda5802ff60 0x7fda58032420 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7fda680096a0 tx=0x7fda68006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:13:52.137 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:52.137+0000 7fda627fc700 1 -- 172.21.15.160:0/3480375543 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fda5c014420 con 0x7fda6c0f6510 2023-12-25T20:13:52.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:13:51 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:13:50.686504+0000 mgr.smithi050.nbzhun (mgr.14162) 204 : cluster [DBG] pgmap v192: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:13:52.342 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:52.341+0000 7fda73c3b700 1 -- 172.21.15.160:0/3480375543 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fda6c0fbf60 con 0x7fda6c0f6510 2023-12-25T20:13:52.343 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:52.342+0000 7fda627fc700 1 -- 172.21.15.160:0/3480375543 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fda5c010230 con 0x7fda6c0f6510 2023-12-25T20:13:52.343 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:13:52.343 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:13:52.344 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:52.343+0000 7fda73c3b700 1 -- 172.21.15.160:0/3480375543 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fda5802ff60 msgr2=0x7fda58032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:13:52.344 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:52.343+0000 7fda73c3b700 1 --2- 172.21.15.160:0/3480375543 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fda5802ff60 0x7fda58032420 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7fda680096a0 tx=0x7fda68006b40).stop 2023-12-25T20:13:52.344 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:52.343+0000 7fda73c3b700 1 -- 172.21.15.160:0/3480375543 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fda6c0f6510 msgr2=0x7fda6c106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:13:52.344 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:52.343+0000 7fda73c3b700 1 --2- 172.21.15.160:0/3480375543 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fda6c0f6510 0x7fda6c106730 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7fda5c013510 tx=0x7fda5c00b040).stop 2023-12-25T20:13:52.344 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:52.343+0000 7fda73c3b700 1 -- 172.21.15.160:0/3480375543 shutdown_connections 2023-12-25T20:13:52.345 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:52.343+0000 7fda73c3b700 1 --2- 172.21.15.160:0/3480375543 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fda5802ff60 0x7fda58032420 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:13:52.345 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:52.343+0000 7fda73c3b700 1 --2- 172.21.15.160:0/3480375543 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fda6c0f6510 0x7fda6c106730 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:13:52.345 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:52.343+0000 7fda73c3b700 1 -- 172.21.15.160:0/3480375543 >> 172.21.15.160:0/3480375543 conn(0x7fda6c0f1c70 msgr2=0x7fda6c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:13:52.345 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:52.343+0000 7fda73c3b700 1 -- 172.21.15.160:0/3480375543 shutdown_connections 2023-12-25T20:13:52.345 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:52.343+0000 7fda73c3b700 1 -- 172.21.15.160:0/3480375543 wait complete. 2023-12-25T20:13:52.345 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:13:53.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:13:52 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:13:52.343504+0000 mon.smithi050 (mon.0) 267 : audit [DBG] from='client.? 172.21.15.160:0/3480375543' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:13:53.715 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:13:53.715 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:13:54.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:13:53 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:13:52.686817+0000 mgr.smithi050.nbzhun (mgr.14162) 205 : cluster [DBG] pgmap v193: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:13:54.902 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:54.898+0000 7f941e6ff700 1 -- 172.21.15.160:0/77660981 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f94180f51a0 msgr2=0x7f94180f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:13:54.903 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:54.898+0000 7f941e6ff700 1 --2- 172.21.15.160:0/77660981 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f94180f51a0 0x7f94180f55c0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 rx=0x7f9414009fb0 tx=0x7f9414009b30).stop 2023-12-25T20:13:54.903 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:54.898+0000 7f941e6ff700 1 -- 172.21.15.160:0/77660981 shutdown_connections 2023-12-25T20:13:54.903 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:54.898+0000 7f941e6ff700 1 --2- 172.21.15.160:0/77660981 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f94180f51a0 0x7f94180f55c0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:13:54.903 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:54.898+0000 7f941e6ff700 1 -- 172.21.15.160:0/77660981 >> 172.21.15.160:0/77660981 conn(0x7f94180f09a0 msgr2=0x7f94180f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:13:54.903 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:54.898+0000 7f941e6ff700 1 -- 172.21.15.160:0/77660981 shutdown_connections 2023-12-25T20:13:54.903 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:54.898+0000 7f941e6ff700 1 -- 172.21.15.160:0/77660981 wait complete. 2023-12-25T20:13:54.903 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:54.899+0000 7f941e6ff700 1 Processor -- start 2023-12-25T20:13:54.903 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:54.899+0000 7f941e6ff700 1 -- start start 2023-12-25T20:13:54.903 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:54.899+0000 7f941e6ff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f94180f51a0 0x7f941810b570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:13:54.903 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:54.899+0000 7f941e6ff700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9418106f60 con 0x7f94180f51a0 2023-12-25T20:13:54.903 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:54.899+0000 7f941d6fd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f94180f51a0 0x7f941810b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:13:54.903 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:54.899+0000 7f941d6fd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f94180f51a0 0x7f941810b570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:45074/0 (socket says 172.21.15.160:45074) 2023-12-25T20:13:54.904 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:54.899+0000 7f941d6fd700 1 -- 172.21.15.160:0/3801130793 learned_addr learned my addr 172.21.15.160:0/3801130793 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:13:54.904 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:54.899+0000 7f941d6fd700 1 -- 172.21.15.160:0/3801130793 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9414005040 con 0x7f94180f51a0 2023-12-25T20:13:54.904 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:54.900+0000 7f941d6fd700 1 --2- 172.21.15.160:0/3801130793 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f94180f51a0 0x7f941810b570 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7f9414013510 tx=0x7f941400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:13:54.904 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:54.900+0000 7f940dffb700 1 -- 172.21.15.160:0/3801130793 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9414009320 con 0x7f94180f51a0 2023-12-25T20:13:54.904 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:54.900+0000 7f940dffb700 1 -- 172.21.15.160:0/3801130793 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9414013f00 con 0x7f94180f51a0 2023-12-25T20:13:54.904 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:54.900+0000 7f941e6ff700 1 -- 172.21.15.160:0/3801130793 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9418060280 con 0x7f94180f51a0 2023-12-25T20:13:54.904 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:54.900+0000 7f940dffb700 1 -- 172.21.15.160:0/3801130793 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9414014dc0 con 0x7f94180f51a0 2023-12-25T20:13:54.904 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:54.900+0000 7f941e6ff700 1 -- 172.21.15.160:0/3801130793 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f941810ba60 con 0x7f94180f51a0 2023-12-25T20:13:54.904 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:54.901+0000 7f940dffb700 1 -- 172.21.15.160:0/3801130793 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9414014420 con 0x7f94180f51a0 2023-12-25T20:13:54.904 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:54.901+0000 7f941e6ff700 1 -- 172.21.15.160:0/3801130793 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9418045ad0 con 0x7f94180f51a0 2023-12-25T20:13:54.904 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:54.901+0000 7f940dffb700 1 --2- 172.21.15.160:0/3801130793 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f940402ffb0 0x7f9404032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:13:54.904 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:54.901+0000 7f940dffb700 1 -- 172.21.15.160:0/3801130793 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9414041040 con 0x7f94180f51a0 2023-12-25T20:13:54.904 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:54.901+0000 7f941cefc700 1 --2- 172.21.15.160:0/3801130793 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f940402ffb0 0x7f9404032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:13:54.904 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:54.902+0000 7f941cefc700 1 --2- 172.21.15.160:0/3801130793 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f940402ffb0 0x7f9404032470 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f94080096a0 tx=0x7f9408006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:13:54.906 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:54.906+0000 7f940dffb700 1 -- 172.21.15.160:0/3801130793 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f94140095f0 con 0x7f94180f51a0 2023-12-25T20:13:55.112 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:55.111+0000 7f941e6ff700 1 -- 172.21.15.160:0/3801130793 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f941810c2b0 con 0x7f94180f51a0 2023-12-25T20:13:55.112 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:55.112+0000 7f940dffb700 1 -- 172.21.15.160:0/3801130793 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f941401db70 con 0x7f94180f51a0 2023-12-25T20:13:55.112 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:13:55.113 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:13:55.114 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:55.113+0000 7f941e6ff700 1 -- 172.21.15.160:0/3801130793 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f940402ffb0 msgr2=0x7f9404032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:13:55.114 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:55.113+0000 7f941e6ff700 1 --2- 172.21.15.160:0/3801130793 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f940402ffb0 0x7f9404032470 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f94080096a0 tx=0x7f9408006b40).stop 2023-12-25T20:13:55.114 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:55.113+0000 7f941e6ff700 1 -- 172.21.15.160:0/3801130793 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f94180f51a0 msgr2=0x7f941810b570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:13:55.114 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:55.113+0000 7f941e6ff700 1 --2- 172.21.15.160:0/3801130793 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f94180f51a0 0x7f941810b570 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7f9414013510 tx=0x7f941400b040).stop 2023-12-25T20:13:55.114 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:55.113+0000 7f941e6ff700 1 -- 172.21.15.160:0/3801130793 shutdown_connections 2023-12-25T20:13:55.114 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:55.113+0000 7f941e6ff700 1 --2- 172.21.15.160:0/3801130793 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f940402ffb0 0x7f9404032470 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:13:55.115 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:55.113+0000 7f941e6ff700 1 --2- 172.21.15.160:0/3801130793 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f94180f51a0 0x7f941810b570 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:13:55.115 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:55.113+0000 7f941e6ff700 1 -- 172.21.15.160:0/3801130793 >> 172.21.15.160:0/3801130793 conn(0x7f94180f09a0 msgr2=0x7f94180f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:13:55.115 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:55.113+0000 7f941e6ff700 1 -- 172.21.15.160:0/3801130793 shutdown_connections 2023-12-25T20:13:55.115 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:55.113+0000 7f941e6ff700 1 -- 172.21.15.160:0/3801130793 wait complete. 2023-12-25T20:13:55.115 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:13:56.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:13:55 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:13:54.687153+0000 mgr.smithi050.nbzhun (mgr.14162) 206 : cluster [DBG] pgmap v194: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:13:56.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:13:55 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:13:55.113374+0000 mon.smithi050 (mon.0) 268 : audit [DBG] from='client.? 172.21.15.160:0/3801130793' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:13:56.514 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:13:56.514 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:13:57.712 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:57.711+0000 7fb9a38ba700 1 -- 172.21.15.160:0/1734100973 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb99c0f5410 msgr2=0x7fb99c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:13:57.712 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:57.711+0000 7fb9a38ba700 1 --2- 172.21.15.160:0/1734100973 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb99c0f5410 0x7fb99c0f5830 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 rx=0x7fb98c009fb0 tx=0x7fb98c009b30).stop 2023-12-25T20:13:57.712 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:57.711+0000 7fb9a38ba700 1 -- 172.21.15.160:0/1734100973 shutdown_connections 2023-12-25T20:13:57.712 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:57.711+0000 7fb9a38ba700 1 --2- 172.21.15.160:0/1734100973 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb99c0f5410 0x7fb99c0f5830 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:13:57.712 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:57.711+0000 7fb9a38ba700 1 -- 172.21.15.160:0/1734100973 >> 172.21.15.160:0/1734100973 conn(0x7fb99c0f0b70 msgr2=0x7fb99c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:13:57.712 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:57.711+0000 7fb9a38ba700 1 -- 172.21.15.160:0/1734100973 shutdown_connections 2023-12-25T20:13:57.712 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:57.711+0000 7fb9a38ba700 1 -- 172.21.15.160:0/1734100973 wait complete. 2023-12-25T20:13:57.714 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:57.711+0000 7fb9a38ba700 1 Processor -- start 2023-12-25T20:13:57.714 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:57.712+0000 7fb9a38ba700 1 -- start start 2023-12-25T20:13:57.714 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:57.712+0000 7fb9a38ba700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb99c0f5410 0x7fb99c109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:13:57.714 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:57.712+0000 7fb9a38ba700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb99c060220 con 0x7fb99c0f5410 2023-12-25T20:13:57.714 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:57.712+0000 7fb9a1656700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb99c0f5410 0x7fb99c109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:13:57.714 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:57.712+0000 7fb9a1656700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb99c0f5410 0x7fb99c109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:35136/0 (socket says 172.21.15.160:35136) 2023-12-25T20:13:57.714 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:57.712+0000 7fb9a1656700 1 -- 172.21.15.160:0/2385268228 learned_addr learned my addr 172.21.15.160:0/2385268228 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:13:57.714 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:57.712+0000 7fb9a1656700 1 -- 172.21.15.160:0/2385268228 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb98c005040 con 0x7fb99c0f5410 2023-12-25T20:13:57.714 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:57.713+0000 7fb9a1656700 1 --2- 172.21.15.160:0/2385268228 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb99c0f5410 0x7fb99c109450 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7fb98c013510 tx=0x7fb98c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:13:57.714 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:57.713+0000 7fb991ffb700 1 -- 172.21.15.160:0/2385268228 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb98c009320 con 0x7fb99c0f5410 2023-12-25T20:13:57.714 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:57.713+0000 7fb991ffb700 1 -- 172.21.15.160:0/2385268228 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb98c013f00 con 0x7fb99c0f5410 2023-12-25T20:13:57.714 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:57.713+0000 7fb9a38ba700 1 -- 172.21.15.160:0/2385268228 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb99c060420 con 0x7fb99c0f5410 2023-12-25T20:13:57.715 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:57.713+0000 7fb991ffb700 1 -- 172.21.15.160:0/2385268228 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb98c014d90 con 0x7fb99c0f5410 2023-12-25T20:13:57.715 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:57.713+0000 7fb9a38ba700 1 -- 172.21.15.160:0/2385268228 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb99c109b00 con 0x7fb99c0f5410 2023-12-25T20:13:57.715 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:57.714+0000 7fb991ffb700 1 -- 172.21.15.160:0/2385268228 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb98c0263a0 con 0x7fb99c0f5410 2023-12-25T20:13:57.715 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:57.714+0000 7fb9a38ba700 1 -- 172.21.15.160:0/2385268228 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb99c059100 con 0x7fb99c0f5410 2023-12-25T20:13:57.716 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:57.714+0000 7fb991ffb700 1 --2- 172.21.15.160:0/2385268228 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fb98802ff60 0x7fb988032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:13:57.716 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:57.714+0000 7fb991ffb700 1 -- 172.21.15.160:0/2385268228 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb98c041390 con 0x7fb99c0f5410 2023-12-25T20:13:57.716 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:57.714+0000 7fb9a0e55700 1 --2- 172.21.15.160:0/2385268228 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fb98802ff60 0x7fb988032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:13:57.716 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:57.715+0000 7fb9a0e55700 1 --2- 172.21.15.160:0/2385268228 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fb98802ff60 0x7fb988032420 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7fb9980096a0 tx=0x7fb998006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:13:57.719 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:57.719+0000 7fb991ffb700 1 -- 172.21.15.160:0/2385268228 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb98c01cb10 con 0x7fb99c0f5410 2023-12-25T20:13:57.925 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:57.924+0000 7fb9a38ba700 1 -- 172.21.15.160:0/2385268228 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb99c045ad0 con 0x7fb99c0f5410 2023-12-25T20:13:57.925 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:57.925+0000 7fb991ffb700 1 -- 172.21.15.160:0/2385268228 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb98c01cb10 con 0x7fb99c0f5410 2023-12-25T20:13:57.926 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:13:57.926 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:13:57.927 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:57.926+0000 7fb9a38ba700 1 -- 172.21.15.160:0/2385268228 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fb98802ff60 msgr2=0x7fb988032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:13:57.927 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:57.926+0000 7fb9a38ba700 1 --2- 172.21.15.160:0/2385268228 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fb98802ff60 0x7fb988032420 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7fb9980096a0 tx=0x7fb998006b40).stop 2023-12-25T20:13:57.927 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:57.926+0000 7fb9a38ba700 1 -- 172.21.15.160:0/2385268228 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb99c0f5410 msgr2=0x7fb99c109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:13:57.927 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:57.926+0000 7fb9a38ba700 1 --2- 172.21.15.160:0/2385268228 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb99c0f5410 0x7fb99c109450 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7fb98c013510 tx=0x7fb98c00b040).stop 2023-12-25T20:13:57.928 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:57.927+0000 7fb9a38ba700 1 -- 172.21.15.160:0/2385268228 shutdown_connections 2023-12-25T20:13:57.928 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:57.927+0000 7fb9a38ba700 1 --2- 172.21.15.160:0/2385268228 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fb98802ff60 0x7fb988032420 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:13:57.928 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:57.927+0000 7fb9a38ba700 1 --2- 172.21.15.160:0/2385268228 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb99c0f5410 0x7fb99c109450 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:13:57.928 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:57.927+0000 7fb9a38ba700 1 -- 172.21.15.160:0/2385268228 >> 172.21.15.160:0/2385268228 conn(0x7fb99c0f0b70 msgr2=0x7fb99c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:13:57.928 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:57.927+0000 7fb9a38ba700 1 -- 172.21.15.160:0/2385268228 shutdown_connections 2023-12-25T20:13:57.928 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:13:57.927+0000 7fb9a38ba700 1 -- 172.21.15.160:0/2385268228 wait complete. 2023-12-25T20:13:57.928 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:13:58.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:13:57 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:13:56.687472+0000 mgr.smithi050.nbzhun (mgr.14162) 207 : cluster [DBG] pgmap v195: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:13:59.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:13:58 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:13:57.926683+0000 mon.smithi050 (mon.0) 269 : audit [DBG] from='client.? 172.21.15.160:0/2385268228' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:13:59.364 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:13:59.364 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:14:00.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:13:59 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:13:58.687704+0000 mgr.smithi050.nbzhun (mgr.14162) 208 : cluster [DBG] pgmap v196: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:14:00.537 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:00.536+0000 7f2ae588b700 1 -- 172.21.15.160:0/484042485 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2ae00f5410 msgr2=0x7f2ae00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:14:00.537 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:00.536+0000 7f2ae588b700 1 --2- 172.21.15.160:0/484042485 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2ae00f5410 0x7f2ae00f5830 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 rx=0x7f2ad0009fb0 tx=0x7f2ad0009b30).stop 2023-12-25T20:14:00.538 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:00.536+0000 7f2ae588b700 1 -- 172.21.15.160:0/484042485 shutdown_connections 2023-12-25T20:14:00.538 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:00.536+0000 7f2ae588b700 1 --2- 172.21.15.160:0/484042485 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2ae00f5410 0x7f2ae00f5830 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:14:00.538 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:00.536+0000 7f2ae588b700 1 -- 172.21.15.160:0/484042485 >> 172.21.15.160:0/484042485 conn(0x7f2ae00f0b70 msgr2=0x7f2ae00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:14:00.538 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:00.536+0000 7f2ae588b700 1 -- 172.21.15.160:0/484042485 shutdown_connections 2023-12-25T20:14:00.538 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:00.536+0000 7f2ae588b700 1 -- 172.21.15.160:0/484042485 wait complete. 2023-12-25T20:14:00.538 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:00.536+0000 7f2ae588b700 1 Processor -- start 2023-12-25T20:14:00.538 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:00.536+0000 7f2ae588b700 1 -- start start 2023-12-25T20:14:00.538 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:00.537+0000 7f2ae588b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2ae00f5410 0x7f2ae0105600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:14:00.538 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:00.537+0000 7f2ae588b700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ae0105af0 con 0x7f2ae00f5410 2023-12-25T20:14:00.538 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:00.537+0000 7f2adeffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2ae00f5410 0x7f2ae0105600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:14:00.538 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:00.537+0000 7f2adeffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2ae00f5410 0x7f2ae0105600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:35150/0 (socket says 172.21.15.160:35150) 2023-12-25T20:14:00.538 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:00.537+0000 7f2adeffd700 1 -- 172.21.15.160:0/2841762738 learned_addr learned my addr 172.21.15.160:0/2841762738 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:14:00.540 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:00.537+0000 7f2adeffd700 1 -- 172.21.15.160:0/2841762738 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ad0005040 con 0x7f2ae00f5410 2023-12-25T20:14:00.540 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:00.537+0000 7f2adeffd700 1 --2- 172.21.15.160:0/2841762738 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2ae00f5410 0x7f2ae0105600 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7f2ad0013510 tx=0x7f2ad000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:14:00.540 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:00.537+0000 7f2ac77fe700 1 -- 172.21.15.160:0/2841762738 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2ad0009320 con 0x7f2ae00f5410 2023-12-25T20:14:00.540 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:00.537+0000 7f2ae588b700 1 -- 172.21.15.160:0/2841762738 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ae0105cf0 con 0x7f2ae00f5410 2023-12-25T20:14:00.540 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:00.537+0000 7f2ac77fe700 1 -- 172.21.15.160:0/2841762738 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2ad0013f00 con 0x7f2ae00f5410 2023-12-25T20:14:00.540 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:00.537+0000 7f2ac77fe700 1 -- 172.21.15.160:0/2841762738 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2ad0014d90 con 0x7f2ae00f5410 2023-12-25T20:14:00.540 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:00.538+0000 7f2ae588b700 1 -- 172.21.15.160:0/2841762738 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ae0106180 con 0x7f2ae00f5410 2023-12-25T20:14:00.540 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:00.538+0000 7f2ac77fe700 1 -- 172.21.15.160:0/2841762738 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2ad00095f0 con 0x7f2ae00f5410 2023-12-25T20:14:00.540 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:00.538+0000 7f2ae588b700 1 -- 172.21.15.160:0/2841762738 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ae0045ad0 con 0x7f2ae00f5410 2023-12-25T20:14:00.540 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:00.538+0000 7f2ac77fe700 1 --2- 172.21.15.160:0/2841762738 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f2ac802ff60 0x7f2ac8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:14:00.540 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:00.538+0000 7f2ac77fe700 1 -- 172.21.15.160:0/2841762738 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2ad0041080 con 0x7f2ae00f5410 2023-12-25T20:14:00.540 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:00.539+0000 7f2ade7fc700 1 --2- 172.21.15.160:0/2841762738 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f2ac802ff60 0x7f2ac8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:14:00.540 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:00.539+0000 7f2ade7fc700 1 --2- 172.21.15.160:0/2841762738 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f2ac802ff60 0x7f2ac8032420 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f2ad40096a0 tx=0x7f2ad4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:14:00.544 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:00.543+0000 7f2ac77fe700 1 -- 172.21.15.160:0/2841762738 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2ad0014420 con 0x7f2ae00f5410 2023-12-25T20:14:00.751 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:00.750+0000 7f2ae588b700 1 -- 172.21.15.160:0/2841762738 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2ae00fae60 con 0x7f2ae00f5410 2023-12-25T20:14:00.752 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:00.751+0000 7f2ac77fe700 1 -- 172.21.15.160:0/2841762738 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2ad0010230 con 0x7f2ae00f5410 2023-12-25T20:14:00.752 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:14:00.752 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:14:00.753 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:00.753+0000 7f2ae588b700 1 -- 172.21.15.160:0/2841762738 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f2ac802ff60 msgr2=0x7f2ac8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:14:00.754 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:00.753+0000 7f2ae588b700 1 --2- 172.21.15.160:0/2841762738 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f2ac802ff60 0x7f2ac8032420 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f2ad40096a0 tx=0x7f2ad4006b40).stop 2023-12-25T20:14:00.754 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:00.753+0000 7f2ae588b700 1 -- 172.21.15.160:0/2841762738 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2ae00f5410 msgr2=0x7f2ae0105600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:14:00.754 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:00.753+0000 7f2ae588b700 1 --2- 172.21.15.160:0/2841762738 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2ae00f5410 0x7f2ae0105600 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7f2ad0013510 tx=0x7f2ad000b040).stop 2023-12-25T20:14:00.754 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:00.753+0000 7f2ae588b700 1 -- 172.21.15.160:0/2841762738 shutdown_connections 2023-12-25T20:14:00.754 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:00.753+0000 7f2ae588b700 1 --2- 172.21.15.160:0/2841762738 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f2ac802ff60 0x7f2ac8032420 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:14:00.754 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:00.753+0000 7f2ae588b700 1 --2- 172.21.15.160:0/2841762738 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2ae00f5410 0x7f2ae0105600 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:14:00.754 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:00.753+0000 7f2ae588b700 1 -- 172.21.15.160:0/2841762738 >> 172.21.15.160:0/2841762738 conn(0x7f2ae00f0b70 msgr2=0x7f2ae00f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:14:00.754 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:00.753+0000 7f2ae588b700 1 -- 172.21.15.160:0/2841762738 shutdown_connections 2023-12-25T20:14:00.754 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:00.753+0000 7f2ae588b700 1 -- 172.21.15.160:0/2841762738 wait complete. 2023-12-25T20:14:00.754 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:14:01.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:14:00 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:14:00.752931+0000 mon.smithi050 (mon.0) 270 : audit [DBG] from='client.? 172.21.15.160:0/2841762738' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:14:02.162 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:14:02.162 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:14:02.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:14:01 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:14:00.688077+0000 mgr.smithi050.nbzhun (mgr.14162) 209 : cluster [DBG] pgmap v197: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:14:03.310 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:03.309+0000 7f1be0443700 1 -- 172.21.15.160:0/771102453 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1bd80f5410 msgr2=0x7f1bd80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:14:03.310 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:03.309+0000 7f1be0443700 1 --2- 172.21.15.160:0/771102453 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1bd80f5410 0x7f1bd80f5830 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 rx=0x7f1bcc009fb0 tx=0x7f1bcc009b30).stop 2023-12-25T20:14:03.310 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:03.309+0000 7f1be0443700 1 -- 172.21.15.160:0/771102453 shutdown_connections 2023-12-25T20:14:03.310 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:03.309+0000 7f1be0443700 1 --2- 172.21.15.160:0/771102453 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1bd80f5410 0x7f1bd80f5830 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:14:03.310 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:03.309+0000 7f1be0443700 1 -- 172.21.15.160:0/771102453 >> 172.21.15.160:0/771102453 conn(0x7f1bd80f0b70 msgr2=0x7f1bd80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:14:03.310 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:03.309+0000 7f1be0443700 1 -- 172.21.15.160:0/771102453 shutdown_connections 2023-12-25T20:14:03.310 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:03.309+0000 7f1be0443700 1 -- 172.21.15.160:0/771102453 wait complete. 2023-12-25T20:14:03.312 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:03.310+0000 7f1be0443700 1 Processor -- start 2023-12-25T20:14:03.312 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:03.310+0000 7f1be0443700 1 -- start start 2023-12-25T20:14:03.313 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:03.310+0000 7f1be0443700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1bd80f5410 0x7f1bd80fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:14:03.313 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:03.310+0000 7f1be0443700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bd80fb820 con 0x7f1bd80f5410 2023-12-25T20:14:03.313 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:03.310+0000 7f1bde1df700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1bd80f5410 0x7f1bd80fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:14:03.313 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:03.310+0000 7f1bde1df700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1bd80f5410 0x7f1bd80fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:35172/0 (socket says 172.21.15.160:35172) 2023-12-25T20:14:03.313 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:03.310+0000 7f1bde1df700 1 -- 172.21.15.160:0/2004106524 learned_addr learned my addr 172.21.15.160:0/2004106524 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:14:03.313 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:03.310+0000 7f1bde1df700 1 -- 172.21.15.160:0/2004106524 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bcc005040 con 0x7f1bd80f5410 2023-12-25T20:14:03.313 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:03.311+0000 7f1bde1df700 1 --2- 172.21.15.160:0/2004106524 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1bd80f5410 0x7f1bd80fd3a0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7f1bcc013510 tx=0x7f1bcc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:14:03.313 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:03.311+0000 7f1bcaffd700 1 -- 172.21.15.160:0/2004106524 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1bcc009320 con 0x7f1bd80f5410 2023-12-25T20:14:03.313 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:03.311+0000 7f1bcaffd700 1 -- 172.21.15.160:0/2004106524 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1bcc013f00 con 0x7f1bd80f5410 2023-12-25T20:14:03.313 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:03.311+0000 7f1be0443700 1 -- 172.21.15.160:0/2004106524 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bd80fb9c0 con 0x7f1bd80f5410 2023-12-25T20:14:03.313 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:03.311+0000 7f1bcaffd700 1 -- 172.21.15.160:0/2004106524 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1bcc014d90 con 0x7f1bd80f5410 2023-12-25T20:14:03.313 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:03.311+0000 7f1be0443700 1 -- 172.21.15.160:0/2004106524 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bd80fbe50 con 0x7f1bd80f5410 2023-12-25T20:14:03.313 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:03.312+0000 7f1bcaffd700 1 -- 172.21.15.160:0/2004106524 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1bcc0095f0 con 0x7f1bd80f5410 2023-12-25T20:14:03.313 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:03.312+0000 7f1be0443700 1 -- 172.21.15.160:0/2004106524 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bd8059100 con 0x7f1bd80f5410 2023-12-25T20:14:03.314 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:03.312+0000 7f1bcaffd700 1 --2- 172.21.15.160:0/2004106524 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f1bc402ff60 0x7f1bc4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:14:03.314 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:03.312+0000 7f1bcaffd700 1 -- 172.21.15.160:0/2004106524 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1bcc0410a0 con 0x7f1bd80f5410 2023-12-25T20:14:03.314 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:03.312+0000 7f1bdd9de700 1 --2- 172.21.15.160:0/2004106524 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f1bc402ff60 0x7f1bc4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:14:03.314 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:03.312+0000 7f1bdd9de700 1 --2- 172.21.15.160:0/2004106524 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f1bc402ff60 0x7f1bc4032420 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f1bd40096a0 tx=0x7f1bd4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:14:03.317 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:03.317+0000 7f1bcaffd700 1 -- 172.21.15.160:0/2004106524 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1bcc01d7f0 con 0x7f1bd80f5410 2023-12-25T20:14:03.524 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:03.523+0000 7f1be0443700 1 -- 172.21.15.160:0/2004106524 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1bd80fae60 con 0x7f1bd80f5410 2023-12-25T20:14:03.524 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:03.524+0000 7f1bcaffd700 1 -- 172.21.15.160:0/2004106524 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1bcc010230 con 0x7f1bd80f5410 2023-12-25T20:14:03.525 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:14:03.525 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:14:03.526 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:03.525+0000 7f1be0443700 1 -- 172.21.15.160:0/2004106524 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f1bc402ff60 msgr2=0x7f1bc4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:14:03.526 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:03.525+0000 7f1be0443700 1 --2- 172.21.15.160:0/2004106524 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f1bc402ff60 0x7f1bc4032420 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f1bd40096a0 tx=0x7f1bd4006b40).stop 2023-12-25T20:14:03.526 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:03.525+0000 7f1be0443700 1 -- 172.21.15.160:0/2004106524 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1bd80f5410 msgr2=0x7f1bd80fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:14:03.526 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:03.525+0000 7f1be0443700 1 --2- 172.21.15.160:0/2004106524 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1bd80f5410 0x7f1bd80fd3a0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7f1bcc013510 tx=0x7f1bcc00b040).stop 2023-12-25T20:14:03.526 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:03.525+0000 7f1be0443700 1 -- 172.21.15.160:0/2004106524 shutdown_connections 2023-12-25T20:14:03.527 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:03.525+0000 7f1be0443700 1 --2- 172.21.15.160:0/2004106524 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f1bc402ff60 0x7f1bc4032420 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:14:03.527 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:03.525+0000 7f1be0443700 1 --2- 172.21.15.160:0/2004106524 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1bd80f5410 0x7f1bd80fd3a0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:14:03.527 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:03.525+0000 7f1be0443700 1 -- 172.21.15.160:0/2004106524 >> 172.21.15.160:0/2004106524 conn(0x7f1bd80f0b70 msgr2=0x7f1bd805fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:14:03.527 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:03.526+0000 7f1be0443700 1 -- 172.21.15.160:0/2004106524 shutdown_connections 2023-12-25T20:14:03.527 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:03.526+0000 7f1be0443700 1 -- 172.21.15.160:0/2004106524 wait complete. 2023-12-25T20:14:03.527 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:14:04.090 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:14:03 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:14:02.688468+0000 mgr.smithi050.nbzhun (mgr.14162) 210 : cluster [DBG] pgmap v198: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:14:04.091 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:14:03 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:14:03.525703+0000 mon.smithi050 (mon.0) 271 : audit [DBG] from='client.? 172.21.15.160:0/2004106524' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:14:04.898 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:14:04.899 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:14:06.110 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:06.109+0000 7f2d31c14700 1 -- 172.21.15.160:0/3266282509 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2d2c0f6a10 msgr2=0x7f2d2c0f6e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:14:06.110 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:06.109+0000 7f2d31c14700 1 --2- 172.21.15.160:0/3266282509 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2d2c0f6a10 0x7f2d2c0f6e30 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 rx=0x7f2d20009fb0 tx=0x7f2d20009b30).stop 2023-12-25T20:14:06.110 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:06.109+0000 7f2d31c14700 1 -- 172.21.15.160:0/3266282509 shutdown_connections 2023-12-25T20:14:06.110 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:06.109+0000 7f2d31c14700 1 --2- 172.21.15.160:0/3266282509 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2d2c0f6a10 0x7f2d2c0f6e30 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:14:06.110 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:06.109+0000 7f2d31c14700 1 -- 172.21.15.160:0/3266282509 >> 172.21.15.160:0/3266282509 conn(0x7f2d2c0f02e0 msgr2=0x7f2d2c0f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:14:06.111 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:06.109+0000 7f2d31c14700 1 -- 172.21.15.160:0/3266282509 shutdown_connections 2023-12-25T20:14:06.111 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:06.109+0000 7f2d31c14700 1 -- 172.21.15.160:0/3266282509 wait complete. 2023-12-25T20:14:06.111 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:06.109+0000 7f2d31c14700 1 Processor -- start 2023-12-25T20:14:06.111 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:06.109+0000 7f2d31c14700 1 -- start start 2023-12-25T20:14:06.111 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:06.110+0000 7f2d31c14700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2d2c0f6a10 0x7f2d2c1055e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:14:06.111 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:06.110+0000 7f2d31c14700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d2c105ad0 con 0x7f2d2c0f6a10 2023-12-25T20:14:06.113 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:06.110+0000 7f2d2b7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2d2c0f6a10 0x7f2d2c1055e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:14:06.113 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:06.110+0000 7f2d2b7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2d2c0f6a10 0x7f2d2c1055e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:35186/0 (socket says 172.21.15.160:35186) 2023-12-25T20:14:06.113 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:06.110+0000 7f2d2b7fe700 1 -- 172.21.15.160:0/2274860759 learned_addr learned my addr 172.21.15.160:0/2274860759 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:14:06.113 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:06.110+0000 7f2d2b7fe700 1 -- 172.21.15.160:0/2274860759 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d20005040 con 0x7f2d2c0f6a10 2023-12-25T20:14:06.113 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:06.110+0000 7f2d2b7fe700 1 --2- 172.21.15.160:0/2274860759 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2d2c0f6a10 0x7f2d2c1055e0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7f2d20014000 tx=0x7f2d2000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:14:06.113 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:06.111+0000 7f2d13fff700 1 -- 172.21.15.160:0/2274860759 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2d200095f0 con 0x7f2d2c0f6a10 2023-12-25T20:14:06.113 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:06.111+0000 7f2d13fff700 1 -- 172.21.15.160:0/2274860759 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2d20007e70 con 0x7f2d2c0f6a10 2023-12-25T20:14:06.113 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:06.111+0000 7f2d31c14700 1 -- 172.21.15.160:0/2274860759 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d2c105cd0 con 0x7f2d2c0f6a10 2023-12-25T20:14:06.113 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:06.111+0000 7f2d13fff700 1 -- 172.21.15.160:0/2274860759 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2d20014ea0 con 0x7f2d2c0f6a10 2023-12-25T20:14:06.113 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:06.111+0000 7f2d31c14700 1 -- 172.21.15.160:0/2274860759 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d2c106160 con 0x7f2d2c0f6a10 2023-12-25T20:14:06.113 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:06.111+0000 7f2d13fff700 1 -- 172.21.15.160:0/2274860759 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2d20025070 con 0x7f2d2c0f6a10 2023-12-25T20:14:06.113 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:06.111+0000 7f2d31c14700 1 -- 172.21.15.160:0/2274860759 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d2c045ad0 con 0x7f2d2c0f6a10 2023-12-25T20:14:06.113 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:06.112+0000 7f2d13fff700 1 --2- 172.21.15.160:0/2274860759 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f2d1402ff60 0x7f2d14032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:14:06.114 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:06.112+0000 7f2d13fff700 1 -- 172.21.15.160:0/2274860759 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2d20040930 con 0x7f2d2c0f6a10 2023-12-25T20:14:06.114 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:06.112+0000 7f2d2affd700 1 --2- 172.21.15.160:0/2274860759 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f2d1402ff60 0x7f2d14032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:14:06.114 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:06.112+0000 7f2d2affd700 1 --2- 172.21.15.160:0/2274860759 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f2d1402ff60 0x7f2d14032420 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f2d1c0096a0 tx=0x7f2d1c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:14:06.117 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:06.116+0000 7f2d13fff700 1 -- 172.21.15.160:0/2274860759 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2d2000f4e0 con 0x7f2d2c0f6a10 2023-12-25T20:14:06.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:14:05 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:14:04.688810+0000 mgr.smithi050.nbzhun (mgr.14162) 211 : cluster [DBG] pgmap v199: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:14:06.320 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:06.320+0000 7f2d31c14700 1 -- 172.21.15.160:0/2274860759 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2d2c107010 con 0x7f2d2c0f6a10 2023-12-25T20:14:06.321 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:06.320+0000 7f2d13fff700 1 -- 172.21.15.160:0/2274860759 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2d2000f300 con 0x7f2d2c0f6a10 2023-12-25T20:14:06.321 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:14:06.321 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:14:06.323 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:06.322+0000 7f2d31c14700 1 -- 172.21.15.160:0/2274860759 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f2d1402ff60 msgr2=0x7f2d14032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:14:06.323 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:06.322+0000 7f2d31c14700 1 --2- 172.21.15.160:0/2274860759 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f2d1402ff60 0x7f2d14032420 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f2d1c0096a0 tx=0x7f2d1c006b40).stop 2023-12-25T20:14:06.323 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:06.322+0000 7f2d31c14700 1 -- 172.21.15.160:0/2274860759 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2d2c0f6a10 msgr2=0x7f2d2c1055e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:14:06.323 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:06.322+0000 7f2d31c14700 1 --2- 172.21.15.160:0/2274860759 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2d2c0f6a10 0x7f2d2c1055e0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7f2d20014000 tx=0x7f2d2000b040).stop 2023-12-25T20:14:06.323 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:06.322+0000 7f2d31c14700 1 -- 172.21.15.160:0/2274860759 shutdown_connections 2023-12-25T20:14:06.323 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:06.322+0000 7f2d31c14700 1 --2- 172.21.15.160:0/2274860759 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f2d1402ff60 0x7f2d14032420 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:14:06.323 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:06.322+0000 7f2d31c14700 1 --2- 172.21.15.160:0/2274860759 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2d2c0f6a10 0x7f2d2c1055e0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:14:06.323 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:06.322+0000 7f2d31c14700 1 -- 172.21.15.160:0/2274860759 >> 172.21.15.160:0/2274860759 conn(0x7f2d2c0f02e0 msgr2=0x7f2d2c0f0d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:14:06.324 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:06.322+0000 7f2d31c14700 1 -- 172.21.15.160:0/2274860759 shutdown_connections 2023-12-25T20:14:06.324 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:06.322+0000 7f2d31c14700 1 -- 172.21.15.160:0/2274860759 wait complete. 2023-12-25T20:14:06.324 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:14:07.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:14:06 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:14:06.322087+0000 mon.smithi050 (mon.0) 272 : audit [DBG] from='client.? 172.21.15.160:0/2274860759' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:14:07.725 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:14:07.725 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:14:08.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:14:07 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:14:06.689161+0000 mgr.smithi050.nbzhun (mgr.14162) 212 : cluster [DBG] pgmap v200: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:14:08.853 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:08.852+0000 7f23e3413700 1 -- 172.21.15.160:0/2426908804 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f23dc0f6510 msgr2=0x7f23dc0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:14:08.853 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:08.852+0000 7f23e3413700 1 --2- 172.21.15.160:0/2426908804 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f23dc0f6510 0x7f23dc0f6930 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 rx=0x7f23cc009fb0 tx=0x7f23cc009b30).stop 2023-12-25T20:14:08.854 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:08.852+0000 7f23e3413700 1 -- 172.21.15.160:0/2426908804 shutdown_connections 2023-12-25T20:14:08.854 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:08.852+0000 7f23e3413700 1 --2- 172.21.15.160:0/2426908804 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f23dc0f6510 0x7f23dc0f6930 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:14:08.854 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:08.852+0000 7f23e3413700 1 -- 172.21.15.160:0/2426908804 >> 172.21.15.160:0/2426908804 conn(0x7f23dc0f1c70 msgr2=0x7f23dc0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:14:08.854 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:08.852+0000 7f23e3413700 1 -- 172.21.15.160:0/2426908804 shutdown_connections 2023-12-25T20:14:08.854 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:08.852+0000 7f23e3413700 1 -- 172.21.15.160:0/2426908804 wait complete. 2023-12-25T20:14:08.854 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:08.853+0000 7f23e3413700 1 Processor -- start 2023-12-25T20:14:08.854 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:08.853+0000 7f23e3413700 1 -- start start 2023-12-25T20:14:08.854 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:08.853+0000 7f23e3413700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f23dc0f6510 0x7f23dc106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:14:08.854 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:08.853+0000 7f23e3413700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23dc106c20 con 0x7f23dc0f6510 2023-12-25T20:14:08.856 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:08.853+0000 7f23e11af700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f23dc0f6510 0x7f23dc106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:14:08.856 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:08.853+0000 7f23e11af700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f23dc0f6510 0x7f23dc106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:44426/0 (socket says 172.21.15.160:44426) 2023-12-25T20:14:08.856 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:08.853+0000 7f23e11af700 1 -- 172.21.15.160:0/586632627 learned_addr learned my addr 172.21.15.160:0/586632627 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:14:08.856 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:08.853+0000 7f23e11af700 1 -- 172.21.15.160:0/586632627 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23cc005040 con 0x7f23dc0f6510 2023-12-25T20:14:08.856 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:08.854+0000 7f23e11af700 1 --2- 172.21.15.160:0/586632627 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f23dc0f6510 0x7f23dc106730 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7f23cc013510 tx=0x7f23cc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:14:08.856 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:08.854+0000 7f23d1ffb700 1 -- 172.21.15.160:0/586632627 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f23cc009320 con 0x7f23dc0f6510 2023-12-25T20:14:08.856 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:08.854+0000 7f23d1ffb700 1 -- 172.21.15.160:0/586632627 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f23cc013f00 con 0x7f23dc0f6510 2023-12-25T20:14:08.856 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:08.854+0000 7f23e3413700 1 -- 172.21.15.160:0/586632627 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23dc106e20 con 0x7f23dc0f6510 2023-12-25T20:14:08.856 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:08.854+0000 7f23d1ffb700 1 -- 172.21.15.160:0/586632627 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f23cc014d90 con 0x7f23dc0f6510 2023-12-25T20:14:08.856 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:08.854+0000 7f23e3413700 1 -- 172.21.15.160:0/586632627 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23dc1072b0 con 0x7f23dc0f6510 2023-12-25T20:14:08.856 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:08.855+0000 7f23e3413700 1 -- 172.21.15.160:0/586632627 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23dc045ad0 con 0x7f23dc0f6510 2023-12-25T20:14:08.856 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:08.855+0000 7f23d1ffb700 1 -- 172.21.15.160:0/586632627 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f23cc0095f0 con 0x7f23dc0f6510 2023-12-25T20:14:08.856 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:08.855+0000 7f23d1ffb700 1 --2- 172.21.15.160:0/586632627 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f23c802ff60 0x7f23c8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:14:08.857 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:08.855+0000 7f23d1ffb700 1 -- 172.21.15.160:0/586632627 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f23cc040fc0 con 0x7f23dc0f6510 2023-12-25T20:14:08.857 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:08.855+0000 7f23e09ae700 1 --2- 172.21.15.160:0/586632627 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f23c802ff60 0x7f23c8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:14:08.857 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:08.856+0000 7f23e09ae700 1 --2- 172.21.15.160:0/586632627 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f23c802ff60 0x7f23c8032420 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f23d80096a0 tx=0x7f23d8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:14:08.860 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:08.860+0000 7f23d1ffb700 1 -- 172.21.15.160:0/586632627 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f23cc014420 con 0x7f23dc0f6510 2023-12-25T20:14:09.065 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:09.065+0000 7f23e3413700 1 -- 172.21.15.160:0/586632627 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f23dc0fbf60 con 0x7f23dc0f6510 2023-12-25T20:14:09.066 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:09.065+0000 7f23d1ffb700 1 -- 172.21.15.160:0/586632627 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f23cc010230 con 0x7f23dc0f6510 2023-12-25T20:14:09.066 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:14:09.066 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:14:09.068 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:09.067+0000 7f23e3413700 1 -- 172.21.15.160:0/586632627 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f23c802ff60 msgr2=0x7f23c8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:14:09.068 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:09.067+0000 7f23e3413700 1 --2- 172.21.15.160:0/586632627 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f23c802ff60 0x7f23c8032420 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f23d80096a0 tx=0x7f23d8006b40).stop 2023-12-25T20:14:09.068 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:09.067+0000 7f23e3413700 1 -- 172.21.15.160:0/586632627 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f23dc0f6510 msgr2=0x7f23dc106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:14:09.068 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:09.067+0000 7f23e3413700 1 --2- 172.21.15.160:0/586632627 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f23dc0f6510 0x7f23dc106730 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7f23cc013510 tx=0x7f23cc00b040).stop 2023-12-25T20:14:09.068 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:09.067+0000 7f23e3413700 1 -- 172.21.15.160:0/586632627 shutdown_connections 2023-12-25T20:14:09.068 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:09.067+0000 7f23e3413700 1 --2- 172.21.15.160:0/586632627 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f23c802ff60 0x7f23c8032420 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:14:09.068 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:09.067+0000 7f23e3413700 1 --2- 172.21.15.160:0/586632627 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f23dc0f6510 0x7f23dc106730 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:14:09.068 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:09.067+0000 7f23e3413700 1 -- 172.21.15.160:0/586632627 >> 172.21.15.160:0/586632627 conn(0x7f23dc0f1c70 msgr2=0x7f23dc0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:14:09.068 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:09.067+0000 7f23e3413700 1 -- 172.21.15.160:0/586632627 shutdown_connections 2023-12-25T20:14:09.068 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:09.067+0000 7f23e3413700 1 -- 172.21.15.160:0/586632627 wait complete. 2023-12-25T20:14:09.069 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:14:10.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:14:09 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:14:08.689556+0000 mgr.smithi050.nbzhun (mgr.14162) 213 : cluster [DBG] pgmap v201: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:14:10.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:14:09 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:14:09.067153+0000 mon.smithi050 (mon.0) 273 : audit [DBG] from='client.? 172.21.15.160:0/586632627' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:14:10.441 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:14:10.441 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:14:11.601 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:11.598+0000 7f4297fff700 1 -- 172.21.15.160:0/2538750357 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f42980f62a0 msgr2=0x7f42980f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:14:11.603 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:11.598+0000 7f4297fff700 1 --2- 172.21.15.160:0/2538750357 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f42980f62a0 0x7f42980f66c0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 rx=0x7f4288009fb0 tx=0x7f4288009b30).stop 2023-12-25T20:14:11.603 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:11.599+0000 7f4297fff700 1 -- 172.21.15.160:0/2538750357 shutdown_connections 2023-12-25T20:14:11.603 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:11.599+0000 7f4297fff700 1 --2- 172.21.15.160:0/2538750357 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f42980f62a0 0x7f42980f66c0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:14:11.603 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:11.599+0000 7f4297fff700 1 -- 172.21.15.160:0/2538750357 >> 172.21.15.160:0/2538750357 conn(0x7f42980f1aa0 msgr2=0x7f42980f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:14:11.603 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:11.599+0000 7f4297fff700 1 -- 172.21.15.160:0/2538750357 shutdown_connections 2023-12-25T20:14:11.603 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:11.599+0000 7f4297fff700 1 -- 172.21.15.160:0/2538750357 wait complete. 2023-12-25T20:14:11.603 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:11.599+0000 7f4297fff700 1 Processor -- start 2023-12-25T20:14:11.603 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:11.599+0000 7f4297fff700 1 -- start start 2023-12-25T20:14:11.603 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:11.599+0000 7f4297fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f42980f62a0 0x7f429806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:14:11.603 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:11.599+0000 7f4297fff700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4298069e20 con 0x7f42980f62a0 2023-12-25T20:14:11.603 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:11.600+0000 7f4296ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f42980f62a0 0x7f429806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:14:11.603 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:11.600+0000 7f4296ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f42980f62a0 0x7f429806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:44446/0 (socket says 172.21.15.160:44446) 2023-12-25T20:14:11.604 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:11.600+0000 7f4296ffd700 1 -- 172.21.15.160:0/4070052998 learned_addr learned my addr 172.21.15.160:0/4070052998 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:14:11.604 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:11.600+0000 7f4296ffd700 1 -- 172.21.15.160:0/4070052998 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4288005040 con 0x7f42980f62a0 2023-12-25T20:14:11.604 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:11.600+0000 7f4296ffd700 1 --2- 172.21.15.160:0/4070052998 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f42980f62a0 0x7f429806b9a0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7f4288013510 tx=0x7f428800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:14:11.604 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:11.600+0000 7f427f7fe700 1 -- 172.21.15.160:0/4070052998 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4288009320 con 0x7f42980f62a0 2023-12-25T20:14:11.604 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:11.600+0000 7f427f7fe700 1 -- 172.21.15.160:0/4070052998 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4288013f00 con 0x7f42980f62a0 2023-12-25T20:14:11.604 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:11.600+0000 7f4297fff700 1 -- 172.21.15.160:0/4070052998 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4298069fc0 con 0x7f42980f62a0 2023-12-25T20:14:11.604 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:11.600+0000 7f427f7fe700 1 -- 172.21.15.160:0/4070052998 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4288014dc0 con 0x7f42980f62a0 2023-12-25T20:14:11.604 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:11.600+0000 7f4297fff700 1 -- 172.21.15.160:0/4070052998 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f429806a450 con 0x7f42980f62a0 2023-12-25T20:14:11.605 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:11.601+0000 7f427f7fe700 1 -- 172.21.15.160:0/4070052998 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f42880095f0 con 0x7f42980f62a0 2023-12-25T20:14:11.605 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:11.601+0000 7f4297fff700 1 -- 172.21.15.160:0/4070052998 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4298059100 con 0x7f42980f62a0 2023-12-25T20:14:11.606 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:11.601+0000 7f427f7fe700 1 --2- 172.21.15.160:0/4070052998 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f428002ffb0 0x7f4280032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:14:11.606 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:11.602+0000 7f427f7fe700 1 -- 172.21.15.160:0/4070052998 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4288041100 con 0x7f42980f62a0 2023-12-25T20:14:11.606 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:11.602+0000 7f42967fc700 1 --2- 172.21.15.160:0/4070052998 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f428002ffb0 0x7f4280032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:14:11.606 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:11.602+0000 7f42967fc700 1 --2- 172.21.15.160:0/4070052998 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f428002ffb0 0x7f4280032470 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f428c0096a0 tx=0x7f428c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:14:11.607 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:11.606+0000 7f427f7fe700 1 -- 172.21.15.160:0/4070052998 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4288014420 con 0x7f42980f62a0 2023-12-25T20:14:11.816 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:11.815+0000 7f4297fff700 1 -- 172.21.15.160:0/4070052998 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f42980fbce0 con 0x7f42980f62a0 2023-12-25T20:14:11.817 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:11.816+0000 7f427f7fe700 1 -- 172.21.15.160:0/4070052998 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4288010230 con 0x7f42980f62a0 2023-12-25T20:14:11.817 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:14:11.817 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:14:11.818 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:11.818+0000 7f4297fff700 1 -- 172.21.15.160:0/4070052998 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f428002ffb0 msgr2=0x7f4280032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:14:11.819 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:11.818+0000 7f4297fff700 1 --2- 172.21.15.160:0/4070052998 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f428002ffb0 0x7f4280032470 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f428c0096a0 tx=0x7f428c006b40).stop 2023-12-25T20:14:11.819 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:11.818+0000 7f4297fff700 1 -- 172.21.15.160:0/4070052998 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f42980f62a0 msgr2=0x7f429806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:14:11.819 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:11.818+0000 7f4297fff700 1 --2- 172.21.15.160:0/4070052998 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f42980f62a0 0x7f429806b9a0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7f4288013510 tx=0x7f428800b040).stop 2023-12-25T20:14:11.819 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:11.818+0000 7f4297fff700 1 -- 172.21.15.160:0/4070052998 shutdown_connections 2023-12-25T20:14:11.819 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:11.818+0000 7f4297fff700 1 --2- 172.21.15.160:0/4070052998 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f428002ffb0 0x7f4280032470 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:14:11.819 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:11.818+0000 7f4297fff700 1 --2- 172.21.15.160:0/4070052998 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f42980f62a0 0x7f429806b9a0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:14:11.819 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:11.818+0000 7f4297fff700 1 -- 172.21.15.160:0/4070052998 >> 172.21.15.160:0/4070052998 conn(0x7f42980f1aa0 msgr2=0x7f42980f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:14:11.819 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:11.818+0000 7f4297fff700 1 -- 172.21.15.160:0/4070052998 shutdown_connections 2023-12-25T20:14:11.819 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:11.818+0000 7f4297fff700 1 -- 172.21.15.160:0/4070052998 wait complete. 2023-12-25T20:14:11.820 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:14:12.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:14:11 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:14:10.689976+0000 mgr.smithi050.nbzhun (mgr.14162) 214 : cluster [DBG] pgmap v202: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:14:12.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:14:11 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:14:11.817905+0000 mon.smithi050 (mon.0) 274 : audit [DBG] from='client.? 172.21.15.160:0/4070052998' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:14:13.193 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:14:13.193 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:14:14.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:14:13 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:14:12.690311+0000 mgr.smithi050.nbzhun (mgr.14162) 215 : cluster [DBG] pgmap v203: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:14:14.363 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:14.361+0000 7f93e5b50700 1 -- 172.21.15.160:0/4256651419 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f93e00f5410 msgr2=0x7f93e00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:14:14.363 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:14.361+0000 7f93e5b50700 1 --2- 172.21.15.160:0/4256651419 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f93e00f5410 0x7f93e00f5830 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 rx=0x7f93d4009fb0 tx=0x7f93d4009b30).stop 2023-12-25T20:14:14.364 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:14.361+0000 7f93e5b50700 1 -- 172.21.15.160:0/4256651419 shutdown_connections 2023-12-25T20:14:14.364 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:14.361+0000 7f93e5b50700 1 --2- 172.21.15.160:0/4256651419 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f93e00f5410 0x7f93e00f5830 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:14:14.364 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:14.361+0000 7f93e5b50700 1 -- 172.21.15.160:0/4256651419 >> 172.21.15.160:0/4256651419 conn(0x7f93e00f0b70 msgr2=0x7f93e00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:14:14.364 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:14.361+0000 7f93e5b50700 1 -- 172.21.15.160:0/4256651419 shutdown_connections 2023-12-25T20:14:14.364 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:14.361+0000 7f93e5b50700 1 -- 172.21.15.160:0/4256651419 wait complete. 2023-12-25T20:14:14.364 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:14.362+0000 7f93e5b50700 1 Processor -- start 2023-12-25T20:14:14.364 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:14.362+0000 7f93e5b50700 1 -- start start 2023-12-25T20:14:14.364 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:14.362+0000 7f93e5b50700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f93e00f5410 0x7f93e0105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:14:14.364 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:14.362+0000 7f93e5b50700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93e0105b20 con 0x7f93e00f5410 2023-12-25T20:14:14.364 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:14.362+0000 7f93df7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f93e00f5410 0x7f93e0105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:14:14.364 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:14.362+0000 7f93df7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f93e00f5410 0x7f93e0105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:44456/0 (socket says 172.21.15.160:44456) 2023-12-25T20:14:14.364 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:14.362+0000 7f93df7fe700 1 -- 172.21.15.160:0/1794770687 learned_addr learned my addr 172.21.15.160:0/1794770687 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:14:14.364 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:14.362+0000 7f93df7fe700 1 -- 172.21.15.160:0/1794770687 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93d4005040 con 0x7f93e00f5410 2023-12-25T20:14:14.365 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:14.363+0000 7f93df7fe700 1 --2- 172.21.15.160:0/1794770687 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f93e00f5410 0x7f93e0105630 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7f93d4013510 tx=0x7f93d400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:14:14.365 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:14.363+0000 7f93c7fff700 1 -- 172.21.15.160:0/1794770687 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f93d4009320 con 0x7f93e00f5410 2023-12-25T20:14:14.365 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:14.363+0000 7f93e5b50700 1 -- 172.21.15.160:0/1794770687 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93e0105d20 con 0x7f93e00f5410 2023-12-25T20:14:14.365 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:14.363+0000 7f93c7fff700 1 -- 172.21.15.160:0/1794770687 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f93d4013f00 con 0x7f93e00f5410 2023-12-25T20:14:14.365 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:14.363+0000 7f93c7fff700 1 -- 172.21.15.160:0/1794770687 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f93d4014d90 con 0x7f93e00f5410 2023-12-25T20:14:14.365 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:14.363+0000 7f93e5b50700 1 -- 172.21.15.160:0/1794770687 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93e01061b0 con 0x7f93e00f5410 2023-12-25T20:14:14.366 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:14.364+0000 7f93c7fff700 1 -- 172.21.15.160:0/1794770687 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f93d40095f0 con 0x7f93e00f5410 2023-12-25T20:14:14.366 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:14.364+0000 7f93e5b50700 1 -- 172.21.15.160:0/1794770687 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93e0045ad0 con 0x7f93e00f5410 2023-12-25T20:14:14.366 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:14.364+0000 7f93c7fff700 1 --2- 172.21.15.160:0/1794770687 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f93c802ff60 0x7f93c8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:14:14.366 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:14.364+0000 7f93c7fff700 1 -- 172.21.15.160:0/1794770687 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f93d4041000 con 0x7f93e00f5410 2023-12-25T20:14:14.366 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:14.364+0000 7f93deffd700 1 --2- 172.21.15.160:0/1794770687 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f93c802ff60 0x7f93c8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:14:14.366 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:14.364+0000 7f93deffd700 1 --2- 172.21.15.160:0/1794770687 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f93c802ff60 0x7f93c8032420 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f93d00096a0 tx=0x7f93d0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:14:14.369 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:14.368+0000 7f93c7fff700 1 -- 172.21.15.160:0/1794770687 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f93d4014420 con 0x7f93e00f5410 2023-12-25T20:14:14.575 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:14.575+0000 7f93e5b50700 1 -- 172.21.15.160:0/1794770687 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f93e00fae60 con 0x7f93e00f5410 2023-12-25T20:14:14.576 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:14.575+0000 7f93c7fff700 1 -- 172.21.15.160:0/1794770687 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f93d4010230 con 0x7f93e00f5410 2023-12-25T20:14:14.576 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:14:14.576 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:14:14.578 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:14.577+0000 7f93e5b50700 1 -- 172.21.15.160:0/1794770687 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f93c802ff60 msgr2=0x7f93c8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:14:14.578 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:14.577+0000 7f93e5b50700 1 --2- 172.21.15.160:0/1794770687 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f93c802ff60 0x7f93c8032420 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f93d00096a0 tx=0x7f93d0006b40).stop 2023-12-25T20:14:14.578 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:14.577+0000 7f93e5b50700 1 -- 172.21.15.160:0/1794770687 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f93e00f5410 msgr2=0x7f93e0105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:14:14.578 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:14.577+0000 7f93e5b50700 1 --2- 172.21.15.160:0/1794770687 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f93e00f5410 0x7f93e0105630 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7f93d4013510 tx=0x7f93d400b040).stop 2023-12-25T20:14:14.578 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:14.577+0000 7f93e5b50700 1 -- 172.21.15.160:0/1794770687 shutdown_connections 2023-12-25T20:14:14.578 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:14.577+0000 7f93e5b50700 1 --2- 172.21.15.160:0/1794770687 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f93c802ff60 0x7f93c8032420 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:14:14.578 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:14.577+0000 7f93e5b50700 1 --2- 172.21.15.160:0/1794770687 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f93e00f5410 0x7f93e0105630 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:14:14.578 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:14.577+0000 7f93e5b50700 1 -- 172.21.15.160:0/1794770687 >> 172.21.15.160:0/1794770687 conn(0x7f93e00f0b70 msgr2=0x7f93e00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:14:14.578 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:14.577+0000 7f93e5b50700 1 -- 172.21.15.160:0/1794770687 shutdown_connections 2023-12-25T20:14:14.578 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:14.577+0000 7f93e5b50700 1 -- 172.21.15.160:0/1794770687 wait complete. 2023-12-25T20:14:14.579 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:14:15.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:14:14 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:14:14.577042+0000 mon.smithi050 (mon.0) 275 : audit [DBG] from='client.? 172.21.15.160:0/1794770687' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:14:15.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:14:14 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:14:14.769248+0000 mon.smithi050 (mon.0) 276 : audit [INF] from='mgr.14162 172.21.15.50:0/3178367813' entity='mgr.smithi050.nbzhun' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi050.nbzhun/mirror_snapshot_schedule"}]: dispatch 2023-12-25T20:14:15.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:14:14 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:14:14.769458+0000 mon.smithi050 (mon.0) 277 : audit [INF] from='mgr.14162 172.21.15.50:0/3178367813' entity='mgr.smithi050.nbzhun' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi050.nbzhun/trash_purge_schedule"}]: dispatch 2023-12-25T20:14:16.009 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:14:16.010 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:14:16.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:14:15 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:14:14.690569+0000 mgr.smithi050.nbzhun (mgr.14162) 216 : cluster [DBG] pgmap v204: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:14:17.189 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:17.186+0000 7f966d31e700 1 -- 172.21.15.160:0/1356577477 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f96680f5410 msgr2=0x7f96680f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:14:17.189 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:17.186+0000 7f966d31e700 1 --2- 172.21.15.160:0/1356577477 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f96680f5410 0x7f96680f5830 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 rx=0x7f9658009fb0 tx=0x7f9658009b30).stop 2023-12-25T20:14:17.189 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:17.186+0000 7f966d31e700 1 -- 172.21.15.160:0/1356577477 shutdown_connections 2023-12-25T20:14:17.189 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:17.186+0000 7f966d31e700 1 --2- 172.21.15.160:0/1356577477 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f96680f5410 0x7f96680f5830 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:14:17.189 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:17.186+0000 7f966d31e700 1 -- 172.21.15.160:0/1356577477 >> 172.21.15.160:0/1356577477 conn(0x7f96680f0b70 msgr2=0x7f96680f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:14:17.189 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:17.186+0000 7f966d31e700 1 -- 172.21.15.160:0/1356577477 shutdown_connections 2023-12-25T20:14:17.189 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:17.186+0000 7f966d31e700 1 -- 172.21.15.160:0/1356577477 wait complete. 2023-12-25T20:14:17.189 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:17.187+0000 7f966d31e700 1 Processor -- start 2023-12-25T20:14:17.189 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:17.187+0000 7f966d31e700 1 -- start start 2023-12-25T20:14:17.189 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:17.187+0000 7f966d31e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f96680f5410 0x7f9668105600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:14:17.190 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:17.187+0000 7f966d31e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9668105af0 con 0x7f96680f5410 2023-12-25T20:14:17.190 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:17.187+0000 7f9666ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f96680f5410 0x7f9668105600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:14:17.190 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:17.187+0000 7f9666ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f96680f5410 0x7f9668105600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:44484/0 (socket says 172.21.15.160:44484) 2023-12-25T20:14:17.190 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:17.187+0000 7f9666ffd700 1 -- 172.21.15.160:0/2699452334 learned_addr learned my addr 172.21.15.160:0/2699452334 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:14:17.190 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:17.188+0000 7f9666ffd700 1 -- 172.21.15.160:0/2699452334 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9658005040 con 0x7f96680f5410 2023-12-25T20:14:17.190 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:17.188+0000 7f9666ffd700 1 --2- 172.21.15.160:0/2699452334 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f96680f5410 0x7f9668105600 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7f9658013510 tx=0x7f965800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:14:17.190 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:17.188+0000 7f965f7fe700 1 -- 172.21.15.160:0/2699452334 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9658009320 con 0x7f96680f5410 2023-12-25T20:14:17.190 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:17.188+0000 7f966d31e700 1 -- 172.21.15.160:0/2699452334 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9668105cf0 con 0x7f96680f5410 2023-12-25T20:14:17.190 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:17.188+0000 7f965f7fe700 1 -- 172.21.15.160:0/2699452334 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9658013f00 con 0x7f96680f5410 2023-12-25T20:14:17.190 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:17.188+0000 7f965f7fe700 1 -- 172.21.15.160:0/2699452334 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9658014d90 con 0x7f96680f5410 2023-12-25T20:14:17.190 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:17.188+0000 7f966d31e700 1 -- 172.21.15.160:0/2699452334 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9668106180 con 0x7f96680f5410 2023-12-25T20:14:17.191 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:17.189+0000 7f965f7fe700 1 -- 172.21.15.160:0/2699452334 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f96580095f0 con 0x7f96680f5410 2023-12-25T20:14:17.191 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:17.189+0000 7f966d31e700 1 -- 172.21.15.160:0/2699452334 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9668045ad0 con 0x7f96680f5410 2023-12-25T20:14:17.192 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:17.189+0000 7f965f7fe700 1 --2- 172.21.15.160:0/2699452334 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f965402ff60 0x7f9654032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:14:17.192 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:17.189+0000 7f965f7fe700 1 -- 172.21.15.160:0/2699452334 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9658041090 con 0x7f96680f5410 2023-12-25T20:14:17.192 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:17.189+0000 7f96667fc700 1 --2- 172.21.15.160:0/2699452334 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f965402ff60 0x7f9654032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:14:17.192 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:17.190+0000 7f96667fc700 1 --2- 172.21.15.160:0/2699452334 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f965402ff60 0x7f9654032420 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f96500096a0 tx=0x7f9650006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:14:17.194 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:17.194+0000 7f965f7fe700 1 -- 172.21.15.160:0/2699452334 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9658014420 con 0x7f96680f5410 2023-12-25T20:14:17.401 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:17.400+0000 7f966d31e700 1 -- 172.21.15.160:0/2699452334 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f96680fae60 con 0x7f96680f5410 2023-12-25T20:14:17.401 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:17.402+0000 7f965f7fe700 1 -- 172.21.15.160:0/2699452334 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f965801d7f0 con 0x7f96680f5410 2023-12-25T20:14:17.403 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:14:17.403 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:14:17.404 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:17.402+0000 7f966d31e700 1 -- 172.21.15.160:0/2699452334 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f965402ff60 msgr2=0x7f9654032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:14:17.404 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:17.402+0000 7f966d31e700 1 --2- 172.21.15.160:0/2699452334 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f965402ff60 0x7f9654032420 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f96500096a0 tx=0x7f9650006b40).stop 2023-12-25T20:14:17.404 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:17.402+0000 7f966d31e700 1 -- 172.21.15.160:0/2699452334 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f96680f5410 msgr2=0x7f9668105600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:14:17.404 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:17.402+0000 7f966d31e700 1 --2- 172.21.15.160:0/2699452334 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f96680f5410 0x7f9668105600 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7f9658013510 tx=0x7f965800b040).stop 2023-12-25T20:14:17.404 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:17.402+0000 7f966d31e700 1 -- 172.21.15.160:0/2699452334 shutdown_connections 2023-12-25T20:14:17.404 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:17.402+0000 7f966d31e700 1 --2- 172.21.15.160:0/2699452334 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f965402ff60 0x7f9654032420 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:14:17.404 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:17.402+0000 7f966d31e700 1 --2- 172.21.15.160:0/2699452334 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f96680f5410 0x7f9668105600 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:14:17.404 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:17.402+0000 7f966d31e700 1 -- 172.21.15.160:0/2699452334 >> 172.21.15.160:0/2699452334 conn(0x7f96680f0b70 msgr2=0x7f96680f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:14:17.404 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:17.402+0000 7f966d31e700 1 -- 172.21.15.160:0/2699452334 shutdown_connections 2023-12-25T20:14:17.404 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:17.402+0000 7f966d31e700 1 -- 172.21.15.160:0/2699452334 wait complete. 2023-12-25T20:14:17.404 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:14:18.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:14:17 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:14:16.690786+0000 mgr.smithi050.nbzhun (mgr.14162) 217 : cluster [DBG] pgmap v205: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:14:18.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:14:17 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:14:17.402512+0000 mon.smithi050 (mon.0) 278 : audit [DBG] from='client.? 172.21.15.160:0/2699452334' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:14:18.826 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:14:18.827 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:14:20.138 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:20.133+0000 7fb04aab6700 1 -- 172.21.15.160:0/2430332721 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb0440f47c0 msgr2=0x7fb0440f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:14:20.138 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:20.133+0000 7fb04aab6700 1 --2- 172.21.15.160:0/2430332721 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb0440f47c0 0x7fb0440f4be0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 rx=0x7fb038009fb0 tx=0x7fb038009b30).stop 2023-12-25T20:14:20.139 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:20.133+0000 7fb04aab6700 1 -- 172.21.15.160:0/2430332721 shutdown_connections 2023-12-25T20:14:20.139 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:20.133+0000 7fb04aab6700 1 --2- 172.21.15.160:0/2430332721 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb0440f47c0 0x7fb0440f4be0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:14:20.139 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:20.133+0000 7fb04aab6700 1 -- 172.21.15.160:0/2430332721 >> 172.21.15.160:0/2430332721 conn(0x7fb0440f02e0 msgr2=0x7fb0440f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:14:20.139 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:20.134+0000 7fb04aab6700 1 -- 172.21.15.160:0/2430332721 shutdown_connections 2023-12-25T20:14:20.139 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:20.134+0000 7fb04aab6700 1 -- 172.21.15.160:0/2430332721 wait complete. 2023-12-25T20:14:20.139 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:20.134+0000 7fb04aab6700 1 Processor -- start 2023-12-25T20:14:20.139 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:20.134+0000 7fb04aab6700 1 -- start start 2023-12-25T20:14:20.139 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:20.134+0000 7fb04aab6700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb0440f47c0 0x7fb0440f7370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:14:20.139 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:20.134+0000 7fb04aab6700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0440f57f0 con 0x7fb0440f47c0 2023-12-25T20:14:20.139 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:20.135+0000 7fb048852700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb0440f47c0 0x7fb0440f7370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:14:20.139 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:20.135+0000 7fb048852700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb0440f47c0 0x7fb0440f7370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:41562/0 (socket says 172.21.15.160:41562) 2023-12-25T20:14:20.139 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:20.135+0000 7fb048852700 1 -- 172.21.15.160:0/865649684 learned_addr learned my addr 172.21.15.160:0/865649684 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:14:20.139 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:20.135+0000 7fb048852700 1 -- 172.21.15.160:0/865649684 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb038005040 con 0x7fb0440f47c0 2023-12-25T20:14:20.140 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:20.135+0000 7fb048852700 1 --2- 172.21.15.160:0/865649684 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb0440f47c0 0x7fb0440f7370 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7fb038013510 tx=0x7fb03800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:14:20.140 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:20.135+0000 7fb0417fa700 1 -- 172.21.15.160:0/865649684 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb038009320 con 0x7fb0440f47c0 2023-12-25T20:14:20.140 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:20.135+0000 7fb0417fa700 1 -- 172.21.15.160:0/865649684 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb038013f00 con 0x7fb0440f47c0 2023-12-25T20:14:20.140 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:20.135+0000 7fb0417fa700 1 -- 172.21.15.160:0/865649684 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb038014d90 con 0x7fb0440f47c0 2023-12-25T20:14:20.140 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:20.135+0000 7fb04aab6700 1 -- 172.21.15.160:0/865649684 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0440f5990 con 0x7fb0440f47c0 2023-12-25T20:14:20.140 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:20.135+0000 7fb04aab6700 1 -- 172.21.15.160:0/865649684 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0440f5e20 con 0x7fb0440f47c0 2023-12-25T20:14:20.140 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:20.136+0000 7fb0417fa700 1 -- 172.21.15.160:0/865649684 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb0380095f0 con 0x7fb0440f47c0 2023-12-25T20:14:20.140 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:20.136+0000 7fb04aab6700 1 -- 172.21.15.160:0/865649684 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb044059100 con 0x7fb0440f47c0 2023-12-25T20:14:20.140 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:20.136+0000 7fb0417fa700 1 --2- 172.21.15.160:0/865649684 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fb03402ff60 0x7fb034032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:14:20.140 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:20.136+0000 7fb0417fa700 1 -- 172.21.15.160:0/865649684 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb038041020 con 0x7fb0440f47c0 2023-12-25T20:14:20.140 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:20.137+0000 7fb043fff700 1 --2- 172.21.15.160:0/865649684 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fb03402ff60 0x7fb034032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:14:20.140 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:20.137+0000 7fb043fff700 1 --2- 172.21.15.160:0/865649684 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fb03402ff60 0x7fb034032420 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7fb0300096a0 tx=0x7fb030006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:14:20.142 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:20.141+0000 7fb0417fa700 1 -- 172.21.15.160:0/865649684 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb038014420 con 0x7fb0440f47c0 2023-12-25T20:14:20.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:14:19 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:14:18.691138+0000 mgr.smithi050.nbzhun (mgr.14162) 218 : cluster [DBG] pgmap v206: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:14:20.348 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:20.347+0000 7fb04aab6700 1 -- 172.21.15.160:0/865649684 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb0440fbf60 con 0x7fb0440f47c0 2023-12-25T20:14:20.348 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:20.348+0000 7fb0417fa700 1 -- 172.21.15.160:0/865649684 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb03801d7f0 con 0x7fb0440f47c0 2023-12-25T20:14:20.348 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:14:20.348 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:14:20.350 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:20.349+0000 7fb04aab6700 1 -- 172.21.15.160:0/865649684 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fb03402ff60 msgr2=0x7fb034032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:14:20.350 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:20.349+0000 7fb04aab6700 1 --2- 172.21.15.160:0/865649684 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fb03402ff60 0x7fb034032420 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7fb0300096a0 tx=0x7fb030006b40).stop 2023-12-25T20:14:20.350 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:20.349+0000 7fb04aab6700 1 -- 172.21.15.160:0/865649684 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb0440f47c0 msgr2=0x7fb0440f7370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:14:20.350 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:20.349+0000 7fb04aab6700 1 --2- 172.21.15.160:0/865649684 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb0440f47c0 0x7fb0440f7370 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7fb038013510 tx=0x7fb03800b040).stop 2023-12-25T20:14:20.350 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:20.349+0000 7fb04aab6700 1 -- 172.21.15.160:0/865649684 shutdown_connections 2023-12-25T20:14:20.350 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:20.349+0000 7fb04aab6700 1 --2- 172.21.15.160:0/865649684 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fb03402ff60 0x7fb034032420 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:14:20.350 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:20.349+0000 7fb04aab6700 1 --2- 172.21.15.160:0/865649684 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb0440f47c0 0x7fb0440f7370 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:14:20.350 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:20.349+0000 7fb04aab6700 1 -- 172.21.15.160:0/865649684 >> 172.21.15.160:0/865649684 conn(0x7fb0440f02e0 msgr2=0x7fb0440f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:14:20.350 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:20.349+0000 7fb04aab6700 1 -- 172.21.15.160:0/865649684 shutdown_connections 2023-12-25T20:14:20.351 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:20.349+0000 7fb04aab6700 1 -- 172.21.15.160:0/865649684 wait complete. 2023-12-25T20:14:20.351 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:14:21.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:14:20 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:14:20.349433+0000 mon.smithi050 (mon.0) 279 : audit [DBG] from='client.? 172.21.15.160:0/865649684' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:14:21.734 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:14:21.734 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:14:22.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:14:21 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:14:20.691464+0000 mgr.smithi050.nbzhun (mgr.14162) 219 : cluster [DBG] pgmap v207: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:14:22.854 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:22.852+0000 7f873c777700 1 -- 172.21.15.160:0/2878573739 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8734060280 msgr2=0x7f87340f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:14:22.854 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:22.852+0000 7f873c777700 1 --2- 172.21.15.160:0/2878573739 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8734060280 0x7f87340f90f0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 rx=0x7f8724009fb0 tx=0x7f8724009b30).stop 2023-12-25T20:14:22.854 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:22.853+0000 7f873c777700 1 -- 172.21.15.160:0/2878573739 shutdown_connections 2023-12-25T20:14:22.854 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:22.853+0000 7f873c777700 1 --2- 172.21.15.160:0/2878573739 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8734060280 0x7f87340f90f0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:14:22.854 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:22.853+0000 7f873c777700 1 -- 172.21.15.160:0/2878573739 >> 172.21.15.160:0/2878573739 conn(0x7f87340f0b70 msgr2=0x7f87340f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:14:22.854 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:22.853+0000 7f873c777700 1 -- 172.21.15.160:0/2878573739 shutdown_connections 2023-12-25T20:14:22.854 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:22.853+0000 7f873c777700 1 -- 172.21.15.160:0/2878573739 wait complete. 2023-12-25T20:14:22.855 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:22.853+0000 7f873c777700 1 Processor -- start 2023-12-25T20:14:22.855 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:22.853+0000 7f873c777700 1 -- start start 2023-12-25T20:14:22.855 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:22.853+0000 7f873c777700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8734060280 0x7f87340f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:14:22.855 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:22.853+0000 7f873c777700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87340f4ff0 con 0x7f8734060280 2023-12-25T20:14:22.855 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:22.853+0000 7f873a513700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8734060280 0x7f87340f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:14:22.855 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:22.853+0000 7f873a513700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8734060280 0x7f87340f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:41580/0 (socket says 172.21.15.160:41580) 2023-12-25T20:14:22.855 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:22.853+0000 7f873a513700 1 -- 172.21.15.160:0/493962545 learned_addr learned my addr 172.21.15.160:0/493962545 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:14:22.855 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:22.854+0000 7f873a513700 1 -- 172.21.15.160:0/493962545 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8724005040 con 0x7f8734060280 2023-12-25T20:14:22.855 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:22.854+0000 7f873a513700 1 --2- 172.21.15.160:0/493962545 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8734060280 0x7f87340f6b70 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7f8724013510 tx=0x7f872400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:14:22.855 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:22.854+0000 7f872affd700 1 -- 172.21.15.160:0/493962545 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8724009320 con 0x7f8734060280 2023-12-25T20:14:22.855 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:22.854+0000 7f873c777700 1 -- 172.21.15.160:0/493962545 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87340f5190 con 0x7f8734060280 2023-12-25T20:14:22.855 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:22.854+0000 7f872affd700 1 -- 172.21.15.160:0/493962545 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8724013f00 con 0x7f8734060280 2023-12-25T20:14:22.855 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:22.854+0000 7f872affd700 1 -- 172.21.15.160:0/493962545 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8724014dc0 con 0x7f8734060280 2023-12-25T20:14:22.855 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:22.854+0000 7f873c777700 1 -- 172.21.15.160:0/493962545 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87340f5620 con 0x7f8734060280 2023-12-25T20:14:22.856 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:22.855+0000 7f872affd700 1 -- 172.21.15.160:0/493962545 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8724014420 con 0x7f8734060280 2023-12-25T20:14:22.856 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:22.855+0000 7f873c777700 1 -- 172.21.15.160:0/493962545 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8734059100 con 0x7f8734060280 2023-12-25T20:14:22.856 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:22.855+0000 7f872affd700 1 --2- 172.21.15.160:0/493962545 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f872002ff60 0x7f8720032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:14:22.856 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:22.855+0000 7f872affd700 1 -- 172.21.15.160:0/493962545 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f87240410f0 con 0x7f8734060280 2023-12-25T20:14:22.856 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:22.855+0000 7f8739d12700 1 --2- 172.21.15.160:0/493962545 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f872002ff60 0x7f8720032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:14:22.857 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:22.856+0000 7f8739d12700 1 --2- 172.21.15.160:0/493962545 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f872002ff60 0x7f8720032420 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f87300096a0 tx=0x7f8730006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:14:22.859 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:22.860+0000 7f872affd700 1 -- 172.21.15.160:0/493962545 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f87240095f0 con 0x7f8734060280 2023-12-25T20:14:23.063 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:23.064+0000 7f873c777700 1 -- 172.21.15.160:0/493962545 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f87340fbf50 con 0x7f8734060280 2023-12-25T20:14:23.064 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:23.064+0000 7f872affd700 1 -- 172.21.15.160:0/493962545 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f872401cb10 con 0x7f8734060280 2023-12-25T20:14:23.064 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:14:23.064 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:14:23.065 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:23.066+0000 7f873c777700 1 -- 172.21.15.160:0/493962545 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f872002ff60 msgr2=0x7f8720032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:14:23.066 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:23.066+0000 7f873c777700 1 --2- 172.21.15.160:0/493962545 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f872002ff60 0x7f8720032420 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f87300096a0 tx=0x7f8730006b40).stop 2023-12-25T20:14:23.066 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:23.066+0000 7f873c777700 1 -- 172.21.15.160:0/493962545 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8734060280 msgr2=0x7f87340f6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:14:23.066 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:23.066+0000 7f873c777700 1 --2- 172.21.15.160:0/493962545 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8734060280 0x7f87340f6b70 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7f8724013510 tx=0x7f872400b040).stop 2023-12-25T20:14:23.066 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:23.066+0000 7f873c777700 1 -- 172.21.15.160:0/493962545 shutdown_connections 2023-12-25T20:14:23.066 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:23.066+0000 7f873c777700 1 --2- 172.21.15.160:0/493962545 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f872002ff60 0x7f8720032420 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:14:23.066 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:23.066+0000 7f873c777700 1 --2- 172.21.15.160:0/493962545 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8734060280 0x7f87340f6b70 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:14:23.066 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:23.066+0000 7f873c777700 1 -- 172.21.15.160:0/493962545 >> 172.21.15.160:0/493962545 conn(0x7f87340f0b70 msgr2=0x7f87340f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:14:23.066 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:23.066+0000 7f873c777700 1 -- 172.21.15.160:0/493962545 shutdown_connections 2023-12-25T20:14:23.066 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:23.066+0000 7f873c777700 1 -- 172.21.15.160:0/493962545 wait complete. 2023-12-25T20:14:23.066 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:14:24.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:14:23 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:14:22.691823+0000 mgr.smithi050.nbzhun (mgr.14162) 220 : cluster [DBG] pgmap v208: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:14:24.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:14:23 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:14:23.064913+0000 mon.smithi050 (mon.0) 280 : audit [DBG] from='client.? 172.21.15.160:0/493962545' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:14:24.518 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:14:24.519 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:14:25.702 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:25.701+0000 7f417f2e0700 1 -- 172.21.15.160:0/1909375786 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4178060200 msgr2=0x7f41780fb570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:14:25.702 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:25.701+0000 7f417f2e0700 1 --2- 172.21.15.160:0/1909375786 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4178060200 0x7f41780fb570 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 rx=0x7f4168009fb0 tx=0x7f4168009b30).stop 2023-12-25T20:14:25.703 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:25.701+0000 7f417f2e0700 1 -- 172.21.15.160:0/1909375786 shutdown_connections 2023-12-25T20:14:25.703 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:25.701+0000 7f417f2e0700 1 --2- 172.21.15.160:0/1909375786 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4178060200 0x7f41780fb570 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:14:25.703 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:25.701+0000 7f417f2e0700 1 -- 172.21.15.160:0/1909375786 >> 172.21.15.160:0/1909375786 conn(0x7f41780f0b70 msgr2=0x7f41780f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:14:25.703 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:25.701+0000 7f417f2e0700 1 -- 172.21.15.160:0/1909375786 shutdown_connections 2023-12-25T20:14:25.703 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:25.701+0000 7f417f2e0700 1 -- 172.21.15.160:0/1909375786 wait complete. 2023-12-25T20:14:25.703 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:25.702+0000 7f417f2e0700 1 Processor -- start 2023-12-25T20:14:25.703 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:25.702+0000 7f417f2e0700 1 -- start start 2023-12-25T20:14:25.703 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:25.702+0000 7f417f2e0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4178060200 0x7f41781088d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:14:25.703 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:25.702+0000 7f417f2e0700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4178108dc0 con 0x7f4178060200 2023-12-25T20:14:25.703 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:25.702+0000 7f417d07c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4178060200 0x7f41781088d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:14:25.703 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:25.702+0000 7f417d07c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4178060200 0x7f41781088d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:41594/0 (socket says 172.21.15.160:41594) 2023-12-25T20:14:25.703 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:25.702+0000 7f417d07c700 1 -- 172.21.15.160:0/844928101 learned_addr learned my addr 172.21.15.160:0/844928101 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:14:25.703 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:25.702+0000 7f417d07c700 1 -- 172.21.15.160:0/844928101 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4168005040 con 0x7f4178060200 2023-12-25T20:14:25.704 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:25.702+0000 7f417d07c700 1 --2- 172.21.15.160:0/844928101 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4178060200 0x7f41781088d0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7f4168013510 tx=0x7f416800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:14:25.704 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:25.703+0000 7f416dffb700 1 -- 172.21.15.160:0/844928101 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4168009320 con 0x7f4178060200 2023-12-25T20:14:25.704 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:25.703+0000 7f417f2e0700 1 -- 172.21.15.160:0/844928101 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4178108fc0 con 0x7f4178060200 2023-12-25T20:14:25.704 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:25.703+0000 7f416dffb700 1 -- 172.21.15.160:0/844928101 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4168013f00 con 0x7f4178060200 2023-12-25T20:14:25.704 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:25.703+0000 7f416dffb700 1 -- 172.21.15.160:0/844928101 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4168014dc0 con 0x7f4178060200 2023-12-25T20:14:25.704 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:25.703+0000 7f417f2e0700 1 -- 172.21.15.160:0/844928101 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4178109450 con 0x7f4178060200 2023-12-25T20:14:25.705 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:25.704+0000 7f417f2e0700 1 -- 172.21.15.160:0/844928101 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4178045ad0 con 0x7f4178060200 2023-12-25T20:14:25.705 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:25.704+0000 7f416dffb700 1 -- 172.21.15.160:0/844928101 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f41680095f0 con 0x7f4178060200 2023-12-25T20:14:25.705 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:25.704+0000 7f416dffb700 1 --2- 172.21.15.160:0/844928101 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f416402ff60 0x7f4164032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:14:25.705 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:25.704+0000 7f416dffb700 1 -- 172.21.15.160:0/844928101 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4168041040 con 0x7f4178060200 2023-12-25T20:14:25.705 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:25.704+0000 7f417c87b700 1 --2- 172.21.15.160:0/844928101 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f416402ff60 0x7f4164032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:14:25.705 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:25.705+0000 7f417c87b700 1 --2- 172.21.15.160:0/844928101 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f416402ff60 0x7f4164032420 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f41740096a0 tx=0x7f4174006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:14:25.708 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:25.709+0000 7f416dffb700 1 -- 172.21.15.160:0/844928101 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4168014420 con 0x7f4178060200 2023-12-25T20:14:25.916 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:25.916+0000 7f417f2e0700 1 -- 172.21.15.160:0/844928101 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f417810a300 con 0x7f4178060200 2023-12-25T20:14:25.917 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:25.917+0000 7f416dffb700 1 -- 172.21.15.160:0/844928101 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4168010230 con 0x7f4178060200 2023-12-25T20:14:25.917 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:14:25.917 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:14:25.919 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:25.918+0000 7f417f2e0700 1 -- 172.21.15.160:0/844928101 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f416402ff60 msgr2=0x7f4164032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:14:25.919 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:25.919+0000 7f417f2e0700 1 --2- 172.21.15.160:0/844928101 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f416402ff60 0x7f4164032420 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f41740096a0 tx=0x7f4174006b40).stop 2023-12-25T20:14:25.919 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:25.919+0000 7f417f2e0700 1 -- 172.21.15.160:0/844928101 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4178060200 msgr2=0x7f41781088d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:14:25.919 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:25.919+0000 7f417f2e0700 1 --2- 172.21.15.160:0/844928101 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4178060200 0x7f41781088d0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7f4168013510 tx=0x7f416800b040).stop 2023-12-25T20:14:25.919 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:25.919+0000 7f417f2e0700 1 -- 172.21.15.160:0/844928101 shutdown_connections 2023-12-25T20:14:25.919 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:25.919+0000 7f417f2e0700 1 --2- 172.21.15.160:0/844928101 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f416402ff60 0x7f4164032420 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:14:25.919 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:25.919+0000 7f417f2e0700 1 --2- 172.21.15.160:0/844928101 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4178060200 0x7f41781088d0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:14:25.919 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:25.919+0000 7f417f2e0700 1 -- 172.21.15.160:0/844928101 >> 172.21.15.160:0/844928101 conn(0x7f41780f0b70 msgr2=0x7f41780f1670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:14:25.919 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:25.919+0000 7f417f2e0700 1 -- 172.21.15.160:0/844928101 shutdown_connections 2023-12-25T20:14:25.919 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:25.919+0000 7f417f2e0700 1 -- 172.21.15.160:0/844928101 wait complete. 2023-12-25T20:14:25.919 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:14:26.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:14:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:14:24.692165+0000 mgr.smithi050.nbzhun (mgr.14162) 221 : cluster [DBG] pgmap v209: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:14:27.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:14:26 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:14:25.917843+0000 mon.smithi050 (mon.0) 281 : audit [DBG] from='client.? 172.21.15.160:0/844928101' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:14:27.341 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:14:27.341 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:14:28.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:14:27 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:14:26.692508+0000 mgr.smithi050.nbzhun (mgr.14162) 222 : cluster [DBG] pgmap v210: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:14:28.586 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:28.585+0000 7f3fdedaf700 1 -- 172.21.15.160:0/2864040229 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3fd80f4740 msgr2=0x7f3fd80f4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:14:28.586 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:28.585+0000 7f3fdedaf700 1 --2- 172.21.15.160:0/2864040229 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3fd80f4740 0x7f3fd80f4b60 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 rx=0x7f3fcc009fb0 tx=0x7f3fcc009b30).stop 2023-12-25T20:14:28.586 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:28.585+0000 7f3fdedaf700 1 -- 172.21.15.160:0/2864040229 shutdown_connections 2023-12-25T20:14:28.586 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:28.585+0000 7f3fdedaf700 1 --2- 172.21.15.160:0/2864040229 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3fd80f4740 0x7f3fd80f4b60 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:14:28.586 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:28.585+0000 7f3fdedaf700 1 -- 172.21.15.160:0/2864040229 >> 172.21.15.160:0/2864040229 conn(0x7f3fd80f0260 msgr2=0x7f3fd80f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:14:28.586 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:28.585+0000 7f3fdedaf700 1 -- 172.21.15.160:0/2864040229 shutdown_connections 2023-12-25T20:14:28.586 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:28.585+0000 7f3fdedaf700 1 -- 172.21.15.160:0/2864040229 wait complete. 2023-12-25T20:14:28.586 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:28.586+0000 7f3fdedaf700 1 Processor -- start 2023-12-25T20:14:28.587 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:28.586+0000 7f3fdedaf700 1 -- start start 2023-12-25T20:14:28.587 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:28.586+0000 7f3fdedaf700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3fd80f4740 0x7f3fd80f72f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:14:28.587 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:28.586+0000 7f3fdedaf700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fd80f5770 con 0x7f3fd80f4740 2023-12-25T20:14:28.587 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:28.586+0000 7f3fdcb4b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3fd80f4740 0x7f3fd80f72f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:14:28.587 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:28.586+0000 7f3fdcb4b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3fd80f4740 0x7f3fd80f72f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:35194/0 (socket says 172.21.15.160:35194) 2023-12-25T20:14:28.587 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:28.586+0000 7f3fdcb4b700 1 -- 172.21.15.160:0/1506294867 learned_addr learned my addr 172.21.15.160:0/1506294867 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:14:28.587 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:28.587+0000 7f3fdcb4b700 1 -- 172.21.15.160:0/1506294867 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fcc005040 con 0x7f3fd80f4740 2023-12-25T20:14:28.588 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:28.587+0000 7f3fdcb4b700 1 --2- 172.21.15.160:0/1506294867 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3fd80f4740 0x7f3fd80f72f0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7f3fcc013510 tx=0x7f3fcc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:14:28.588 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:28.587+0000 7f3fd57fa700 1 -- 172.21.15.160:0/1506294867 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3fcc009320 con 0x7f3fd80f4740 2023-12-25T20:14:28.588 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:28.587+0000 7f3fdedaf700 1 -- 172.21.15.160:0/1506294867 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fd80f5910 con 0x7f3fd80f4740 2023-12-25T20:14:28.588 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:28.587+0000 7f3fd57fa700 1 -- 172.21.15.160:0/1506294867 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3fcc013f00 con 0x7f3fd80f4740 2023-12-25T20:14:28.589 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:28.587+0000 7f3fd57fa700 1 -- 172.21.15.160:0/1506294867 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3fcc014d90 con 0x7f3fd80f4740 2023-12-25T20:14:28.589 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:28.587+0000 7f3fdedaf700 1 -- 172.21.15.160:0/1506294867 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fd80f5da0 con 0x7f3fd80f4740 2023-12-25T20:14:28.589 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:28.588+0000 7f3fd57fa700 1 -- 172.21.15.160:0/1506294867 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3fcc0095f0 con 0x7f3fd80f4740 2023-12-25T20:14:28.589 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:28.588+0000 7f3fdedaf700 1 -- 172.21.15.160:0/1506294867 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fd8059100 con 0x7f3fd80f4740 2023-12-25T20:14:28.589 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:28.588+0000 7f3fd57fa700 1 --2- 172.21.15.160:0/1506294867 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f3fc802ff60 0x7f3fc8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:14:28.589 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:28.588+0000 7f3fd57fa700 1 -- 172.21.15.160:0/1506294867 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3fcc0410c0 con 0x7f3fd80f4740 2023-12-25T20:14:28.589 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:28.588+0000 7f3fd7fff700 1 --2- 172.21.15.160:0/1506294867 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f3fc802ff60 0x7f3fc8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:14:28.589 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:28.589+0000 7f3fd7fff700 1 --2- 172.21.15.160:0/1506294867 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f3fc802ff60 0x7f3fc8032420 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f3fc40096a0 tx=0x7f3fc4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:14:28.592 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:28.593+0000 7f3fd57fa700 1 -- 172.21.15.160:0/1506294867 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3fcc014420 con 0x7f3fd80f4740 2023-12-25T20:14:28.796 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:28.796+0000 7f3fdedaf700 1 -- 172.21.15.160:0/1506294867 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3fd80fbf40 con 0x7f3fd80f4740 2023-12-25T20:14:28.796 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:28.797+0000 7f3fd57fa700 1 -- 172.21.15.160:0/1506294867 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3fcc010230 con 0x7f3fd80f4740 2023-12-25T20:14:28.797 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:14:28.797 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:14:28.798 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:28.798+0000 7f3fdedaf700 1 -- 172.21.15.160:0/1506294867 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f3fc802ff60 msgr2=0x7f3fc8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:14:28.798 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:28.798+0000 7f3fdedaf700 1 --2- 172.21.15.160:0/1506294867 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f3fc802ff60 0x7f3fc8032420 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f3fc40096a0 tx=0x7f3fc4006b40).stop 2023-12-25T20:14:28.798 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:28.798+0000 7f3fdedaf700 1 -- 172.21.15.160:0/1506294867 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3fd80f4740 msgr2=0x7f3fd80f72f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:14:28.798 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:28.798+0000 7f3fdedaf700 1 --2- 172.21.15.160:0/1506294867 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3fd80f4740 0x7f3fd80f72f0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7f3fcc013510 tx=0x7f3fcc00b040).stop 2023-12-25T20:14:28.798 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:28.798+0000 7f3fdedaf700 1 -- 172.21.15.160:0/1506294867 shutdown_connections 2023-12-25T20:14:28.799 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:28.798+0000 7f3fdedaf700 1 --2- 172.21.15.160:0/1506294867 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f3fc802ff60 0x7f3fc8032420 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:14:28.799 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:28.798+0000 7f3fdedaf700 1 --2- 172.21.15.160:0/1506294867 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3fd80f4740 0x7f3fd80f72f0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:14:28.799 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:28.798+0000 7f3fdedaf700 1 -- 172.21.15.160:0/1506294867 >> 172.21.15.160:0/1506294867 conn(0x7f3fd80f0260 msgr2=0x7f3fd80f0d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:14:28.799 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:28.798+0000 7f3fdedaf700 1 -- 172.21.15.160:0/1506294867 shutdown_connections 2023-12-25T20:14:28.799 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:28.798+0000 7f3fdedaf700 1 -- 172.21.15.160:0/1506294867 wait complete. 2023-12-25T20:14:28.799 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:14:29.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:14:28 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:14:28.797388+0000 mon.smithi050 (mon.0) 282 : audit [DBG] from='client.? 172.21.15.160:0/1506294867' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:14:30.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:14:29 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:14:28.692868+0000 mgr.smithi050.nbzhun (mgr.14162) 223 : cluster [DBG] pgmap v211: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:14:30.219 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:14:30.219 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:14:31.413 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:31.413+0000 7fc9996d0700 1 -- 172.21.15.160:0/582318977 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc9940f47c0 msgr2=0x7fc9940f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:14:31.413 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:31.413+0000 7fc9996d0700 1 --2- 172.21.15.160:0/582318977 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc9940f47c0 0x7fc9940f4be0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 rx=0x7fc988009fb0 tx=0x7fc988009b30).stop 2023-12-25T20:14:31.414 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:31.413+0000 7fc9996d0700 1 -- 172.21.15.160:0/582318977 shutdown_connections 2023-12-25T20:14:31.414 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:31.413+0000 7fc9996d0700 1 --2- 172.21.15.160:0/582318977 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc9940f47c0 0x7fc9940f4be0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:14:31.414 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:31.413+0000 7fc9996d0700 1 -- 172.21.15.160:0/582318977 >> 172.21.15.160:0/582318977 conn(0x7fc9940f02e0 msgr2=0x7fc9940f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:14:31.414 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:31.414+0000 7fc9996d0700 1 -- 172.21.15.160:0/582318977 shutdown_connections 2023-12-25T20:14:31.414 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:31.414+0000 7fc9996d0700 1 -- 172.21.15.160:0/582318977 wait complete. 2023-12-25T20:14:31.415 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:31.414+0000 7fc9996d0700 1 Processor -- start 2023-12-25T20:14:31.415 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:31.414+0000 7fc9996d0700 1 -- start start 2023-12-25T20:14:31.415 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:31.414+0000 7fc9996d0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc9940f47c0 0x7fc994106700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:14:31.415 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:31.414+0000 7fc9996d0700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc994106bf0 con 0x7fc9940f47c0 2023-12-25T20:14:31.415 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:31.415+0000 7fc992ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc9940f47c0 0x7fc994106700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:14:31.415 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:31.415+0000 7fc992ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc9940f47c0 0x7fc994106700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:35218/0 (socket says 172.21.15.160:35218) 2023-12-25T20:14:31.416 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:31.415+0000 7fc992ffd700 1 -- 172.21.15.160:0/4165258962 learned_addr learned my addr 172.21.15.160:0/4165258962 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:14:31.416 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:31.415+0000 7fc992ffd700 1 -- 172.21.15.160:0/4165258962 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc988005040 con 0x7fc9940f47c0 2023-12-25T20:14:31.416 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:31.415+0000 7fc992ffd700 1 --2- 172.21.15.160:0/4165258962 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc9940f47c0 0x7fc994106700 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7fc988013510 tx=0x7fc98800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:14:31.416 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:31.415+0000 7fc97b7fe700 1 -- 172.21.15.160:0/4165258962 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc988009320 con 0x7fc9940f47c0 2023-12-25T20:14:31.416 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:31.415+0000 7fc97b7fe700 1 -- 172.21.15.160:0/4165258962 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc988013f00 con 0x7fc9940f47c0 2023-12-25T20:14:31.416 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:31.415+0000 7fc9996d0700 1 -- 172.21.15.160:0/4165258962 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc994106df0 con 0x7fc9940f47c0 2023-12-25T20:14:31.416 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:31.415+0000 7fc97b7fe700 1 -- 172.21.15.160:0/4165258962 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc988014d90 con 0x7fc9940f47c0 2023-12-25T20:14:31.416 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:31.415+0000 7fc9996d0700 1 -- 172.21.15.160:0/4165258962 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc994107280 con 0x7fc9940f47c0 2023-12-25T20:14:31.417 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:31.416+0000 7fc97b7fe700 1 -- 172.21.15.160:0/4165258962 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc9880095f0 con 0x7fc9940f47c0 2023-12-25T20:14:31.417 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:31.416+0000 7fc9996d0700 1 -- 172.21.15.160:0/4165258962 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc994045ad0 con 0x7fc9940f47c0 2023-12-25T20:14:31.417 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:31.416+0000 7fc97b7fe700 1 --2- 172.21.15.160:0/4165258962 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fc97c02ff60 0x7fc97c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:14:31.417 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:31.416+0000 7fc97b7fe700 1 -- 172.21.15.160:0/4165258962 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc9880410a0 con 0x7fc9940f47c0 2023-12-25T20:14:31.417 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:31.417+0000 7fc9927fc700 1 --2- 172.21.15.160:0/4165258962 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fc97c02ff60 0x7fc97c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:14:31.417 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:31.417+0000 7fc9927fc700 1 --2- 172.21.15.160:0/4165258962 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fc97c02ff60 0x7fc97c032420 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7fc9840096a0 tx=0x7fc984006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:14:31.421 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:31.421+0000 7fc97b7fe700 1 -- 172.21.15.160:0/4165258962 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc98801d7f0 con 0x7fc9940f47c0 2023-12-25T20:14:31.625 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:31.625+0000 7fc9996d0700 1 -- 172.21.15.160:0/4165258962 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc994108130 con 0x7fc9940f47c0 2023-12-25T20:14:31.626 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:31.626+0000 7fc97b7fe700 1 -- 172.21.15.160:0/4165258962 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc988010230 con 0x7fc9940f47c0 2023-12-25T20:14:31.626 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:14:31.626 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:14:31.628 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:31.628+0000 7fc9996d0700 1 -- 172.21.15.160:0/4165258962 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fc97c02ff60 msgr2=0x7fc97c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:14:31.628 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:31.628+0000 7fc9996d0700 1 --2- 172.21.15.160:0/4165258962 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fc97c02ff60 0x7fc97c032420 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7fc9840096a0 tx=0x7fc984006b40).stop 2023-12-25T20:14:31.628 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:31.628+0000 7fc9996d0700 1 -- 172.21.15.160:0/4165258962 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc9940f47c0 msgr2=0x7fc994106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:14:31.628 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:31.628+0000 7fc9996d0700 1 --2- 172.21.15.160:0/4165258962 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc9940f47c0 0x7fc994106700 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7fc988013510 tx=0x7fc98800b040).stop 2023-12-25T20:14:31.628 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:31.628+0000 7fc9996d0700 1 -- 172.21.15.160:0/4165258962 shutdown_connections 2023-12-25T20:14:31.628 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:31.628+0000 7fc9996d0700 1 --2- 172.21.15.160:0/4165258962 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fc97c02ff60 0x7fc97c032420 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:14:31.628 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:31.628+0000 7fc9996d0700 1 --2- 172.21.15.160:0/4165258962 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc9940f47c0 0x7fc994106700 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:14:31.628 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:31.628+0000 7fc9996d0700 1 -- 172.21.15.160:0/4165258962 >> 172.21.15.160:0/4165258962 conn(0x7fc9940f02e0 msgr2=0x7fc9940f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:14:31.628 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:31.628+0000 7fc9996d0700 1 -- 172.21.15.160:0/4165258962 shutdown_connections 2023-12-25T20:14:31.628 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:31.628+0000 7fc9996d0700 1 -- 172.21.15.160:0/4165258962 wait complete. 2023-12-25T20:14:31.629 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:14:32.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:14:31 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:14:30.693108+0000 mgr.smithi050.nbzhun (mgr.14162) 224 : cluster [DBG] pgmap v212: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:14:32.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:14:31 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:14:31.626916+0000 mon.smithi050 (mon.0) 283 : audit [DBG] from='client.? 172.21.15.160:0/4165258962' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:14:33.041 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:14:33.041 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:14:34.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:14:33 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:14:32.693460+0000 mgr.smithi050.nbzhun (mgr.14162) 225 : cluster [DBG] pgmap v213: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:14:34.244 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:34.243+0000 7fe04ffd4700 1 -- 172.21.15.160:0/1492262302 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe048060280 msgr2=0x7fe0480f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:14:34.244 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:34.243+0000 7fe04ffd4700 1 --2- 172.21.15.160:0/1492262302 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe048060280 0x7fe0480f90f0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 rx=0x7fe038009fb0 tx=0x7fe038009b30).stop 2023-12-25T20:14:34.244 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:34.243+0000 7fe04ffd4700 1 -- 172.21.15.160:0/1492262302 shutdown_connections 2023-12-25T20:14:34.244 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:34.243+0000 7fe04ffd4700 1 --2- 172.21.15.160:0/1492262302 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe048060280 0x7fe0480f90f0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:14:34.244 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:34.243+0000 7fe04ffd4700 1 -- 172.21.15.160:0/1492262302 >> 172.21.15.160:0/1492262302 conn(0x7fe0480f0b70 msgr2=0x7fe0480f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:14:34.245 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:34.243+0000 7fe04ffd4700 1 -- 172.21.15.160:0/1492262302 shutdown_connections 2023-12-25T20:14:34.245 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:34.243+0000 7fe04ffd4700 1 -- 172.21.15.160:0/1492262302 wait complete. 2023-12-25T20:14:34.245 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:34.244+0000 7fe04ffd4700 1 Processor -- start 2023-12-25T20:14:34.245 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:34.244+0000 7fe04ffd4700 1 -- start start 2023-12-25T20:14:34.245 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:34.244+0000 7fe04ffd4700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe048060280 0x7fe0481066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:14:34.245 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:34.244+0000 7fe04ffd4700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe048106bd0 con 0x7fe048060280 2023-12-25T20:14:34.245 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:34.244+0000 7fe04dd70700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe048060280 0x7fe0481066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:14:34.245 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:34.244+0000 7fe04dd70700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe048060280 0x7fe0481066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:35238/0 (socket says 172.21.15.160:35238) 2023-12-25T20:14:34.245 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:34.244+0000 7fe04dd70700 1 -- 172.21.15.160:0/1983711116 learned_addr learned my addr 172.21.15.160:0/1983711116 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:14:34.245 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:34.245+0000 7fe04dd70700 1 -- 172.21.15.160:0/1983711116 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe038005040 con 0x7fe048060280 2023-12-25T20:14:34.245 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:34.245+0000 7fe04dd70700 1 --2- 172.21.15.160:0/1983711116 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe048060280 0x7fe0481066e0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7fe038013510 tx=0x7fe03800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:14:34.246 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:34.245+0000 7fe03e7fc700 1 -- 172.21.15.160:0/1983711116 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe038009320 con 0x7fe048060280 2023-12-25T20:14:34.247 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:34.245+0000 7fe03e7fc700 1 -- 172.21.15.160:0/1983711116 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe038013f00 con 0x7fe048060280 2023-12-25T20:14:34.247 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:34.245+0000 7fe03e7fc700 1 -- 172.21.15.160:0/1983711116 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe038014dc0 con 0x7fe048060280 2023-12-25T20:14:34.247 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:34.245+0000 7fe04ffd4700 1 -- 172.21.15.160:0/1983711116 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe048106dd0 con 0x7fe048060280 2023-12-25T20:14:34.247 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:34.245+0000 7fe04ffd4700 1 -- 172.21.15.160:0/1983711116 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe048107260 con 0x7fe048060280 2023-12-25T20:14:34.247 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:34.246+0000 7fe03e7fc700 1 -- 172.21.15.160:0/1983711116 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe0380095f0 con 0x7fe048060280 2023-12-25T20:14:34.247 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:34.246+0000 7fe03e7fc700 1 --2- 172.21.15.160:0/1983711116 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fe03402ffb0 0x7fe034032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:14:34.247 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:34.246+0000 7fe04ffd4700 1 -- 172.21.15.160:0/1983711116 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe048045ad0 con 0x7fe048060280 2023-12-25T20:14:34.247 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:34.246+0000 7fe03e7fc700 1 -- 172.21.15.160:0/1983711116 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe0380410b0 con 0x7fe048060280 2023-12-25T20:14:34.247 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:34.247+0000 7fe04d56f700 1 --2- 172.21.15.160:0/1983711116 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fe03402ffb0 0x7fe034032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:14:34.247 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:34.247+0000 7fe04d56f700 1 --2- 172.21.15.160:0/1983711116 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fe03402ffb0 0x7fe034032470 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7fe0440096a0 tx=0x7fe044006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:14:34.250 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:34.251+0000 7fe03e7fc700 1 -- 172.21.15.160:0/1983711116 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe03801c320 con 0x7fe048060280 2023-12-25T20:14:34.454 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:34.455+0000 7fe04ffd4700 1 -- 172.21.15.160:0/1983711116 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe048108110 con 0x7fe048060280 2023-12-25T20:14:34.455 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:34.455+0000 7fe03e7fc700 1 -- 172.21.15.160:0/1983711116 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe03800f340 con 0x7fe048060280 2023-12-25T20:14:34.455 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:14:34.456 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:14:34.457 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:34.457+0000 7fe04ffd4700 1 -- 172.21.15.160:0/1983711116 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fe03402ffb0 msgr2=0x7fe034032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:14:34.457 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:34.457+0000 7fe04ffd4700 1 --2- 172.21.15.160:0/1983711116 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fe03402ffb0 0x7fe034032470 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7fe0440096a0 tx=0x7fe044006b40).stop 2023-12-25T20:14:34.457 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:34.457+0000 7fe04ffd4700 1 -- 172.21.15.160:0/1983711116 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe048060280 msgr2=0x7fe0481066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:14:34.457 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:34.457+0000 7fe04ffd4700 1 --2- 172.21.15.160:0/1983711116 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe048060280 0x7fe0481066e0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7fe038013510 tx=0x7fe03800b040).stop 2023-12-25T20:14:34.457 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:34.457+0000 7fe04ffd4700 1 -- 172.21.15.160:0/1983711116 shutdown_connections 2023-12-25T20:14:34.457 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:34.457+0000 7fe04ffd4700 1 --2- 172.21.15.160:0/1983711116 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fe03402ffb0 0x7fe034032470 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:14:34.457 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:34.457+0000 7fe04ffd4700 1 --2- 172.21.15.160:0/1983711116 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe048060280 0x7fe0481066e0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:14:34.457 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:34.457+0000 7fe04ffd4700 1 -- 172.21.15.160:0/1983711116 >> 172.21.15.160:0/1983711116 conn(0x7fe0480f0b70 msgr2=0x7fe0480f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:14:34.457 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:34.457+0000 7fe04ffd4700 1 -- 172.21.15.160:0/1983711116 shutdown_connections 2023-12-25T20:14:34.458 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:34.457+0000 7fe04ffd4700 1 -- 172.21.15.160:0/1983711116 wait complete. 2023-12-25T20:14:34.458 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:14:35.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:14:34 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:14:34.456167+0000 mon.smithi050 (mon.0) 284 : audit [DBG] from='client.? 172.21.15.160:0/1983711116' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:14:35.888 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:14:35.888 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:14:36.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:14:35 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:14:34.693746+0000 mgr.smithi050.nbzhun (mgr.14162) 226 : cluster [DBG] pgmap v214: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:14:37.100 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:37.100+0000 7f6312aa5700 1 -- 172.21.15.160:0/2858720810 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f630c0f5410 msgr2=0x7f630c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:14:37.100 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:37.100+0000 7f6312aa5700 1 --2- 172.21.15.160:0/2858720810 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f630c0f5410 0x7f630c0f5830 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 rx=0x7f6300009fb0 tx=0x7f6300009b30).stop 2023-12-25T20:14:37.100 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:37.100+0000 7f6312aa5700 1 -- 172.21.15.160:0/2858720810 shutdown_connections 2023-12-25T20:14:37.101 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:37.100+0000 7f6312aa5700 1 --2- 172.21.15.160:0/2858720810 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f630c0f5410 0x7f630c0f5830 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:14:37.101 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:37.100+0000 7f6312aa5700 1 -- 172.21.15.160:0/2858720810 >> 172.21.15.160:0/2858720810 conn(0x7f630c0f0b70 msgr2=0x7f630c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:14:37.101 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:37.100+0000 7f6312aa5700 1 -- 172.21.15.160:0/2858720810 shutdown_connections 2023-12-25T20:14:37.101 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:37.100+0000 7f6312aa5700 1 -- 172.21.15.160:0/2858720810 wait complete. 2023-12-25T20:14:37.101 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:37.100+0000 7f6312aa5700 1 Processor -- start 2023-12-25T20:14:37.101 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:37.100+0000 7f6312aa5700 1 -- start start 2023-12-25T20:14:37.101 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:37.101+0000 7f6312aa5700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f630c0f5410 0x7f630c1093f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:14:37.101 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:37.101+0000 7f6312aa5700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f630c060280 con 0x7f630c0f5410 2023-12-25T20:14:37.103 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:37.101+0000 7f6310841700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f630c0f5410 0x7f630c1093f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:14:37.103 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:37.101+0000 7f6310841700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f630c0f5410 0x7f630c1093f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:35250/0 (socket says 172.21.15.160:35250) 2023-12-25T20:14:37.103 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:37.101+0000 7f6310841700 1 -- 172.21.15.160:0/3089972136 learned_addr learned my addr 172.21.15.160:0/3089972136 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:14:37.103 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:37.101+0000 7f6310841700 1 -- 172.21.15.160:0/3089972136 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6300005040 con 0x7f630c0f5410 2023-12-25T20:14:37.103 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:37.102+0000 7f6310841700 1 --2- 172.21.15.160:0/3089972136 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f630c0f5410 0x7f630c1093f0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7f6300013510 tx=0x7f630000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:14:37.103 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:37.102+0000 7f63097fa700 1 -- 172.21.15.160:0/3089972136 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6300009320 con 0x7f630c0f5410 2023-12-25T20:14:37.103 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:37.102+0000 7f63097fa700 1 -- 172.21.15.160:0/3089972136 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6300013f00 con 0x7f630c0f5410 2023-12-25T20:14:37.103 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:37.102+0000 7f6312aa5700 1 -- 172.21.15.160:0/3089972136 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f630c060480 con 0x7f630c0f5410 2023-12-25T20:14:37.104 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:37.102+0000 7f63097fa700 1 -- 172.21.15.160:0/3089972136 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6300014d90 con 0x7f630c0f5410 2023-12-25T20:14:37.104 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:37.102+0000 7f6312aa5700 1 -- 172.21.15.160:0/3089972136 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f630c109b20 con 0x7f630c0f5410 2023-12-25T20:14:37.104 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:37.103+0000 7f63097fa700 1 -- 172.21.15.160:0/3089972136 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f63000095f0 con 0x7f630c0f5410 2023-12-25T20:14:37.104 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:37.103+0000 7f6312aa5700 1 -- 172.21.15.160:0/3089972136 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f630c045ad0 con 0x7f630c0f5410 2023-12-25T20:14:37.104 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:37.103+0000 7f63097fa700 1 --2- 172.21.15.160:0/3089972136 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f62fc02ff60 0x7f62fc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:14:37.104 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:37.103+0000 7f63097fa700 1 -- 172.21.15.160:0/3089972136 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6300040fe0 con 0x7f630c0f5410 2023-12-25T20:14:37.104 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:37.103+0000 7f630bfff700 1 --2- 172.21.15.160:0/3089972136 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f62fc02ff60 0x7f62fc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:14:37.104 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:37.103+0000 7f630bfff700 1 --2- 172.21.15.160:0/3089972136 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f62fc02ff60 0x7f62fc032420 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f62f80096a0 tx=0x7f62f8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:14:37.107 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:37.108+0000 7f63097fa700 1 -- 172.21.15.160:0/3089972136 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f630001d7f0 con 0x7f630c0f5410 2023-12-25T20:14:37.313 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:37.314+0000 7f6312aa5700 1 -- 172.21.15.160:0/3089972136 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f630c10a340 con 0x7f630c0f5410 2023-12-25T20:14:37.314 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:37.314+0000 7f63097fa700 1 -- 172.21.15.160:0/3089972136 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6300010230 con 0x7f630c0f5410 2023-12-25T20:14:37.314 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:14:37.314 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:14:37.316 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:37.316+0000 7f6312aa5700 1 -- 172.21.15.160:0/3089972136 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f62fc02ff60 msgr2=0x7f62fc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:14:37.316 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:37.316+0000 7f6312aa5700 1 --2- 172.21.15.160:0/3089972136 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f62fc02ff60 0x7f62fc032420 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f62f80096a0 tx=0x7f62f8006b40).stop 2023-12-25T20:14:37.316 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:37.316+0000 7f6312aa5700 1 -- 172.21.15.160:0/3089972136 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f630c0f5410 msgr2=0x7f630c1093f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:14:37.316 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:37.316+0000 7f6312aa5700 1 --2- 172.21.15.160:0/3089972136 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f630c0f5410 0x7f630c1093f0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7f6300013510 tx=0x7f630000b040).stop 2023-12-25T20:14:37.316 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:37.316+0000 7f6312aa5700 1 -- 172.21.15.160:0/3089972136 shutdown_connections 2023-12-25T20:14:37.316 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:37.316+0000 7f6312aa5700 1 --2- 172.21.15.160:0/3089972136 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f62fc02ff60 0x7f62fc032420 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:14:37.316 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:37.316+0000 7f6312aa5700 1 --2- 172.21.15.160:0/3089972136 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f630c0f5410 0x7f630c1093f0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:14:37.316 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:37.316+0000 7f6312aa5700 1 -- 172.21.15.160:0/3089972136 >> 172.21.15.160:0/3089972136 conn(0x7f630c0f0b70 msgr2=0x7f630c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:14:37.317 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:37.316+0000 7f6312aa5700 1 -- 172.21.15.160:0/3089972136 shutdown_connections 2023-12-25T20:14:37.317 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:37.316+0000 7f6312aa5700 1 -- 172.21.15.160:0/3089972136 wait complete. 2023-12-25T20:14:37.317 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:14:38.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:14:37 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:14:36.694093+0000 mgr.smithi050.nbzhun (mgr.14162) 227 : cluster [DBG] pgmap v215: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:14:38.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:14:37 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:14:37.315224+0000 mon.smithi050 (mon.0) 285 : audit [DBG] from='client.? 172.21.15.160:0/3089972136' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:14:38.744 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:14:38.745 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:14:39.907 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:39.905+0000 7fcffcf4e700 1 -- 172.21.15.160:0/1487225993 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcff80601e0 msgr2=0x7fcff80f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:14:39.907 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:39.905+0000 7fcffcf4e700 1 --2- 172.21.15.160:0/1487225993 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcff80601e0 0x7fcff80f8f20 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 rx=0x7fcfec009fb0 tx=0x7fcfec009b30).stop 2023-12-25T20:14:39.907 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:39.905+0000 7fcffcf4e700 1 -- 172.21.15.160:0/1487225993 shutdown_connections 2023-12-25T20:14:39.907 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:39.905+0000 7fcffcf4e700 1 --2- 172.21.15.160:0/1487225993 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcff80601e0 0x7fcff80f8f20 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:14:39.907 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:39.905+0000 7fcffcf4e700 1 -- 172.21.15.160:0/1487225993 >> 172.21.15.160:0/1487225993 conn(0x7fcff80f09a0 msgr2=0x7fcff80f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:14:39.907 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:39.905+0000 7fcffcf4e700 1 -- 172.21.15.160:0/1487225993 shutdown_connections 2023-12-25T20:14:39.907 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:39.905+0000 7fcffcf4e700 1 -- 172.21.15.160:0/1487225993 wait complete. 2023-12-25T20:14:39.907 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:39.905+0000 7fcffcf4e700 1 Processor -- start 2023-12-25T20:14:39.908 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:39.905+0000 7fcffcf4e700 1 -- start start 2023-12-25T20:14:39.908 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:39.906+0000 7fcffcf4e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcff80601e0 0x7fcff80f69a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:14:39.908 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:39.906+0000 7fcffcf4e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcff80f4e20 con 0x7fcff80601e0 2023-12-25T20:14:39.908 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:39.906+0000 7fcff77fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcff80601e0 0x7fcff80f69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:14:39.908 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:39.906+0000 7fcff77fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcff80601e0 0x7fcff80f69a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:53182/0 (socket says 172.21.15.160:53182) 2023-12-25T20:14:39.908 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:39.906+0000 7fcff77fe700 1 -- 172.21.15.160:0/84006817 learned_addr learned my addr 172.21.15.160:0/84006817 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:14:39.908 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:39.906+0000 7fcff77fe700 1 -- 172.21.15.160:0/84006817 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfec005040 con 0x7fcff80601e0 2023-12-25T20:14:39.908 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:39.906+0000 7fcff77fe700 1 --2- 172.21.15.160:0/84006817 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcff80601e0 0x7fcff80f69a0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7fcfec013510 tx=0x7fcfec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:14:39.908 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:39.907+0000 7fcfdffff700 1 -- 172.21.15.160:0/84006817 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcfec009320 con 0x7fcff80601e0 2023-12-25T20:14:39.908 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:39.907+0000 7fcfdffff700 1 -- 172.21.15.160:0/84006817 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcfec013f00 con 0x7fcff80601e0 2023-12-25T20:14:39.908 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:39.907+0000 7fcffcf4e700 1 -- 172.21.15.160:0/84006817 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcff80f4fc0 con 0x7fcff80601e0 2023-12-25T20:14:39.909 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:39.907+0000 7fcfdffff700 1 -- 172.21.15.160:0/84006817 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcfec014d90 con 0x7fcff80601e0 2023-12-25T20:14:39.909 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:39.907+0000 7fcffcf4e700 1 -- 172.21.15.160:0/84006817 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcff80f5450 con 0x7fcff80601e0 2023-12-25T20:14:39.910 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:39.907+0000 7fcfdffff700 1 -- 172.21.15.160:0/84006817 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcfec0095f0 con 0x7fcff80601e0 2023-12-25T20:14:39.910 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:39.907+0000 7fcffcf4e700 1 -- 172.21.15.160:0/84006817 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcff8059100 con 0x7fcff80601e0 2023-12-25T20:14:39.910 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:39.908+0000 7fcfdffff700 1 --2- 172.21.15.160:0/84006817 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fcfe002ff60 0x7fcfe0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:14:39.910 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:39.908+0000 7fcfdffff700 1 -- 172.21.15.160:0/84006817 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcfec0410c0 con 0x7fcff80601e0 2023-12-25T20:14:39.910 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:39.908+0000 7fcff6ffd700 1 --2- 172.21.15.160:0/84006817 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fcfe002ff60 0x7fcfe0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:14:39.910 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:39.908+0000 7fcff6ffd700 1 --2- 172.21.15.160:0/84006817 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fcfe002ff60 0x7fcfe0032420 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7fcfe80096a0 tx=0x7fcfe8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:14:39.912 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:39.913+0000 7fcfdffff700 1 -- 172.21.15.160:0/84006817 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcfec014420 con 0x7fcff80601e0 2023-12-25T20:14:40.121 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:40.121+0000 7fcffcf4e700 1 -- 172.21.15.160:0/84006817 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcff80fbcd0 con 0x7fcff80601e0 2023-12-25T20:14:40.121 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:40.122+0000 7fcfdffff700 1 -- 172.21.15.160:0/84006817 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fcfec01d7f0 con 0x7fcff80601e0 2023-12-25T20:14:40.121 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:14:40.122 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:14:40.123 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:40.123+0000 7fcffcf4e700 1 -- 172.21.15.160:0/84006817 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fcfe002ff60 msgr2=0x7fcfe0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:14:40.123 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:40.123+0000 7fcffcf4e700 1 --2- 172.21.15.160:0/84006817 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fcfe002ff60 0x7fcfe0032420 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7fcfe80096a0 tx=0x7fcfe8006b40).stop 2023-12-25T20:14:40.123 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:40.123+0000 7fcffcf4e700 1 -- 172.21.15.160:0/84006817 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcff80601e0 msgr2=0x7fcff80f69a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:14:40.123 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:40.123+0000 7fcffcf4e700 1 --2- 172.21.15.160:0/84006817 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcff80601e0 0x7fcff80f69a0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7fcfec013510 tx=0x7fcfec00b040).stop 2023-12-25T20:14:40.123 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:40.123+0000 7fcffcf4e700 1 -- 172.21.15.160:0/84006817 shutdown_connections 2023-12-25T20:14:40.123 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:40.123+0000 7fcffcf4e700 1 --2- 172.21.15.160:0/84006817 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fcfe002ff60 0x7fcfe0032420 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:14:40.124 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:40.123+0000 7fcffcf4e700 1 --2- 172.21.15.160:0/84006817 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcff80601e0 0x7fcff80f69a0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:14:40.124 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:40.123+0000 7fcffcf4e700 1 -- 172.21.15.160:0/84006817 >> 172.21.15.160:0/84006817 conn(0x7fcff80f09a0 msgr2=0x7fcff80f1420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:14:40.124 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:40.123+0000 7fcffcf4e700 1 -- 172.21.15.160:0/84006817 shutdown_connections 2023-12-25T20:14:40.124 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:40.123+0000 7fcffcf4e700 1 -- 172.21.15.160:0/84006817 wait complete. 2023-12-25T20:14:40.124 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:14:40.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:14:39 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:14:38.694388+0000 mgr.smithi050.nbzhun (mgr.14162) 228 : cluster [DBG] pgmap v216: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:14:41.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:14:40 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:14:40.122442+0000 mon.smithi050 (mon.0) 286 : audit [DBG] from='client.? 172.21.15.160:0/84006817' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:14:41.511 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:14:41.511 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:14:42.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:14:41 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:14:40.694681+0000 mgr.smithi050.nbzhun (mgr.14162) 229 : cluster [DBG] pgmap v217: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:14:42.649 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:42.648+0000 7f1388a85700 1 -- 172.21.15.160:0/3323276237 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f13840f5410 msgr2=0x7f13840f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:14:42.649 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:42.648+0000 7f1388a85700 1 --2- 172.21.15.160:0/3323276237 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f13840f5410 0x7f13840f5830 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 rx=0x7f1378009fb0 tx=0x7f1378009b30).stop 2023-12-25T20:14:42.649 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:42.649+0000 7f1388a85700 1 -- 172.21.15.160:0/3323276237 shutdown_connections 2023-12-25T20:14:42.649 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:42.649+0000 7f1388a85700 1 --2- 172.21.15.160:0/3323276237 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f13840f5410 0x7f13840f5830 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:14:42.649 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:42.649+0000 7f1388a85700 1 -- 172.21.15.160:0/3323276237 >> 172.21.15.160:0/3323276237 conn(0x7f13840f0b70 msgr2=0x7f13840f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:14:42.649 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:42.649+0000 7f1388a85700 1 -- 172.21.15.160:0/3323276237 shutdown_connections 2023-12-25T20:14:42.649 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:42.649+0000 7f1388a85700 1 -- 172.21.15.160:0/3323276237 wait complete. 2023-12-25T20:14:42.649 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:42.649+0000 7f1388a85700 1 Processor -- start 2023-12-25T20:14:42.650 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:42.649+0000 7f1388a85700 1 -- start start 2023-12-25T20:14:42.650 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:42.649+0000 7f1388a85700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f13840f5410 0x7f1384109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:14:42.650 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:42.649+0000 7f1388a85700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1384060240 con 0x7f13840f5410 2023-12-25T20:14:42.651 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:42.650+0000 7f138259c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f13840f5410 0x7f1384109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:14:42.651 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:42.650+0000 7f138259c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f13840f5410 0x7f1384109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:53198/0 (socket says 172.21.15.160:53198) 2023-12-25T20:14:42.651 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:42.650+0000 7f138259c700 1 -- 172.21.15.160:0/3820505091 learned_addr learned my addr 172.21.15.160:0/3820505091 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:14:42.651 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:42.650+0000 7f138259c700 1 -- 172.21.15.160:0/3820505091 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1378005040 con 0x7f13840f5410 2023-12-25T20:14:42.652 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:42.650+0000 7f138259c700 1 --2- 172.21.15.160:0/3820505091 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f13840f5410 0x7f1384109430 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7f1378013510 tx=0x7f137800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:14:42.652 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:42.650+0000 7f1372ffd700 1 -- 172.21.15.160:0/3820505091 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1378009320 con 0x7f13840f5410 2023-12-25T20:14:42.652 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:42.650+0000 7f1372ffd700 1 -- 172.21.15.160:0/3820505091 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1378013f00 con 0x7f13840f5410 2023-12-25T20:14:42.652 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:42.650+0000 7f1388a85700 1 -- 172.21.15.160:0/3820505091 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1384060440 con 0x7f13840f5410 2023-12-25T20:14:42.652 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:42.650+0000 7f1372ffd700 1 -- 172.21.15.160:0/3820505091 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1378014d90 con 0x7f13840f5410 2023-12-25T20:14:42.652 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:42.650+0000 7f1388a85700 1 -- 172.21.15.160:0/3820505091 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1384109ae0 con 0x7f13840f5410 2023-12-25T20:14:42.652 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:42.651+0000 7f1372ffd700 1 -- 172.21.15.160:0/3820505091 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1378014420 con 0x7f13840f5410 2023-12-25T20:14:42.652 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:42.651+0000 7f1388a85700 1 -- 172.21.15.160:0/3820505091 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1384045ad0 con 0x7f13840f5410 2023-12-25T20:14:42.652 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:42.651+0000 7f1372ffd700 1 --2- 172.21.15.160:0/3820505091 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f136c02ff60 0x7f136c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:14:42.652 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:42.651+0000 7f1372ffd700 1 -- 172.21.15.160:0/3820505091 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1378040ff0 con 0x7f13840f5410 2023-12-25T20:14:42.652 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:42.652+0000 7f1381d9b700 1 --2- 172.21.15.160:0/3820505091 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f136c02ff60 0x7f136c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:14:42.652 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:42.652+0000 7f1381d9b700 1 --2- 172.21.15.160:0/3820505091 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f136c02ff60 0x7f136c032420 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f13740096a0 tx=0x7f1374006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:14:42.655 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:42.656+0000 7f1372ffd700 1 -- 172.21.15.160:0/3820505091 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f13780095f0 con 0x7f13840f5410 2023-12-25T20:14:42.865 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:42.865+0000 7f1388a85700 1 -- 172.21.15.160:0/3820505091 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f13840fae60 con 0x7f13840f5410 2023-12-25T20:14:42.865 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:42.866+0000 7f1372ffd700 1 -- 172.21.15.160:0/3820505091 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f137801db70 con 0x7f13840f5410 2023-12-25T20:14:42.866 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:14:42.866 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:14:42.867 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:42.867+0000 7f1388a85700 1 -- 172.21.15.160:0/3820505091 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f136c02ff60 msgr2=0x7f136c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:14:42.867 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:42.867+0000 7f1388a85700 1 --2- 172.21.15.160:0/3820505091 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f136c02ff60 0x7f136c032420 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f13740096a0 tx=0x7f1374006b40).stop 2023-12-25T20:14:42.867 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:42.867+0000 7f1388a85700 1 -- 172.21.15.160:0/3820505091 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f13840f5410 msgr2=0x7f1384109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:14:42.867 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:42.867+0000 7f1388a85700 1 --2- 172.21.15.160:0/3820505091 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f13840f5410 0x7f1384109430 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7f1378013510 tx=0x7f137800b040).stop 2023-12-25T20:14:42.867 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:42.867+0000 7f1388a85700 1 -- 172.21.15.160:0/3820505091 shutdown_connections 2023-12-25T20:14:42.867 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:42.867+0000 7f1388a85700 1 --2- 172.21.15.160:0/3820505091 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f136c02ff60 0x7f136c032420 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:14:42.868 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:42.867+0000 7f1388a85700 1 --2- 172.21.15.160:0/3820505091 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f13840f5410 0x7f1384109430 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:14:42.868 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:42.867+0000 7f1388a85700 1 -- 172.21.15.160:0/3820505091 >> 172.21.15.160:0/3820505091 conn(0x7f13840f0b70 msgr2=0x7f13840f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:14:42.868 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:42.867+0000 7f1388a85700 1 -- 172.21.15.160:0/3820505091 shutdown_connections 2023-12-25T20:14:42.868 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:42.867+0000 7f1388a85700 1 -- 172.21.15.160:0/3820505091 wait complete. 2023-12-25T20:14:42.868 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:14:43.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:14:42 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:14:42.866517+0000 mon.smithi050 (mon.0) 287 : audit [DBG] from='client.? 172.21.15.160:0/3820505091' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:14:44.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:14:43 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:14:42.695040+0000 mgr.smithi050.nbzhun (mgr.14162) 230 : cluster [DBG] pgmap v218: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:14:44.262 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:14:44.262 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:14:45.471 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:45.471+0000 7fbd8e85d700 1 -- 172.21.15.160:0/3243661364 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbd880f6480 msgr2=0x7fbd880f68a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:14:45.471 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:45.471+0000 7fbd8e85d700 1 --2- 172.21.15.160:0/3243661364 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbd880f6480 0x7fbd880f68a0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 rx=0x7fbd7c009fb0 tx=0x7fbd7c009b30).stop 2023-12-25T20:14:45.471 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:45.471+0000 7fbd8e85d700 1 -- 172.21.15.160:0/3243661364 shutdown_connections 2023-12-25T20:14:45.471 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:45.471+0000 7fbd8e85d700 1 --2- 172.21.15.160:0/3243661364 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbd880f6480 0x7fbd880f68a0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:14:45.471 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:45.471+0000 7fbd8e85d700 1 -- 172.21.15.160:0/3243661364 >> 172.21.15.160:0/3243661364 conn(0x7fbd880f1c00 msgr2=0x7fbd880f4020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:14:45.471 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:45.471+0000 7fbd8e85d700 1 -- 172.21.15.160:0/3243661364 shutdown_connections 2023-12-25T20:14:45.471 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:45.471+0000 7fbd8e85d700 1 -- 172.21.15.160:0/3243661364 wait complete. 2023-12-25T20:14:45.472 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:45.471+0000 7fbd8e85d700 1 Processor -- start 2023-12-25T20:14:45.472 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:45.472+0000 7fbd8e85d700 1 -- start start 2023-12-25T20:14:45.473 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:45.472+0000 7fbd8e85d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbd880f6480 0x7fbd881066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:14:45.473 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:45.472+0000 7fbd8e85d700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd88106bc0 con 0x7fbd880f6480 2023-12-25T20:14:45.473 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:45.472+0000 7fbd87fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbd880f6480 0x7fbd881066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:14:45.473 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:45.472+0000 7fbd87fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbd880f6480 0x7fbd881066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:53218/0 (socket says 172.21.15.160:53218) 2023-12-25T20:14:45.473 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:45.472+0000 7fbd87fff700 1 -- 172.21.15.160:0/3228030900 learned_addr learned my addr 172.21.15.160:0/3228030900 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:14:45.473 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:45.473+0000 7fbd87fff700 1 -- 172.21.15.160:0/3228030900 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd7c005040 con 0x7fbd880f6480 2023-12-25T20:14:45.473 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:45.473+0000 7fbd87fff700 1 --2- 172.21.15.160:0/3228030900 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbd880f6480 0x7fbd881066d0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7fbd7c013510 tx=0x7fbd7c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:14:45.473 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:45.473+0000 7fbd84ff9700 1 -- 172.21.15.160:0/3228030900 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbd7c009320 con 0x7fbd880f6480 2023-12-25T20:14:45.473 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:45.473+0000 7fbd8e85d700 1 -- 172.21.15.160:0/3228030900 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd88106dc0 con 0x7fbd880f6480 2023-12-25T20:14:45.473 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:45.473+0000 7fbd84ff9700 1 -- 172.21.15.160:0/3228030900 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbd7c013f00 con 0x7fbd880f6480 2023-12-25T20:14:45.473 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:45.473+0000 7fbd84ff9700 1 -- 172.21.15.160:0/3228030900 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbd7c014d90 con 0x7fbd880f6480 2023-12-25T20:14:45.473 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:45.473+0000 7fbd8e85d700 1 -- 172.21.15.160:0/3228030900 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd88107250 con 0x7fbd880f6480 2023-12-25T20:14:45.477 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:45.474+0000 7fbd84ff9700 1 -- 172.21.15.160:0/3228030900 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbd7c0263a0 con 0x7fbd880f6480 2023-12-25T20:14:45.477 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:45.474+0000 7fbd84ff9700 1 --2- 172.21.15.160:0/3228030900 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fbd7002ffb0 0x7fbd70032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:14:45.477 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:45.474+0000 7fbd84ff9700 1 -- 172.21.15.160:0/3228030900 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbd7c0410f0 con 0x7fbd880f6480 2023-12-25T20:14:45.477 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:45.474+0000 7fbd8e85d700 1 -- 172.21.15.160:0/3228030900 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd88045ad0 con 0x7fbd880f6480 2023-12-25T20:14:45.477 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:45.474+0000 7fbd877fe700 1 --2- 172.21.15.160:0/3228030900 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fbd7002ffb0 0x7fbd70032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:14:45.477 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:45.475+0000 7fbd877fe700 1 --2- 172.21.15.160:0/3228030900 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fbd7002ffb0 0x7fbd70032470 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7fbd780096a0 tx=0x7fbd78006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:14:45.478 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:45.479+0000 7fbd84ff9700 1 -- 172.21.15.160:0/3228030900 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbd7c012920 con 0x7fbd880f6480 2023-12-25T20:14:45.685 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:45.686+0000 7fbd8e85d700 1 -- 172.21.15.160:0/3228030900 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbd880fc030 con 0x7fbd880f6480 2023-12-25T20:14:45.686 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:45.686+0000 7fbd84ff9700 1 -- 172.21.15.160:0/3228030900 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbd7c02b020 con 0x7fbd880f6480 2023-12-25T20:14:45.686 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:14:45.686 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:14:45.687 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:45.688+0000 7fbd8e85d700 1 -- 172.21.15.160:0/3228030900 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fbd7002ffb0 msgr2=0x7fbd70032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:14:45.688 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:45.688+0000 7fbd8e85d700 1 --2- 172.21.15.160:0/3228030900 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fbd7002ffb0 0x7fbd70032470 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7fbd780096a0 tx=0x7fbd78006b40).stop 2023-12-25T20:14:45.688 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:45.688+0000 7fbd8e85d700 1 -- 172.21.15.160:0/3228030900 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbd880f6480 msgr2=0x7fbd881066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:14:45.688 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:45.688+0000 7fbd8e85d700 1 --2- 172.21.15.160:0/3228030900 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbd880f6480 0x7fbd881066d0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7fbd7c013510 tx=0x7fbd7c00b040).stop 2023-12-25T20:14:45.688 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:45.688+0000 7fbd8e85d700 1 -- 172.21.15.160:0/3228030900 shutdown_connections 2023-12-25T20:14:45.688 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:45.688+0000 7fbd8e85d700 1 --2- 172.21.15.160:0/3228030900 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fbd7002ffb0 0x7fbd70032470 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:14:45.688 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:45.688+0000 7fbd8e85d700 1 --2- 172.21.15.160:0/3228030900 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbd880f6480 0x7fbd881066d0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:14:45.688 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:45.688+0000 7fbd8e85d700 1 -- 172.21.15.160:0/3228030900 >> 172.21.15.160:0/3228030900 conn(0x7fbd880f1c00 msgr2=0x7fbd880f2720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:14:45.688 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:45.688+0000 7fbd8e85d700 1 -- 172.21.15.160:0/3228030900 shutdown_connections 2023-12-25T20:14:45.688 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:45.688+0000 7fbd8e85d700 1 -- 172.21.15.160:0/3228030900 wait complete. 2023-12-25T20:14:45.689 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:14:46.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:14:45 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:14:44.695418+0000 mgr.smithi050.nbzhun (mgr.14162) 231 : cluster [DBG] pgmap v219: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:14:46.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:14:45 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:14:45.687068+0000 mon.smithi050 (mon.0) 288 : audit [DBG] from='client.? 172.21.15.160:0/3228030900' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:14:47.062 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:14:47.062 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:14:48.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:14:47 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:14:46.695724+0000 mgr.smithi050.nbzhun (mgr.14162) 232 : cluster [DBG] pgmap v220: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:14:48.271 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:48.271+0000 7f0341959700 1 -- 172.21.15.160:0/3811073894 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f033c0f5410 msgr2=0x7f033c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:14:48.271 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:48.271+0000 7f0341959700 1 --2- 172.21.15.160:0/3811073894 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f033c0f5410 0x7f033c0f5830 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 rx=0x7f0330009fb0 tx=0x7f0330009b30).stop 2023-12-25T20:14:48.271 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:48.271+0000 7f0341959700 1 -- 172.21.15.160:0/3811073894 shutdown_connections 2023-12-25T20:14:48.271 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:48.271+0000 7f0341959700 1 --2- 172.21.15.160:0/3811073894 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f033c0f5410 0x7f033c0f5830 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:14:48.271 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:48.271+0000 7f0341959700 1 -- 172.21.15.160:0/3811073894 >> 172.21.15.160:0/3811073894 conn(0x7f033c0f0b70 msgr2=0x7f033c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:14:48.272 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:48.271+0000 7f0341959700 1 -- 172.21.15.160:0/3811073894 shutdown_connections 2023-12-25T20:14:48.272 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:48.271+0000 7f0341959700 1 -- 172.21.15.160:0/3811073894 wait complete. 2023-12-25T20:14:48.273 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:48.272+0000 7f0341959700 1 Processor -- start 2023-12-25T20:14:48.273 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:48.272+0000 7f0341959700 1 -- start start 2023-12-25T20:14:48.273 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:48.272+0000 7f0341959700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f033c0f5410 0x7f033c105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:14:48.273 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:48.272+0000 7f0341959700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f033c105b20 con 0x7f033c0f5410 2023-12-25T20:14:48.273 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:48.272+0000 7f033affd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f033c0f5410 0x7f033c105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:14:48.273 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:48.273+0000 7f033affd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f033c0f5410 0x7f033c105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:45932/0 (socket says 172.21.15.160:45932) 2023-12-25T20:14:48.273 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:48.273+0000 7f033affd700 1 -- 172.21.15.160:0/635038089 learned_addr learned my addr 172.21.15.160:0/635038089 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:14:48.274 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:48.273+0000 7f033affd700 1 -- 172.21.15.160:0/635038089 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0330005040 con 0x7f033c0f5410 2023-12-25T20:14:48.274 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:48.273+0000 7f033affd700 1 --2- 172.21.15.160:0/635038089 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f033c0f5410 0x7f033c105630 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7f0330013510 tx=0x7f033000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:14:48.274 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:48.273+0000 7f03237fe700 1 -- 172.21.15.160:0/635038089 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0330009320 con 0x7f033c0f5410 2023-12-25T20:14:48.274 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:48.273+0000 7f0341959700 1 -- 172.21.15.160:0/635038089 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f033c105d20 con 0x7f033c0f5410 2023-12-25T20:14:48.274 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:48.273+0000 7f03237fe700 1 -- 172.21.15.160:0/635038089 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0330013f00 con 0x7f033c0f5410 2023-12-25T20:14:48.274 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:48.273+0000 7f03237fe700 1 -- 172.21.15.160:0/635038089 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0330014d90 con 0x7f033c0f5410 2023-12-25T20:14:48.274 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:48.273+0000 7f0341959700 1 -- 172.21.15.160:0/635038089 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f033c1061b0 con 0x7f033c0f5410 2023-12-25T20:14:48.275 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:48.274+0000 7f03237fe700 1 -- 172.21.15.160:0/635038089 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f03300095f0 con 0x7f033c0f5410 2023-12-25T20:14:48.275 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:48.274+0000 7f0341959700 1 -- 172.21.15.160:0/635038089 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f033c045ad0 con 0x7f033c0f5410 2023-12-25T20:14:48.275 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:48.274+0000 7f03237fe700 1 --2- 172.21.15.160:0/635038089 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f032402ff60 0x7f0324032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:14:48.275 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:48.274+0000 7f03237fe700 1 -- 172.21.15.160:0/635038089 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f03300410c0 con 0x7f033c0f5410 2023-12-25T20:14:48.275 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:48.274+0000 7f033a7fc700 1 --2- 172.21.15.160:0/635038089 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f032402ff60 0x7f0324032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:14:48.275 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:48.275+0000 7f033a7fc700 1 --2- 172.21.15.160:0/635038089 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f032402ff60 0x7f0324032420 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f032c0096a0 tx=0x7f032c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:14:48.279 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:48.279+0000 7f03237fe700 1 -- 172.21.15.160:0/635038089 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f033001d7f0 con 0x7f033c0f5410 2023-12-25T20:14:48.483 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:48.484+0000 7f0341959700 1 -- 172.21.15.160:0/635038089 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f033c0fae60 con 0x7f033c0f5410 2023-12-25T20:14:48.484 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:48.484+0000 7f03237fe700 1 -- 172.21.15.160:0/635038089 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0330010230 con 0x7f033c0f5410 2023-12-25T20:14:48.484 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:14:48.484 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:14:48.486 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:48.486+0000 7f0341959700 1 -- 172.21.15.160:0/635038089 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f032402ff60 msgr2=0x7f0324032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:14:48.486 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:48.486+0000 7f0341959700 1 --2- 172.21.15.160:0/635038089 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f032402ff60 0x7f0324032420 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f032c0096a0 tx=0x7f032c006b40).stop 2023-12-25T20:14:48.486 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:48.486+0000 7f0341959700 1 -- 172.21.15.160:0/635038089 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f033c0f5410 msgr2=0x7f033c105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:14:48.486 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:48.486+0000 7f0341959700 1 --2- 172.21.15.160:0/635038089 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f033c0f5410 0x7f033c105630 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7f0330013510 tx=0x7f033000b040).stop 2023-12-25T20:14:48.486 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:48.486+0000 7f0341959700 1 -- 172.21.15.160:0/635038089 shutdown_connections 2023-12-25T20:14:48.486 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:48.486+0000 7f0341959700 1 --2- 172.21.15.160:0/635038089 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f032402ff60 0x7f0324032420 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:14:48.486 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:48.486+0000 7f0341959700 1 --2- 172.21.15.160:0/635038089 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f033c0f5410 0x7f033c105630 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:14:48.486 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:48.486+0000 7f0341959700 1 -- 172.21.15.160:0/635038089 >> 172.21.15.160:0/635038089 conn(0x7f033c0f0b70 msgr2=0x7f033c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:14:48.486 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:48.486+0000 7f0341959700 1 -- 172.21.15.160:0/635038089 shutdown_connections 2023-12-25T20:14:48.486 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:48.486+0000 7f0341959700 1 -- 172.21.15.160:0/635038089 wait complete. 2023-12-25T20:14:48.487 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:14:49.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:14:48 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:14:48.485123+0000 mon.smithi050 (mon.0) 289 : audit [DBG] from='client.? 172.21.15.160:0/635038089' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:14:49.882 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:14:49.882 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:14:50.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:14:49 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:14:48.696036+0000 mgr.smithi050.nbzhun (mgr.14162) 233 : cluster [DBG] pgmap v221: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:14:51.086 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:51.086+0000 7ffa31312700 1 -- 172.21.15.160:0/1188765838 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffa2c0f5410 msgr2=0x7ffa2c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:14:51.087 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:51.086+0000 7ffa31312700 1 --2- 172.21.15.160:0/1188765838 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffa2c0f5410 0x7ffa2c0f5830 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 rx=0x7ffa1c009fb0 tx=0x7ffa1c009b30).stop 2023-12-25T20:14:51.087 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:51.087+0000 7ffa31312700 1 -- 172.21.15.160:0/1188765838 shutdown_connections 2023-12-25T20:14:51.087 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:51.087+0000 7ffa31312700 1 --2- 172.21.15.160:0/1188765838 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffa2c0f5410 0x7ffa2c0f5830 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:14:51.087 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:51.087+0000 7ffa31312700 1 -- 172.21.15.160:0/1188765838 >> 172.21.15.160:0/1188765838 conn(0x7ffa2c0f0b70 msgr2=0x7ffa2c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:14:51.087 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:51.087+0000 7ffa31312700 1 -- 172.21.15.160:0/1188765838 shutdown_connections 2023-12-25T20:14:51.087 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:51.087+0000 7ffa31312700 1 -- 172.21.15.160:0/1188765838 wait complete. 2023-12-25T20:14:51.088 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:51.087+0000 7ffa31312700 1 Processor -- start 2023-12-25T20:14:51.088 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:51.087+0000 7ffa31312700 1 -- start start 2023-12-25T20:14:51.088 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:51.088+0000 7ffa31312700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffa2c0f5410 0x7ffa2c1093f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:14:51.089 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:51.088+0000 7ffa31312700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa2c060220 con 0x7ffa2c0f5410 2023-12-25T20:14:51.089 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:51.088+0000 7ffa2affd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffa2c0f5410 0x7ffa2c1093f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:14:51.089 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:51.088+0000 7ffa2affd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffa2c0f5410 0x7ffa2c1093f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:45958/0 (socket says 172.21.15.160:45958) 2023-12-25T20:14:51.089 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:51.088+0000 7ffa2affd700 1 -- 172.21.15.160:0/4257345478 learned_addr learned my addr 172.21.15.160:0/4257345478 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:14:51.089 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:51.088+0000 7ffa2affd700 1 -- 172.21.15.160:0/4257345478 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa1c005040 con 0x7ffa2c0f5410 2023-12-25T20:14:51.089 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:51.088+0000 7ffa2affd700 1 --2- 172.21.15.160:0/4257345478 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffa2c0f5410 0x7ffa2c1093f0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7ffa1c014000 tx=0x7ffa1c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:14:51.089 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:51.089+0000 7ffa237fe700 1 -- 172.21.15.160:0/4257345478 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffa1c0095f0 con 0x7ffa2c0f5410 2023-12-25T20:14:51.089 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:51.089+0000 7ffa31312700 1 -- 172.21.15.160:0/4257345478 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa2c060420 con 0x7ffa2c0f5410 2023-12-25T20:14:51.089 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:51.089+0000 7ffa237fe700 1 -- 172.21.15.160:0/4257345478 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ffa1c007e70 con 0x7ffa2c0f5410 2023-12-25T20:14:51.089 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:51.089+0000 7ffa237fe700 1 -- 172.21.15.160:0/4257345478 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffa1c014ea0 con 0x7ffa2c0f5410 2023-12-25T20:14:51.089 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:51.089+0000 7ffa31312700 1 -- 172.21.15.160:0/4257345478 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa2c109aa0 con 0x7ffa2c0f5410 2023-12-25T20:14:51.090 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:51.089+0000 7ffa237fe700 1 -- 172.21.15.160:0/4257345478 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ffa1c025070 con 0x7ffa2c0f5410 2023-12-25T20:14:51.090 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:51.090+0000 7ffa31312700 1 -- 172.21.15.160:0/4257345478 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa2c059100 con 0x7ffa2c0f5410 2023-12-25T20:14:51.090 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:51.090+0000 7ffa237fe700 1 --2- 172.21.15.160:0/4257345478 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7ffa1802ff60 0x7ffa18032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:14:51.090 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:51.090+0000 7ffa237fe700 1 -- 172.21.15.160:0/4257345478 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ffa1c040930 con 0x7ffa2c0f5410 2023-12-25T20:14:51.090 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:51.090+0000 7ffa23fff700 1 --2- 172.21.15.160:0/4257345478 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7ffa1802ff60 0x7ffa18032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:14:51.090 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:51.090+0000 7ffa23fff700 1 --2- 172.21.15.160:0/4257345478 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7ffa1802ff60 0x7ffa18032420 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7ffa140096a0 tx=0x7ffa14006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:14:51.094 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:51.095+0000 7ffa237fe700 1 -- 172.21.15.160:0/4257345478 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ffa1c00f4e0 con 0x7ffa2c0f5410 2023-12-25T20:14:51.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:14:50 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cephadm 2023-12-25T20:14:49.759461+0000 mgr.smithi050.nbzhun (mgr.14162) 234 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon node-exporter.smithi050 ... 2023-12-25T20:14:51.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:14:50 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: Verifying port 9100 ... 2023-12-25T20:14:51.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:14:50 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: Non-zero exit code 1 from systemctl start ceph-02c8f25c-a361-11ee-95a5-87774f69a715@node-exporter.smithi050 2023-12-25T20:14:51.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:14:50 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: systemctl: stderr Job for ceph-02c8f25c-a361-11ee-95a5-87774f69a715@node-exporter.smithi050.service failed because a timeout was exceeded. 2023-12-25T20:14:51.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:14:50 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: systemctl: stderr See "systemctl status ceph-02c8f25c-a361-11ee-95a5-87774f69a715@node-exporter.smithi050.service" and "journalctl -xe" for details. 2023-12-25T20:14:51.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:14:50 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: Traceback (most recent call last): 2023-12-25T20:14:51.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:14:50 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: File "/var/lib/ceph/02c8f25c-a361-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2023-12-25T20:14:51.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:14:50 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: main() 2023-12-25T20:14:51.220 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:14:50 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: File "/var/lib/ceph/02c8f25c-a361-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2023-12-25T20:14:51.220 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:14:50 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: r = ctx.func(ctx) 2023-12-25T20:14:51.220 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:14:50 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: File "/var/lib/ceph/02c8f25c-a361-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2023-12-25T20:14:51.220 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:14:50 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: return func(ctx) 2023-12-25T20:14:51.220 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:14:50 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: File "/var/lib/ceph/02c8f25c-a361-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2023-12-25T20:14:51.220 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:14:50 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: ports=daemon_ports) 2023-12-25T20:14:51.220 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:14:50 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: File "/var/lib/ceph/02c8f25c-a361-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2608, in deploy_daemon 2023-12-25T20:14:51.220 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:14:50 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: c, osd_fsid=osd_fsid, ports=ports) 2023-12-25T20:14:51.220 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:14:50 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: File "/var/lib/ceph/02c8f25c-a361-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2806, in deploy_daemon_units 2023-12-25T20:14:51.220 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:14:50 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: call_throws(ctx, ['systemctl', 'start', unit_name]) 2023-12-25T20:14:51.220 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:14:50 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: File "/var/lib/ceph/02c8f25c-a361-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2023-12-25T20:14:51.220 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:14:50 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2023-12-25T20:14:51.220 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:14:50 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: RuntimeError: Failed command: systemctl start ceph-02c8f25c-a361-11ee-95a5-87774f69a715@node-exporter.smithi050 2023-12-25T20:14:51.220 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:14:50 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: Traceback (most recent call last): 2023-12-25T20:14:51.220 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:14:50 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2023-12-25T20:14:51.221 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:14:50 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: yield (conn, connr) 2023-12-25T20:14:51.221 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:14:50 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2023-12-25T20:14:51.221 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:14:50 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: code, '\n'.join(err))) 2023-12-25T20:14:51.221 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:14:50 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon node-exporter.smithi050 ... 2023-12-25T20:14:51.221 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:14:50 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: Verifying port 9100 ... 2023-12-25T20:14:51.221 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:14:50 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: Non-zero exit code 1 from systemctl start ceph-02c8f25c-a361-11ee-95a5-87774f69a715@node-exporter.smithi050 2023-12-25T20:14:51.221 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:14:50 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: systemctl: stderr Job for ceph-02c8f25c-a361-11ee-95a5-87774f69a715@node-exporter.smithi050.service failed because a timeout was exceeded. 2023-12-25T20:14:51.221 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:14:50 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: systemctl: stderr See "systemctl status ceph-02c8f25c-a361-11ee-95a5-87774f69a715@node-exporter.smithi050.service" and "journalctl -xe" for details. 2023-12-25T20:14:51.221 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:14:50 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: Traceback (most recent call last): 2023-12-25T20:14:51.221 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:14:50 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: File "/var/lib/ceph/02c8f25c-a361-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2023-12-25T20:14:51.221 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:14:50 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: main() 2023-12-25T20:14:51.221 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:14:50 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: File "/var/lib/ceph/02c8f25c-a361-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2023-12-25T20:14:51.221 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:14:50 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: r = ctx.func(ctx) 2023-12-25T20:14:51.221 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:14:50 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: File "/var/lib/ceph/02c8f25c-a361-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2023-12-25T20:14:51.222 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:14:50 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: return func(ctx) 2023-12-25T20:14:51.222 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:14:50 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: File "/var/lib/ceph/02c8f25c-a361-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2023-12-25T20:14:51.222 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:14:50 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: ports=daemon_ports) 2023-12-25T20:14:51.222 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:14:50 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: File "/var/lib/ceph/02c8f25c-a361-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2608, in deploy_daemon 2023-12-25T20:14:51.222 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:14:50 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: c, osd_fsid=osd_fsid, ports=ports) 2023-12-25T20:14:51.222 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:14:50 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: File "/var/lib/ceph/02c8f25c-a361-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2806, in deploy_daemon_units 2023-12-25T20:14:51.222 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:14:50 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: call_throws(ctx, ['systemctl', 'start', unit_name]) 2023-12-25T20:14:51.222 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:14:50 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: File "/var/lib/ceph/02c8f25c-a361-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2023-12-25T20:14:51.222 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:14:50 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2023-12-25T20:14:51.222 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:14:50 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: RuntimeError: Failed command: systemctl start ceph-02c8f25c-a361-11ee-95a5-87774f69a715@node-exporter.smithi050 2023-12-25T20:14:51.222 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:14:50 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cephadm 2023-12-25T20:14:49.765161+0000 mgr.smithi050.nbzhun (mgr.14162) 235 : cephadm [INF] Deploying daemon prometheus.smithi050 on smithi050 2023-12-25T20:14:51.298 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:51.298+0000 7ffa31312700 1 -- 172.21.15.160:0/4257345478 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ffa2c045ad0 con 0x7ffa2c0f5410 2023-12-25T20:14:51.298 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:51.299+0000 7ffa237fe700 1 -- 172.21.15.160:0/4257345478 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ffa1c00f300 con 0x7ffa2c0f5410 2023-12-25T20:14:51.298 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:14:51.298 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:14:51.300 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:51.300+0000 7ffa31312700 1 -- 172.21.15.160:0/4257345478 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7ffa1802ff60 msgr2=0x7ffa18032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:14:51.301 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:51.300+0000 7ffa31312700 1 --2- 172.21.15.160:0/4257345478 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7ffa1802ff60 0x7ffa18032420 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7ffa140096a0 tx=0x7ffa14006b40).stop 2023-12-25T20:14:51.301 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:51.300+0000 7ffa31312700 1 -- 172.21.15.160:0/4257345478 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffa2c0f5410 msgr2=0x7ffa2c1093f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:14:51.301 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:51.300+0000 7ffa31312700 1 --2- 172.21.15.160:0/4257345478 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffa2c0f5410 0x7ffa2c1093f0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7ffa1c014000 tx=0x7ffa1c00b040).stop 2023-12-25T20:14:51.301 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:51.300+0000 7ffa31312700 1 -- 172.21.15.160:0/4257345478 shutdown_connections 2023-12-25T20:14:51.301 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:51.300+0000 7ffa31312700 1 --2- 172.21.15.160:0/4257345478 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7ffa1802ff60 0x7ffa18032420 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:14:51.301 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:51.300+0000 7ffa31312700 1 --2- 172.21.15.160:0/4257345478 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffa2c0f5410 0x7ffa2c1093f0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:14:51.301 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:51.300+0000 7ffa31312700 1 -- 172.21.15.160:0/4257345478 >> 172.21.15.160:0/4257345478 conn(0x7ffa2c0f0b70 msgr2=0x7ffa2c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:14:51.301 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:51.300+0000 7ffa31312700 1 -- 172.21.15.160:0/4257345478 shutdown_connections 2023-12-25T20:14:51.301 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:51.300+0000 7ffa31312700 1 -- 172.21.15.160:0/4257345478 wait complete. 2023-12-25T20:14:51.301 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:14:52.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:14:51 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:14:50.696421+0000 mgr.smithi050.nbzhun (mgr.14162) 236 : cluster [DBG] pgmap v222: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:14:52.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:14:51 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:14:51.299394+0000 mon.smithi050 (mon.0) 290 : audit [DBG] from='client.? 172.21.15.160:0/4257345478' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:14:52.696 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:14:52.696 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:14:53.854 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:53.853+0000 7f41cffa2700 1 -- 172.21.15.160:0/3496310903 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f41c80f5410 msgr2=0x7f41c80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:14:53.854 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:53.853+0000 7f41cffa2700 1 --2- 172.21.15.160:0/3496310903 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f41c80f5410 0x7f41c80f5830 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 rx=0x7f41b8009fb0 tx=0x7f41b8009b30).stop 2023-12-25T20:14:53.854 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:53.853+0000 7f41cffa2700 1 -- 172.21.15.160:0/3496310903 shutdown_connections 2023-12-25T20:14:53.854 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:53.853+0000 7f41cffa2700 1 --2- 172.21.15.160:0/3496310903 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f41c80f5410 0x7f41c80f5830 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:14:53.854 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:53.853+0000 7f41cffa2700 1 -- 172.21.15.160:0/3496310903 >> 172.21.15.160:0/3496310903 conn(0x7f41c80f0b70 msgr2=0x7f41c80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:14:53.854 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:53.853+0000 7f41cffa2700 1 -- 172.21.15.160:0/3496310903 shutdown_connections 2023-12-25T20:14:53.854 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:53.853+0000 7f41cffa2700 1 -- 172.21.15.160:0/3496310903 wait complete. 2023-12-25T20:14:53.854 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:53.854+0000 7f41cffa2700 1 Processor -- start 2023-12-25T20:14:53.854 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:53.854+0000 7f41cffa2700 1 -- start start 2023-12-25T20:14:53.854 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:53.854+0000 7f41cffa2700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f41c80f5410 0x7f41c8105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:14:53.855 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:53.854+0000 7f41cffa2700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41c8105b20 con 0x7f41c80f5410 2023-12-25T20:14:53.855 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:53.854+0000 7f41cdd3e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f41c80f5410 0x7f41c8105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:14:53.855 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:53.854+0000 7f41cdd3e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f41c80f5410 0x7f41c8105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:45966/0 (socket says 172.21.15.160:45966) 2023-12-25T20:14:53.855 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:53.854+0000 7f41cdd3e700 1 -- 172.21.15.160:0/1812716548 learned_addr learned my addr 172.21.15.160:0/1812716548 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:14:53.855 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:53.854+0000 7f41cdd3e700 1 -- 172.21.15.160:0/1812716548 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41b8005040 con 0x7f41c80f5410 2023-12-25T20:14:53.856 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:53.855+0000 7f41cdd3e700 1 --2- 172.21.15.160:0/1812716548 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f41c80f5410 0x7f41c8105630 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7f41b8013510 tx=0x7f41b800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:14:53.856 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:53.855+0000 7f41be7fc700 1 -- 172.21.15.160:0/1812716548 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f41b8009320 con 0x7f41c80f5410 2023-12-25T20:14:53.856 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:53.855+0000 7f41cffa2700 1 -- 172.21.15.160:0/1812716548 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41c8105d20 con 0x7f41c80f5410 2023-12-25T20:14:53.856 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:53.855+0000 7f41be7fc700 1 -- 172.21.15.160:0/1812716548 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f41b8013f00 con 0x7f41c80f5410 2023-12-25T20:14:53.856 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:53.855+0000 7f41be7fc700 1 -- 172.21.15.160:0/1812716548 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f41b8014d90 con 0x7f41c80f5410 2023-12-25T20:14:53.856 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:53.855+0000 7f41cffa2700 1 -- 172.21.15.160:0/1812716548 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41c81061b0 con 0x7f41c80f5410 2023-12-25T20:14:53.857 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:53.855+0000 7f41be7fc700 1 -- 172.21.15.160:0/1812716548 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f41b80263a0 con 0x7f41c80f5410 2023-12-25T20:14:53.857 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:53.856+0000 7f41cffa2700 1 -- 172.21.15.160:0/1812716548 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41c8045ad0 con 0x7f41c80f5410 2023-12-25T20:14:53.857 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:53.856+0000 7f41be7fc700 1 --2- 172.21.15.160:0/1812716548 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f41b402ff60 0x7f41b4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:14:53.857 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:53.856+0000 7f41be7fc700 1 -- 172.21.15.160:0/1812716548 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f41b80410c0 con 0x7f41c80f5410 2023-12-25T20:14:53.857 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:53.856+0000 7f41cd53d700 1 --2- 172.21.15.160:0/1812716548 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f41b402ff60 0x7f41b4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:14:53.857 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:53.856+0000 7f41cd53d700 1 --2- 172.21.15.160:0/1812716548 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f41b402ff60 0x7f41b4032420 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f41c40096a0 tx=0x7f41c4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:14:53.860 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:53.861+0000 7f41be7fc700 1 -- 172.21.15.160:0/1812716548 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f41b80095f0 con 0x7f41c80f5410 2023-12-25T20:14:54.068 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:54.068+0000 7f41cffa2700 1 -- 172.21.15.160:0/1812716548 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f41c80fae60 con 0x7f41c80f5410 2023-12-25T20:14:54.068 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:54.068+0000 7f41be7fc700 1 -- 172.21.15.160:0/1812716548 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f41b801db70 con 0x7f41c80f5410 2023-12-25T20:14:54.068 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:14:54.068 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:14:54.070 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:54.070+0000 7f41cffa2700 1 -- 172.21.15.160:0/1812716548 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f41b402ff60 msgr2=0x7f41b4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:14:54.070 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:54.070+0000 7f41cffa2700 1 --2- 172.21.15.160:0/1812716548 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f41b402ff60 0x7f41b4032420 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f41c40096a0 tx=0x7f41c4006b40).stop 2023-12-25T20:14:54.070 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:54.070+0000 7f41cffa2700 1 -- 172.21.15.160:0/1812716548 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f41c80f5410 msgr2=0x7f41c8105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:14:54.070 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:54.070+0000 7f41cffa2700 1 --2- 172.21.15.160:0/1812716548 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f41c80f5410 0x7f41c8105630 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7f41b8013510 tx=0x7f41b800b040).stop 2023-12-25T20:14:54.070 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:54.070+0000 7f41cffa2700 1 -- 172.21.15.160:0/1812716548 shutdown_connections 2023-12-25T20:14:54.070 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:54.070+0000 7f41cffa2700 1 --2- 172.21.15.160:0/1812716548 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f41b402ff60 0x7f41b4032420 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:14:54.070 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:54.070+0000 7f41cffa2700 1 --2- 172.21.15.160:0/1812716548 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f41c80f5410 0x7f41c8105630 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:14:54.070 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:54.070+0000 7f41cffa2700 1 -- 172.21.15.160:0/1812716548 >> 172.21.15.160:0/1812716548 conn(0x7f41c80f0b70 msgr2=0x7f41c80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:14:54.070 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:54.070+0000 7f41cffa2700 1 -- 172.21.15.160:0/1812716548 shutdown_connections 2023-12-25T20:14:54.070 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:54.070+0000 7f41cffa2700 1 -- 172.21.15.160:0/1812716548 wait complete. 2023-12-25T20:14:54.071 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:14:54.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:14:53 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:14:52.696803+0000 mgr.smithi050.nbzhun (mgr.14162) 237 : cluster [DBG] pgmap v223: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:14:55.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:14:54 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:14:54.069306+0000 mon.smithi050 (mon.0) 291 : audit [DBG] from='client.? 172.21.15.160:0/1812716548' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:14:55.501 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:14:55.501 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:14:56.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:14:55 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:14:54.697108+0000 mgr.smithi050.nbzhun (mgr.14162) 238 : cluster [DBG] pgmap v224: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:14:56.654 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:56.649+0000 7f2ba3212700 1 -- 172.21.15.160:0/426957699 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2b9c0f5410 msgr2=0x7f2b9c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:14:56.654 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:56.649+0000 7f2ba3212700 1 --2- 172.21.15.160:0/426957699 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2b9c0f5410 0x7f2b9c0f5830 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 rx=0x7f2b90009fb0 tx=0x7f2b90009b30).stop 2023-12-25T20:14:56.654 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:56.649+0000 7f2ba3212700 1 -- 172.21.15.160:0/426957699 shutdown_connections 2023-12-25T20:14:56.654 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:56.649+0000 7f2ba3212700 1 --2- 172.21.15.160:0/426957699 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2b9c0f5410 0x7f2b9c0f5830 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:14:56.654 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:56.649+0000 7f2ba3212700 1 -- 172.21.15.160:0/426957699 >> 172.21.15.160:0/426957699 conn(0x7f2b9c0f0b70 msgr2=0x7f2b9c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:14:56.654 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:56.649+0000 7f2ba3212700 1 -- 172.21.15.160:0/426957699 shutdown_connections 2023-12-25T20:14:56.654 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:56.650+0000 7f2ba3212700 1 -- 172.21.15.160:0/426957699 wait complete. 2023-12-25T20:14:56.655 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:56.650+0000 7f2ba3212700 1 Processor -- start 2023-12-25T20:14:56.655 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:56.650+0000 7f2ba3212700 1 -- start start 2023-12-25T20:14:56.655 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:56.650+0000 7f2ba3212700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2b9c0f5410 0x7f2b9c1093f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:14:56.655 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:56.650+0000 7f2ba3212700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b9c060280 con 0x7f2b9c0f5410 2023-12-25T20:14:56.655 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:56.651+0000 7f2ba0fae700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2b9c0f5410 0x7f2b9c1093f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:14:56.655 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:56.651+0000 7f2ba0fae700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2b9c0f5410 0x7f2b9c1093f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:45990/0 (socket says 172.21.15.160:45990) 2023-12-25T20:14:56.655 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:56.651+0000 7f2ba0fae700 1 -- 172.21.15.160:0/4162936581 learned_addr learned my addr 172.21.15.160:0/4162936581 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:14:56.655 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:56.651+0000 7f2ba0fae700 1 -- 172.21.15.160:0/4162936581 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b90005040 con 0x7f2b9c0f5410 2023-12-25T20:14:56.656 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:56.651+0000 7f2ba0fae700 1 --2- 172.21.15.160:0/4162936581 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2b9c0f5410 0x7f2b9c1093f0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7f2b90014000 tx=0x7f2b9000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:14:56.656 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:56.651+0000 7f2b997fa700 1 -- 172.21.15.160:0/4162936581 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2b900095f0 con 0x7f2b9c0f5410 2023-12-25T20:14:56.656 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:56.651+0000 7f2ba3212700 1 -- 172.21.15.160:0/4162936581 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b9c060480 con 0x7f2b9c0f5410 2023-12-25T20:14:56.656 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:56.651+0000 7f2b997fa700 1 -- 172.21.15.160:0/4162936581 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2b90007e70 con 0x7f2b9c0f5410 2023-12-25T20:14:56.656 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:56.651+0000 7f2b997fa700 1 -- 172.21.15.160:0/4162936581 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2b90014ea0 con 0x7f2b9c0f5410 2023-12-25T20:14:56.656 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:56.651+0000 7f2ba3212700 1 -- 172.21.15.160:0/4162936581 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b9c109b20 con 0x7f2b9c0f5410 2023-12-25T20:14:56.656 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:56.652+0000 7f2b997fa700 1 -- 172.21.15.160:0/4162936581 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2b900079c0 con 0x7f2b9c0f5410 2023-12-25T20:14:56.656 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:56.652+0000 7f2ba3212700 1 -- 172.21.15.160:0/4162936581 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b9c045ad0 con 0x7f2b9c0f5410 2023-12-25T20:14:56.657 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:56.652+0000 7f2b997fa700 1 --2- 172.21.15.160:0/4162936581 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f2b8c02ffb0 0x7f2b8c032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:14:56.657 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:56.652+0000 7f2b997fa700 1 -- 172.21.15.160:0/4162936581 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2b90040920 con 0x7f2b9c0f5410 2023-12-25T20:14:56.657 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:56.652+0000 7f2b9bfff700 1 --2- 172.21.15.160:0/4162936581 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f2b8c02ffb0 0x7f2b8c032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:14:56.657 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:56.653+0000 7f2b9bfff700 1 --2- 172.21.15.160:0/4162936581 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f2b8c02ffb0 0x7f2b8c032470 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f2b880096a0 tx=0x7f2b88006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:14:56.657 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:56.657+0000 7f2b997fa700 1 -- 172.21.15.160:0/4162936581 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2b9000f950 con 0x7f2b9c0f5410 2023-12-25T20:14:56.864 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:56.863+0000 7f2ba3212700 1 -- 172.21.15.160:0/4162936581 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2b9c10a340 con 0x7f2b9c0f5410 2023-12-25T20:14:56.865 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:56.864+0000 7f2b997fa700 1 -- 172.21.15.160:0/4162936581 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2b90010230 con 0x7f2b9c0f5410 2023-12-25T20:14:56.865 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:14:56.865 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:14:56.866 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:56.865+0000 7f2ba3212700 1 -- 172.21.15.160:0/4162936581 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f2b8c02ffb0 msgr2=0x7f2b8c032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:14:56.866 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:56.865+0000 7f2ba3212700 1 --2- 172.21.15.160:0/4162936581 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f2b8c02ffb0 0x7f2b8c032470 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f2b880096a0 tx=0x7f2b88006b40).stop 2023-12-25T20:14:56.866 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:56.865+0000 7f2ba3212700 1 -- 172.21.15.160:0/4162936581 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2b9c0f5410 msgr2=0x7f2b9c1093f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:14:56.866 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:56.865+0000 7f2ba3212700 1 --2- 172.21.15.160:0/4162936581 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2b9c0f5410 0x7f2b9c1093f0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7f2b90014000 tx=0x7f2b9000b040).stop 2023-12-25T20:14:56.866 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:56.865+0000 7f2ba3212700 1 -- 172.21.15.160:0/4162936581 shutdown_connections 2023-12-25T20:14:56.867 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:56.865+0000 7f2ba3212700 1 --2- 172.21.15.160:0/4162936581 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f2b8c02ffb0 0x7f2b8c032470 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:14:56.867 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:56.866+0000 7f2ba3212700 1 --2- 172.21.15.160:0/4162936581 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2b9c0f5410 0x7f2b9c1093f0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:14:56.867 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:56.866+0000 7f2ba3212700 1 -- 172.21.15.160:0/4162936581 >> 172.21.15.160:0/4162936581 conn(0x7f2b9c0f0b70 msgr2=0x7f2b9c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:14:56.867 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:56.866+0000 7f2ba3212700 1 -- 172.21.15.160:0/4162936581 shutdown_connections 2023-12-25T20:14:56.867 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:56.866+0000 7f2ba3212700 1 -- 172.21.15.160:0/4162936581 wait complete. 2023-12-25T20:14:56.867 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:14:57.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:14:56 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:14:56.865668+0000 mon.smithi050 (mon.0) 292 : audit [DBG] from='client.? 172.21.15.160:0/4162936581' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:14:58.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:14:57 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:14:56.697435+0000 mgr.smithi050.nbzhun (mgr.14162) 239 : cluster [DBG] pgmap v225: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:14:58.287 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:14:58.287 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:14:59.436 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:59.434+0000 7f8939a10700 1 -- 172.21.15.160:0/3169886979 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f89340f5410 msgr2=0x7f89340f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:14:59.436 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:59.434+0000 7f8939a10700 1 --2- 172.21.15.160:0/3169886979 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f89340f5410 0x7f89340f5830 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 rx=0x7f8924009fb0 tx=0x7f8924009b30).stop 2023-12-25T20:14:59.436 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:59.434+0000 7f8939a10700 1 -- 172.21.15.160:0/3169886979 shutdown_connections 2023-12-25T20:14:59.436 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:59.434+0000 7f8939a10700 1 --2- 172.21.15.160:0/3169886979 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f89340f5410 0x7f89340f5830 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:14:59.436 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:59.434+0000 7f8939a10700 1 -- 172.21.15.160:0/3169886979 >> 172.21.15.160:0/3169886979 conn(0x7f89340f0b70 msgr2=0x7f89340f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:14:59.436 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:59.434+0000 7f8939a10700 1 -- 172.21.15.160:0/3169886979 shutdown_connections 2023-12-25T20:14:59.436 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:59.434+0000 7f8939a10700 1 -- 172.21.15.160:0/3169886979 wait complete. 2023-12-25T20:14:59.436 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:59.435+0000 7f8939a10700 1 Processor -- start 2023-12-25T20:14:59.436 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:59.435+0000 7f8939a10700 1 -- start start 2023-12-25T20:14:59.437 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:59.435+0000 7f8939a10700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f89340f5410 0x7f8934109400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:14:59.437 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:59.435+0000 7f8939a10700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8934060240 con 0x7f89340f5410 2023-12-25T20:14:59.438 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:59.436+0000 7f8932ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f89340f5410 0x7f8934109400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:14:59.438 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:59.436+0000 7f8932ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f89340f5410 0x7f8934109400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:45206/0 (socket says 172.21.15.160:45206) 2023-12-25T20:14:59.438 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:59.436+0000 7f8932ffd700 1 -- 172.21.15.160:0/1669619105 learned_addr learned my addr 172.21.15.160:0/1669619105 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:14:59.438 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:59.436+0000 7f8932ffd700 1 -- 172.21.15.160:0/1669619105 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8924005040 con 0x7f89340f5410 2023-12-25T20:14:59.438 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:59.436+0000 7f8932ffd700 1 --2- 172.21.15.160:0/1669619105 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f89340f5410 0x7f8934109400 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7f8924013510 tx=0x7f892400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:14:59.438 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:59.436+0000 7f891b7fe700 1 -- 172.21.15.160:0/1669619105 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8924009320 con 0x7f89340f5410 2023-12-25T20:14:59.439 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:59.436+0000 7f891b7fe700 1 -- 172.21.15.160:0/1669619105 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8924013f00 con 0x7f89340f5410 2023-12-25T20:14:59.439 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:59.436+0000 7f8939a10700 1 -- 172.21.15.160:0/1669619105 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8934060440 con 0x7f89340f5410 2023-12-25T20:14:59.439 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:59.436+0000 7f891b7fe700 1 -- 172.21.15.160:0/1669619105 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8924014d90 con 0x7f89340f5410 2023-12-25T20:14:59.439 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:59.436+0000 7f8939a10700 1 -- 172.21.15.160:0/1669619105 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8934109ab0 con 0x7f89340f5410 2023-12-25T20:14:59.439 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:59.437+0000 7f891b7fe700 1 -- 172.21.15.160:0/1669619105 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f89240095f0 con 0x7f89340f5410 2023-12-25T20:14:59.439 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:59.437+0000 7f8939a10700 1 -- 172.21.15.160:0/1669619105 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8934045ad0 con 0x7f89340f5410 2023-12-25T20:14:59.439 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:59.437+0000 7f891b7fe700 1 --2- 172.21.15.160:0/1669619105 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f891c02ff60 0x7f891c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:14:59.439 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:59.437+0000 7f891b7fe700 1 -- 172.21.15.160:0/1669619105 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f89240410c0 con 0x7f89340f5410 2023-12-25T20:14:59.439 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:59.438+0000 7f89327fc700 1 --2- 172.21.15.160:0/1669619105 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f891c02ff60 0x7f891c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:14:59.439 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:59.438+0000 7f89327fc700 1 --2- 172.21.15.160:0/1669619105 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f891c02ff60 0x7f891c032420 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f89280096a0 tx=0x7f8928006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:14:59.443 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:59.442+0000 7f891b7fe700 1 -- 172.21.15.160:0/1669619105 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8924014420 con 0x7f89340f5410 2023-12-25T20:14:59.649 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:59.648+0000 7f8939a10700 1 -- 172.21.15.160:0/1669619105 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f89340fae60 con 0x7f89340f5410 2023-12-25T20:14:59.649 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:59.648+0000 7f891b7fe700 1 -- 172.21.15.160:0/1669619105 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f892401d7f0 con 0x7f89340f5410 2023-12-25T20:14:59.649 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:14:59.650 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:14:59.650 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:59.650+0000 7f8939a10700 1 -- 172.21.15.160:0/1669619105 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f891c02ff60 msgr2=0x7f891c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:14:59.651 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:59.650+0000 7f8939a10700 1 --2- 172.21.15.160:0/1669619105 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f891c02ff60 0x7f891c032420 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f89280096a0 tx=0x7f8928006b40).stop 2023-12-25T20:14:59.651 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:59.650+0000 7f8939a10700 1 -- 172.21.15.160:0/1669619105 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f89340f5410 msgr2=0x7f8934109400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:14:59.651 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:59.650+0000 7f8939a10700 1 --2- 172.21.15.160:0/1669619105 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f89340f5410 0x7f8934109400 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7f8924013510 tx=0x7f892400b040).stop 2023-12-25T20:14:59.651 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:59.650+0000 7f8939a10700 1 -- 172.21.15.160:0/1669619105 shutdown_connections 2023-12-25T20:14:59.651 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:59.650+0000 7f8939a10700 1 --2- 172.21.15.160:0/1669619105 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f891c02ff60 0x7f891c032420 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:14:59.651 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:59.650+0000 7f8939a10700 1 --2- 172.21.15.160:0/1669619105 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f89340f5410 0x7f8934109400 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:14:59.651 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:59.650+0000 7f8939a10700 1 -- 172.21.15.160:0/1669619105 >> 172.21.15.160:0/1669619105 conn(0x7f89340f0b70 msgr2=0x7f89340f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:14:59.651 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:59.650+0000 7f8939a10700 1 -- 172.21.15.160:0/1669619105 shutdown_connections 2023-12-25T20:14:59.651 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:14:59.650+0000 7f8939a10700 1 -- 172.21.15.160:0/1669619105 wait complete. 2023-12-25T20:14:59.651 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:15:00.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:14:59 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:14:58.697813+0000 mgr.smithi050.nbzhun (mgr.14162) 240 : cluster [DBG] pgmap v226: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:15:00.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:14:59 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:14:59.650313+0000 mon.smithi050 (mon.0) 293 : audit [DBG] from='client.? 172.21.15.160:0/1669619105' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:15:01.029 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:15:01.029 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:15:02.161 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:02.159+0000 7f7e9939f700 1 -- 172.21.15.160:0/1964072287 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7e940f51a0 msgr2=0x7f7e940f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:15:02.161 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:02.159+0000 7f7e9939f700 1 --2- 172.21.15.160:0/1964072287 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7e940f51a0 0x7f7e940f55c0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 rx=0x7f7e84009fb0 tx=0x7f7e84009b30).stop 2023-12-25T20:15:02.161 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:02.159+0000 7f7e9939f700 1 -- 172.21.15.160:0/1964072287 shutdown_connections 2023-12-25T20:15:02.161 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:02.159+0000 7f7e9939f700 1 --2- 172.21.15.160:0/1964072287 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7e940f51a0 0x7f7e940f55c0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:15:02.161 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:02.159+0000 7f7e9939f700 1 -- 172.21.15.160:0/1964072287 >> 172.21.15.160:0/1964072287 conn(0x7f7e940f09a0 msgr2=0x7f7e940f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:15:02.161 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:02.159+0000 7f7e9939f700 1 -- 172.21.15.160:0/1964072287 shutdown_connections 2023-12-25T20:15:02.161 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:02.159+0000 7f7e9939f700 1 -- 172.21.15.160:0/1964072287 wait complete. 2023-12-25T20:15:02.161 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:02.160+0000 7f7e9939f700 1 Processor -- start 2023-12-25T20:15:02.161 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:02.160+0000 7f7e9939f700 1 -- start start 2023-12-25T20:15:02.162 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:02.160+0000 7f7e9939f700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7e940f51a0 0x7f7e941091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:15:02.162 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:02.160+0000 7f7e9939f700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e94060240 con 0x7f7e940f51a0 2023-12-25T20:15:02.162 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:02.160+0000 7f7e93fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7e940f51a0 0x7f7e941091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:15:02.162 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:02.160+0000 7f7e93fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7e940f51a0 0x7f7e941091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:45228/0 (socket says 172.21.15.160:45228) 2023-12-25T20:15:02.162 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:02.160+0000 7f7e93fff700 1 -- 172.21.15.160:0/1453669372 learned_addr learned my addr 172.21.15.160:0/1453669372 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:15:02.163 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:02.161+0000 7f7e93fff700 1 -- 172.21.15.160:0/1453669372 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e84005040 con 0x7f7e940f51a0 2023-12-25T20:15:02.164 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:02.161+0000 7f7e93fff700 1 --2- 172.21.15.160:0/1453669372 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7e940f51a0 0x7f7e941091b0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7f7e84013510 tx=0x7f7e8400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:15:02.164 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:02.161+0000 7f7e90ff9700 1 -- 172.21.15.160:0/1453669372 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7e84009320 con 0x7f7e940f51a0 2023-12-25T20:15:02.164 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:02.161+0000 7f7e9939f700 1 -- 172.21.15.160:0/1453669372 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e94060440 con 0x7f7e940f51a0 2023-12-25T20:15:02.164 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:02.161+0000 7f7e90ff9700 1 -- 172.21.15.160:0/1453669372 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7e84013f00 con 0x7f7e940f51a0 2023-12-25T20:15:02.164 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:02.161+0000 7f7e90ff9700 1 -- 172.21.15.160:0/1453669372 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7e84014dc0 con 0x7f7e940f51a0 2023-12-25T20:15:02.164 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:02.161+0000 7f7e9939f700 1 -- 172.21.15.160:0/1453669372 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e94109860 con 0x7f7e940f51a0 2023-12-25T20:15:02.164 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:02.162+0000 7f7e90ff9700 1 -- 172.21.15.160:0/1453669372 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7e840095f0 con 0x7f7e940f51a0 2023-12-25T20:15:02.164 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:02.162+0000 7f7e90ff9700 1 --2- 172.21.15.160:0/1453669372 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f7e7c02ff60 0x7f7e7c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:15:02.164 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:02.162+0000 7f7e9939f700 1 -- 172.21.15.160:0/1453669372 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e94045ad0 con 0x7f7e940f51a0 2023-12-25T20:15:02.164 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:02.162+0000 7f7e90ff9700 1 -- 172.21.15.160:0/1453669372 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7e840410c0 con 0x7f7e940f51a0 2023-12-25T20:15:02.164 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:02.162+0000 7f7e937fe700 1 --2- 172.21.15.160:0/1453669372 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f7e7c02ff60 0x7f7e7c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:15:02.164 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:02.163+0000 7f7e937fe700 1 --2- 172.21.15.160:0/1453669372 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f7e7c02ff60 0x7f7e7c032420 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f7e880096a0 tx=0x7f7e88006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:15:02.167 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:02.167+0000 7f7e90ff9700 1 -- 172.21.15.160:0/1453669372 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7e8401c790 con 0x7f7e940f51a0 2023-12-25T20:15:02.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:15:01 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:15:00.698207+0000 mgr.smithi050.nbzhun (mgr.14162) 241 : cluster [DBG] pgmap v227: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:15:02.373 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:02.372+0000 7f7e9939f700 1 -- 172.21.15.160:0/1453669372 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7e9410a080 con 0x7f7e940f51a0 2023-12-25T20:15:02.373 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:02.374+0000 7f7e90ff9700 1 -- 172.21.15.160:0/1453669372 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7e8403a020 con 0x7f7e940f51a0 2023-12-25T20:15:02.374 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:15:02.374 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:15:02.375 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:02.374+0000 7f7e9939f700 1 -- 172.21.15.160:0/1453669372 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f7e7c02ff60 msgr2=0x7f7e7c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:15:02.375 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:02.374+0000 7f7e9939f700 1 --2- 172.21.15.160:0/1453669372 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f7e7c02ff60 0x7f7e7c032420 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f7e880096a0 tx=0x7f7e88006b40).stop 2023-12-25T20:15:02.376 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:02.375+0000 7f7e9939f700 1 -- 172.21.15.160:0/1453669372 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7e940f51a0 msgr2=0x7f7e941091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:15:02.376 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:02.375+0000 7f7e9939f700 1 --2- 172.21.15.160:0/1453669372 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7e940f51a0 0x7f7e941091b0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7f7e84013510 tx=0x7f7e8400b040).stop 2023-12-25T20:15:02.376 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:02.375+0000 7f7e9939f700 1 -- 172.21.15.160:0/1453669372 shutdown_connections 2023-12-25T20:15:02.376 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:02.375+0000 7f7e9939f700 1 --2- 172.21.15.160:0/1453669372 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f7e7c02ff60 0x7f7e7c032420 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:15:02.376 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:02.375+0000 7f7e9939f700 1 --2- 172.21.15.160:0/1453669372 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7e940f51a0 0x7f7e941091b0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:15:02.376 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:02.375+0000 7f7e9939f700 1 -- 172.21.15.160:0/1453669372 >> 172.21.15.160:0/1453669372 conn(0x7f7e940f09a0 msgr2=0x7f7e940f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:15:02.376 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:02.375+0000 7f7e9939f700 1 -- 172.21.15.160:0/1453669372 shutdown_connections 2023-12-25T20:15:02.376 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:02.375+0000 7f7e9939f700 1 -- 172.21.15.160:0/1453669372 wait complete. 2023-12-25T20:15:02.376 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:15:03.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:15:02 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:15:02.374733+0000 mon.smithi050 (mon.0) 294 : audit [DBG] from='client.? 172.21.15.160:0/1453669372' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:15:03.763 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:15:03.764 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:15:04.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:15:03 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:15:02.698519+0000 mgr.smithi050.nbzhun (mgr.14162) 242 : cluster [DBG] pgmap v228: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:15:04.963 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:04.962+0000 7fd42a3fb700 1 -- 172.21.15.160:0/3413571573 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd4240f62c0 msgr2=0x7fd4240f66e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:15:04.963 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:04.962+0000 7fd42a3fb700 1 --2- 172.21.15.160:0/3413571573 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd4240f62c0 0x7fd4240f66e0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 rx=0x7fd420009fb0 tx=0x7fd420009b30).stop 2023-12-25T20:15:04.963 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:04.962+0000 7fd42a3fb700 1 -- 172.21.15.160:0/3413571573 shutdown_connections 2023-12-25T20:15:04.963 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:04.962+0000 7fd42a3fb700 1 --2- 172.21.15.160:0/3413571573 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd4240f62c0 0x7fd4240f66e0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:15:04.963 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:04.962+0000 7fd42a3fb700 1 -- 172.21.15.160:0/3413571573 >> 172.21.15.160:0/3413571573 conn(0x7fd4240f1ac0 msgr2=0x7fd4240f3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:15:04.964 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:04.962+0000 7fd42a3fb700 1 -- 172.21.15.160:0/3413571573 shutdown_connections 2023-12-25T20:15:04.964 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:04.962+0000 7fd42a3fb700 1 -- 172.21.15.160:0/3413571573 wait complete. 2023-12-25T20:15:04.965 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:04.962+0000 7fd42a3fb700 1 Processor -- start 2023-12-25T20:15:04.965 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:04.963+0000 7fd42a3fb700 1 -- start start 2023-12-25T20:15:04.965 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:04.963+0000 7fd42a3fb700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd4240f62c0 0x7fd424106520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:15:04.965 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:04.963+0000 7fd42a3fb700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd424106a10 con 0x7fd4240f62c0 2023-12-25T20:15:04.965 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:04.963+0000 7fd4293f9700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd4240f62c0 0x7fd424106520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:15:04.965 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:04.963+0000 7fd4293f9700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd4240f62c0 0x7fd424106520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:45238/0 (socket says 172.21.15.160:45238) 2023-12-25T20:15:04.966 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:04.963+0000 7fd4293f9700 1 -- 172.21.15.160:0/1972230857 learned_addr learned my addr 172.21.15.160:0/1972230857 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:15:04.966 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:04.963+0000 7fd4293f9700 1 -- 172.21.15.160:0/1972230857 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd420005040 con 0x7fd4240f62c0 2023-12-25T20:15:04.966 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:04.963+0000 7fd4293f9700 1 --2- 172.21.15.160:0/1972230857 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd4240f62c0 0x7fd424106520 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7fd420013510 tx=0x7fd42000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:15:04.966 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:04.964+0000 7fd419ffb700 1 -- 172.21.15.160:0/1972230857 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd420009320 con 0x7fd4240f62c0 2023-12-25T20:15:04.966 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:04.964+0000 7fd42a3fb700 1 -- 172.21.15.160:0/1972230857 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd424106c10 con 0x7fd4240f62c0 2023-12-25T20:15:04.966 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:04.964+0000 7fd419ffb700 1 -- 172.21.15.160:0/1972230857 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd420013f00 con 0x7fd4240f62c0 2023-12-25T20:15:04.966 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:04.964+0000 7fd42a3fb700 1 -- 172.21.15.160:0/1972230857 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4241070a0 con 0x7fd4240f62c0 2023-12-25T20:15:04.966 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:04.964+0000 7fd419ffb700 1 -- 172.21.15.160:0/1972230857 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd420014de0 con 0x7fd4240f62c0 2023-12-25T20:15:04.967 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:04.965+0000 7fd419ffb700 1 -- 172.21.15.160:0/1972230857 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd420014420 con 0x7fd4240f62c0 2023-12-25T20:15:04.967 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:04.965+0000 7fd42a3fb700 1 -- 172.21.15.160:0/1972230857 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd424045ad0 con 0x7fd4240f62c0 2023-12-25T20:15:04.967 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:04.965+0000 7fd419ffb700 1 --2- 172.21.15.160:0/1972230857 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fd41002ff60 0x7fd410032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:15:04.967 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:04.965+0000 7fd419ffb700 1 -- 172.21.15.160:0/1972230857 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd4200410a0 con 0x7fd4240f62c0 2023-12-25T20:15:04.967 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:04.965+0000 7fd428bf8700 1 --2- 172.21.15.160:0/1972230857 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fd41002ff60 0x7fd410032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:15:04.967 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:04.966+0000 7fd428bf8700 1 --2- 172.21.15.160:0/1972230857 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fd41002ff60 0x7fd410032420 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7fd4140096a0 tx=0x7fd414006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:15:04.970 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:04.969+0000 7fd419ffb700 1 -- 172.21.15.160:0/1972230857 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd4200095f0 con 0x7fd4240f62c0 2023-12-25T20:15:05.176 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:05.175+0000 7fd42a3fb700 1 -- 172.21.15.160:0/1972230857 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd424107f50 con 0x7fd4240f62c0 2023-12-25T20:15:05.177 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:05.176+0000 7fd419ffb700 1 -- 172.21.15.160:0/1972230857 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd42002b2d0 con 0x7fd4240f62c0 2023-12-25T20:15:05.177 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:15:05.177 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:15:05.179 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:05.178+0000 7fd42a3fb700 1 -- 172.21.15.160:0/1972230857 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fd41002ff60 msgr2=0x7fd410032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:15:05.179 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:05.178+0000 7fd42a3fb700 1 --2- 172.21.15.160:0/1972230857 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fd41002ff60 0x7fd410032420 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7fd4140096a0 tx=0x7fd414006b40).stop 2023-12-25T20:15:05.179 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:05.178+0000 7fd42a3fb700 1 -- 172.21.15.160:0/1972230857 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd4240f62c0 msgr2=0x7fd424106520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:15:05.180 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:05.178+0000 7fd42a3fb700 1 --2- 172.21.15.160:0/1972230857 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd4240f62c0 0x7fd424106520 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7fd420013510 tx=0x7fd42000b040).stop 2023-12-25T20:15:05.180 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:05.178+0000 7fd42a3fb700 1 -- 172.21.15.160:0/1972230857 shutdown_connections 2023-12-25T20:15:05.180 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:05.178+0000 7fd42a3fb700 1 --2- 172.21.15.160:0/1972230857 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fd41002ff60 0x7fd410032420 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:15:05.180 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:05.178+0000 7fd42a3fb700 1 --2- 172.21.15.160:0/1972230857 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd4240f62c0 0x7fd424106520 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:15:05.180 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:05.178+0000 7fd42a3fb700 1 -- 172.21.15.160:0/1972230857 >> 172.21.15.160:0/1972230857 conn(0x7fd4240f1ac0 msgr2=0x7fd4240f2530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:15:05.180 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:05.178+0000 7fd42a3fb700 1 -- 172.21.15.160:0/1972230857 shutdown_connections 2023-12-25T20:15:05.180 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:05.178+0000 7fd42a3fb700 1 -- 172.21.15.160:0/1972230857 wait complete. 2023-12-25T20:15:05.181 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:15:06.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:15:05 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:15:04.698762+0000 mgr.smithi050.nbzhun (mgr.14162) 243 : cluster [DBG] pgmap v229: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:15:06.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:15:05 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:15:05.177775+0000 mon.smithi050 (mon.0) 295 : audit [DBG] from='client.? 172.21.15.160:0/1972230857' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:15:06.561 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:15:06.562 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:15:07.682 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:07.681+0000 7f2dad742700 1 -- 172.21.15.160:0/4102163180 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2da80f5410 msgr2=0x7f2da80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:15:07.682 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:07.681+0000 7f2dad742700 1 --2- 172.21.15.160:0/4102163180 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2da80f5410 0x7f2da80f5830 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 rx=0x7f2d9c009fb0 tx=0x7f2d9c009b30).stop 2023-12-25T20:15:07.682 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:07.681+0000 7f2dad742700 1 -- 172.21.15.160:0/4102163180 shutdown_connections 2023-12-25T20:15:07.682 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:07.681+0000 7f2dad742700 1 --2- 172.21.15.160:0/4102163180 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2da80f5410 0x7f2da80f5830 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:15:07.682 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:07.681+0000 7f2dad742700 1 -- 172.21.15.160:0/4102163180 >> 172.21.15.160:0/4102163180 conn(0x7f2da80f0b70 msgr2=0x7f2da80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:15:07.683 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:07.681+0000 7f2dad742700 1 -- 172.21.15.160:0/4102163180 shutdown_connections 2023-12-25T20:15:07.683 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:07.681+0000 7f2dad742700 1 -- 172.21.15.160:0/4102163180 wait complete. 2023-12-25T20:15:07.684 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:07.682+0000 7f2dad742700 1 Processor -- start 2023-12-25T20:15:07.685 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:07.682+0000 7f2dad742700 1 -- start start 2023-12-25T20:15:07.685 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:07.682+0000 7f2dad742700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2da80f5410 0x7f2da8109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:15:07.685 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:07.682+0000 7f2dad742700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2da8060240 con 0x7f2da80f5410 2023-12-25T20:15:07.685 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:07.682+0000 7f2da6ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2da80f5410 0x7f2da8109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:15:07.685 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:07.682+0000 7f2da6ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2da80f5410 0x7f2da8109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:53580/0 (socket says 172.21.15.160:53580) 2023-12-25T20:15:07.685 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:07.682+0000 7f2da6ffd700 1 -- 172.21.15.160:0/3350893029 learned_addr learned my addr 172.21.15.160:0/3350893029 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:15:07.685 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:07.683+0000 7f2da6ffd700 1 -- 172.21.15.160:0/3350893029 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d9c005040 con 0x7f2da80f5410 2023-12-25T20:15:07.685 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:07.683+0000 7f2da6ffd700 1 --2- 172.21.15.160:0/3350893029 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2da80f5410 0x7f2da8109430 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7f2d9c013510 tx=0x7f2d9c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:15:07.685 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:07.683+0000 7f2d8f7fe700 1 -- 172.21.15.160:0/3350893029 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2d9c009320 con 0x7f2da80f5410 2023-12-25T20:15:07.685 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:07.683+0000 7f2dad742700 1 -- 172.21.15.160:0/3350893029 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2da8060440 con 0x7f2da80f5410 2023-12-25T20:15:07.685 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:07.683+0000 7f2d8f7fe700 1 -- 172.21.15.160:0/3350893029 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2d9c013f00 con 0x7f2da80f5410 2023-12-25T20:15:07.685 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:07.683+0000 7f2d8f7fe700 1 -- 172.21.15.160:0/3350893029 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2d9c014d90 con 0x7f2da80f5410 2023-12-25T20:15:07.685 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:07.683+0000 7f2dad742700 1 -- 172.21.15.160:0/3350893029 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2da8109ae0 con 0x7f2da80f5410 2023-12-25T20:15:07.686 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:07.684+0000 7f2d8f7fe700 1 -- 172.21.15.160:0/3350893029 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2d9c0263a0 con 0x7f2da80f5410 2023-12-25T20:15:07.686 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:07.684+0000 7f2d8f7fe700 1 --2- 172.21.15.160:0/3350893029 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f2d9002ff60 0x7f2d90032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:15:07.686 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:07.684+0000 7f2d8f7fe700 1 -- 172.21.15.160:0/3350893029 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2d9c041010 con 0x7f2da80f5410 2023-12-25T20:15:07.686 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:07.684+0000 7f2dad742700 1 -- 172.21.15.160:0/3350893029 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2da8045ad0 con 0x7f2da80f5410 2023-12-25T20:15:07.686 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:07.684+0000 7f2da67fc700 1 --2- 172.21.15.160:0/3350893029 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f2d9002ff60 0x7f2d90032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:15:07.686 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:07.685+0000 7f2da67fc700 1 --2- 172.21.15.160:0/3350893029 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f2d9002ff60 0x7f2d90032420 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f2d980096a0 tx=0x7f2d98006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:15:07.689 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:07.689+0000 7f2d8f7fe700 1 -- 172.21.15.160:0/3350893029 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2d9c012920 con 0x7f2da80f5410 2023-12-25T20:15:07.898 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:07.897+0000 7f2dad742700 1 -- 172.21.15.160:0/3350893029 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2da810a230 con 0x7f2da80f5410 2023-12-25T20:15:07.900 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:07.900+0000 7f2d8f7fe700 1 -- 172.21.15.160:0/3350893029 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2d9c02b020 con 0x7f2da80f5410 2023-12-25T20:15:07.900 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:15:07.900 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:15:07.901 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:07.901+0000 7f2dad742700 1 -- 172.21.15.160:0/3350893029 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f2d9002ff60 msgr2=0x7f2d90032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:15:07.902 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:07.901+0000 7f2dad742700 1 --2- 172.21.15.160:0/3350893029 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f2d9002ff60 0x7f2d90032420 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f2d980096a0 tx=0x7f2d98006b40).stop 2023-12-25T20:15:07.902 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:07.901+0000 7f2dad742700 1 -- 172.21.15.160:0/3350893029 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2da80f5410 msgr2=0x7f2da8109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:15:07.902 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:07.901+0000 7f2dad742700 1 --2- 172.21.15.160:0/3350893029 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2da80f5410 0x7f2da8109430 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7f2d9c013510 tx=0x7f2d9c00b040).stop 2023-12-25T20:15:07.902 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:07.901+0000 7f2dad742700 1 -- 172.21.15.160:0/3350893029 shutdown_connections 2023-12-25T20:15:07.902 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:07.901+0000 7f2dad742700 1 --2- 172.21.15.160:0/3350893029 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f2d9002ff60 0x7f2d90032420 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:15:07.902 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:07.901+0000 7f2dad742700 1 --2- 172.21.15.160:0/3350893029 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2da80f5410 0x7f2da8109430 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:15:07.902 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:07.901+0000 7f2dad742700 1 -- 172.21.15.160:0/3350893029 >> 172.21.15.160:0/3350893029 conn(0x7f2da80f0b70 msgr2=0x7f2da80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:15:07.902 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:07.901+0000 7f2dad742700 1 -- 172.21.15.160:0/3350893029 shutdown_connections 2023-12-25T20:15:07.902 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:07.901+0000 7f2dad742700 1 -- 172.21.15.160:0/3350893029 wait complete. 2023-12-25T20:15:07.902 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:15:08.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:15:07 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:15:06.699063+0000 mgr.smithi050.nbzhun (mgr.14162) 244 : cluster [DBG] pgmap v230: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:15:09.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:15:08 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:15:07.900972+0000 mon.smithi050 (mon.0) 296 : audit [DBG] from='client.? 172.21.15.160:0/3350893029' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:15:09.329 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:15:09.330 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:15:10.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:15:09 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:15:08.699419+0000 mgr.smithi050.nbzhun (mgr.14162) 245 : cluster [DBG] pgmap v231: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:15:10.520 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:10.519+0000 7fa64df31700 1 -- 172.21.15.160:0/4094534164 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa6480f6510 msgr2=0x7fa6480f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:15:10.520 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:10.519+0000 7fa64df31700 1 --2- 172.21.15.160:0/4094534164 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa6480f6510 0x7fa6480f6930 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 rx=0x7fa63c009fb0 tx=0x7fa63c009b30).stop 2023-12-25T20:15:10.520 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:10.519+0000 7fa64df31700 1 -- 172.21.15.160:0/4094534164 shutdown_connections 2023-12-25T20:15:10.520 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:10.519+0000 7fa64df31700 1 --2- 172.21.15.160:0/4094534164 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa6480f6510 0x7fa6480f6930 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:15:10.520 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:10.519+0000 7fa64df31700 1 -- 172.21.15.160:0/4094534164 >> 172.21.15.160:0/4094534164 conn(0x7fa6480f1c70 msgr2=0x7fa6480f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:15:10.520 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:10.519+0000 7fa64df31700 1 -- 172.21.15.160:0/4094534164 shutdown_connections 2023-12-25T20:15:10.520 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:10.519+0000 7fa64df31700 1 -- 172.21.15.160:0/4094534164 wait complete. 2023-12-25T20:15:10.521 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:10.519+0000 7fa64df31700 1 Processor -- start 2023-12-25T20:15:10.523 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:10.520+0000 7fa64df31700 1 -- start start 2023-12-25T20:15:10.523 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:10.520+0000 7fa64df31700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa6480f6510 0x7fa6481066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:15:10.523 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:10.520+0000 7fa64df31700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa648106bc0 con 0x7fa6480f6510 2023-12-25T20:15:10.523 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:10.520+0000 7fa6477fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa6480f6510 0x7fa6481066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:15:10.523 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:10.520+0000 7fa6477fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa6480f6510 0x7fa6481066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:53590/0 (socket says 172.21.15.160:53590) 2023-12-25T20:15:10.523 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:10.520+0000 7fa6477fe700 1 -- 172.21.15.160:0/3263638328 learned_addr learned my addr 172.21.15.160:0/3263638328 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:15:10.523 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:10.520+0000 7fa6477fe700 1 -- 172.21.15.160:0/3263638328 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa63c005040 con 0x7fa6480f6510 2023-12-25T20:15:10.523 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:10.520+0000 7fa6477fe700 1 --2- 172.21.15.160:0/3263638328 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa6480f6510 0x7fa6481066d0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7fa63c013510 tx=0x7fa63c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:15:10.523 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:10.521+0000 7fa62ffff700 1 -- 172.21.15.160:0/3263638328 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa63c009320 con 0x7fa6480f6510 2023-12-25T20:15:10.523 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:10.521+0000 7fa64df31700 1 -- 172.21.15.160:0/3263638328 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa648106dc0 con 0x7fa6480f6510 2023-12-25T20:15:10.523 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:10.521+0000 7fa62ffff700 1 -- 172.21.15.160:0/3263638328 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa63c013f00 con 0x7fa6480f6510 2023-12-25T20:15:10.524 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:10.521+0000 7fa62ffff700 1 -- 172.21.15.160:0/3263638328 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa63c014d90 con 0x7fa6480f6510 2023-12-25T20:15:10.524 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:10.521+0000 7fa64df31700 1 -- 172.21.15.160:0/3263638328 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa648107250 con 0x7fa6480f6510 2023-12-25T20:15:10.524 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:10.521+0000 7fa62ffff700 1 -- 172.21.15.160:0/3263638328 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa63c0095f0 con 0x7fa6480f6510 2023-12-25T20:15:10.524 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:10.521+0000 7fa64df31700 1 -- 172.21.15.160:0/3263638328 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa648045ad0 con 0x7fa6480f6510 2023-12-25T20:15:10.524 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:10.522+0000 7fa62ffff700 1 --2- 172.21.15.160:0/3263638328 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fa63002ffb0 0x7fa630032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:15:10.524 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:10.522+0000 7fa62ffff700 1 -- 172.21.15.160:0/3263638328 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa63c0410a0 con 0x7fa6480f6510 2023-12-25T20:15:10.524 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:10.522+0000 7fa646ffd700 1 --2- 172.21.15.160:0/3263638328 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fa63002ffb0 0x7fa630032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:15:10.524 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:10.522+0000 7fa646ffd700 1 --2- 172.21.15.160:0/3263638328 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fa63002ffb0 0x7fa630032470 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7fa6380096a0 tx=0x7fa638006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:15:10.527 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:10.526+0000 7fa62ffff700 1 -- 172.21.15.160:0/3263638328 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa63c06d2b0 con 0x7fa6480f6510 2023-12-25T20:15:10.732 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:10.731+0000 7fa64df31700 1 -- 172.21.15.160:0/3263638328 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa648108100 con 0x7fa6480f6510 2023-12-25T20:15:10.733 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:10.733+0000 7fa62ffff700 1 -- 172.21.15.160:0/3263638328 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa63c014420 con 0x7fa6480f6510 2023-12-25T20:15:10.733 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:15:10.733 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:15:10.735 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:10.734+0000 7fa64df31700 1 -- 172.21.15.160:0/3263638328 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fa63002ffb0 msgr2=0x7fa630032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:15:10.735 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:10.734+0000 7fa64df31700 1 --2- 172.21.15.160:0/3263638328 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fa63002ffb0 0x7fa630032470 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7fa6380096a0 tx=0x7fa638006b40).stop 2023-12-25T20:15:10.735 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:10.734+0000 7fa64df31700 1 -- 172.21.15.160:0/3263638328 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa6480f6510 msgr2=0x7fa6481066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:15:10.735 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:10.734+0000 7fa64df31700 1 --2- 172.21.15.160:0/3263638328 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa6480f6510 0x7fa6481066d0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7fa63c013510 tx=0x7fa63c00b040).stop 2023-12-25T20:15:10.735 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:10.734+0000 7fa64df31700 1 -- 172.21.15.160:0/3263638328 shutdown_connections 2023-12-25T20:15:10.735 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:10.734+0000 7fa64df31700 1 --2- 172.21.15.160:0/3263638328 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fa63002ffb0 0x7fa630032470 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:15:10.735 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:10.734+0000 7fa64df31700 1 --2- 172.21.15.160:0/3263638328 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa6480f6510 0x7fa6481066d0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:15:10.735 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:10.734+0000 7fa64df31700 1 -- 172.21.15.160:0/3263638328 >> 172.21.15.160:0/3263638328 conn(0x7fa6480f1c70 msgr2=0x7fa6480f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:15:10.735 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:10.734+0000 7fa64df31700 1 -- 172.21.15.160:0/3263638328 shutdown_connections 2023-12-25T20:15:10.736 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:10.734+0000 7fa64df31700 1 -- 172.21.15.160:0/3263638328 wait complete. 2023-12-25T20:15:10.736 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:15:11.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:15:10 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:15:10.733849+0000 mon.smithi050 (mon.0) 297 : audit [DBG] from='client.? 172.21.15.160:0/3263638328' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:15:12.128 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:15:12.129 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:15:12.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:15:11 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:15:10.699669+0000 mgr.smithi050.nbzhun (mgr.14162) 246 : cluster [DBG] pgmap v232: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:15:13.329 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:13.327+0000 7f127994e700 1 -- 172.21.15.160:0/3043774580 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f12740f5380 msgr2=0x7f12740f57a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:15:13.329 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:13.327+0000 7f127994e700 1 --2- 172.21.15.160:0/3043774580 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f12740f5380 0x7f12740f57a0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 rx=0x7f1264009fb0 tx=0x7f1264009b30).stop 2023-12-25T20:15:13.329 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:13.327+0000 7f127994e700 1 -- 172.21.15.160:0/3043774580 shutdown_connections 2023-12-25T20:15:13.329 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:13.327+0000 7f127994e700 1 --2- 172.21.15.160:0/3043774580 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f12740f5380 0x7f12740f57a0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:15:13.330 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:13.327+0000 7f127994e700 1 -- 172.21.15.160:0/3043774580 >> 172.21.15.160:0/3043774580 conn(0x7f12740f0b00 msgr2=0x7f12740f2f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:15:13.330 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:13.327+0000 7f127994e700 1 -- 172.21.15.160:0/3043774580 shutdown_connections 2023-12-25T20:15:13.330 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:13.328+0000 7f127994e700 1 -- 172.21.15.160:0/3043774580 wait complete. 2023-12-25T20:15:13.330 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:13.328+0000 7f127994e700 1 Processor -- start 2023-12-25T20:15:13.330 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:13.328+0000 7f127994e700 1 -- start start 2023-12-25T20:15:13.330 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:13.328+0000 7f127994e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f12740f5380 0x7f12741066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:15:13.330 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:13.328+0000 7f127994e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1274106bd0 con 0x7f12740f5380 2023-12-25T20:15:13.330 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:13.328+0000 7f1272ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f12740f5380 0x7f12741066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:15:13.330 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:13.329+0000 7f1272ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f12740f5380 0x7f12741066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:53620/0 (socket says 172.21.15.160:53620) 2023-12-25T20:15:13.330 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:13.329+0000 7f1272ffd700 1 -- 172.21.15.160:0/1929070382 learned_addr learned my addr 172.21.15.160:0/1929070382 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:15:13.331 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:13.329+0000 7f1272ffd700 1 -- 172.21.15.160:0/1929070382 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1264005040 con 0x7f12740f5380 2023-12-25T20:15:13.332 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:13.329+0000 7f1272ffd700 1 --2- 172.21.15.160:0/1929070382 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f12740f5380 0x7f12741066e0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7f1264013510 tx=0x7f126400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:15:13.332 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:13.329+0000 7f126b7fe700 1 -- 172.21.15.160:0/1929070382 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1264009320 con 0x7f12740f5380 2023-12-25T20:15:13.332 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:13.329+0000 7f126b7fe700 1 -- 172.21.15.160:0/1929070382 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1264013f00 con 0x7f12740f5380 2023-12-25T20:15:13.332 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:13.329+0000 7f127994e700 1 -- 172.21.15.160:0/1929070382 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1274106dd0 con 0x7f12740f5380 2023-12-25T20:15:13.332 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:13.329+0000 7f126b7fe700 1 -- 172.21.15.160:0/1929070382 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1264014d90 con 0x7f12740f5380 2023-12-25T20:15:13.332 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:13.329+0000 7f127994e700 1 -- 172.21.15.160:0/1929070382 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1274107260 con 0x7f12740f5380 2023-12-25T20:15:13.332 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:13.330+0000 7f126b7fe700 1 -- 172.21.15.160:0/1929070382 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f12640095f0 con 0x7f12740f5380 2023-12-25T20:15:13.332 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:13.330+0000 7f126b7fe700 1 --2- 172.21.15.160:0/1929070382 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f126002ff60 0x7f1260032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:15:13.332 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:13.330+0000 7f127994e700 1 -- 172.21.15.160:0/1929070382 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1274045ad0 con 0x7f12740f5380 2023-12-25T20:15:13.332 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:13.330+0000 7f126b7fe700 1 -- 172.21.15.160:0/1929070382 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1264041080 con 0x7f12740f5380 2023-12-25T20:15:13.332 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:13.331+0000 7f12727fc700 1 --2- 172.21.15.160:0/1929070382 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f126002ff60 0x7f1260032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:15:13.332 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:13.331+0000 7f12727fc700 1 --2- 172.21.15.160:0/1929070382 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f126002ff60 0x7f1260032420 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f125c0096a0 tx=0x7f125c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:15:13.336 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:13.335+0000 7f126b7fe700 1 -- 172.21.15.160:0/1929070382 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1264014420 con 0x7f12740f5380 2023-12-25T20:15:13.541 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:13.540+0000 7f127994e700 1 -- 172.21.15.160:0/1929070382 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f12741080b0 con 0x7f12740f5380 2023-12-25T20:15:13.542 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:13.541+0000 7f126b7fe700 1 -- 172.21.15.160:0/1929070382 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1264010230 con 0x7f12740f5380 2023-12-25T20:15:13.542 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:15:13.542 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:15:13.543 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:13.543+0000 7f127994e700 1 -- 172.21.15.160:0/1929070382 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f126002ff60 msgr2=0x7f1260032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:15:13.543 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:13.543+0000 7f127994e700 1 --2- 172.21.15.160:0/1929070382 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f126002ff60 0x7f1260032420 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f125c0096a0 tx=0x7f125c006b40).stop 2023-12-25T20:15:13.544 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:13.543+0000 7f127994e700 1 -- 172.21.15.160:0/1929070382 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f12740f5380 msgr2=0x7f12741066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:15:13.544 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:13.543+0000 7f127994e700 1 --2- 172.21.15.160:0/1929070382 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f12740f5380 0x7f12741066e0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7f1264013510 tx=0x7f126400b040).stop 2023-12-25T20:15:13.544 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:13.543+0000 7f127994e700 1 -- 172.21.15.160:0/1929070382 shutdown_connections 2023-12-25T20:15:13.544 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:13.543+0000 7f127994e700 1 --2- 172.21.15.160:0/1929070382 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f126002ff60 0x7f1260032420 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:15:13.544 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:13.543+0000 7f127994e700 1 --2- 172.21.15.160:0/1929070382 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f12740f5380 0x7f12741066e0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:15:13.544 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:13.543+0000 7f127994e700 1 -- 172.21.15.160:0/1929070382 >> 172.21.15.160:0/1929070382 conn(0x7f12740f0b00 msgr2=0x7f127405fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:15:13.544 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:13.543+0000 7f127994e700 1 -- 172.21.15.160:0/1929070382 shutdown_connections 2023-12-25T20:15:13.544 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:13.543+0000 7f127994e700 1 -- 172.21.15.160:0/1929070382 wait complete. 2023-12-25T20:15:13.545 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:15:14.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:15:13 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:15:12.699961+0000 mgr.smithi050.nbzhun (mgr.14162) 247 : cluster [DBG] pgmap v233: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:15:14.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:15:13 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:15:13.542875+0000 mon.smithi050 (mon.0) 298 : audit [DBG] from='client.? 172.21.15.160:0/1929070382' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:15:14.921 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:15:14.921 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:15:15.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:15:14 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:15:14.770713+0000 mon.smithi050 (mon.0) 299 : audit [INF] from='mgr.14162 172.21.15.50:0/3178367813' entity='mgr.smithi050.nbzhun' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi050.nbzhun/mirror_snapshot_schedule"}]: dispatch 2023-12-25T20:15:15.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:15:14 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:15:14.770917+0000 mon.smithi050 (mon.0) 300 : audit [INF] from='mgr.14162 172.21.15.50:0/3178367813' entity='mgr.smithi050.nbzhun' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi050.nbzhun/trash_purge_schedule"}]: dispatch 2023-12-25T20:15:16.020 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:16.016+0000 7f5dee252700 1 -- 172.21.15.160:0/1343196932 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5de80f5410 msgr2=0x7f5de80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:15:16.020 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:16.016+0000 7f5dee252700 1 --2- 172.21.15.160:0/1343196932 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5de80f5410 0x7f5de80f5830 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 rx=0x7f5ddc009fb0 tx=0x7f5ddc009b30).stop 2023-12-25T20:15:16.020 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:16.016+0000 7f5dee252700 1 -- 172.21.15.160:0/1343196932 shutdown_connections 2023-12-25T20:15:16.020 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:16.016+0000 7f5dee252700 1 --2- 172.21.15.160:0/1343196932 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5de80f5410 0x7f5de80f5830 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:15:16.020 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:16.016+0000 7f5dee252700 1 -- 172.21.15.160:0/1343196932 >> 172.21.15.160:0/1343196932 conn(0x7f5de80f0b70 msgr2=0x7f5de80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:15:16.020 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:16.016+0000 7f5dee252700 1 -- 172.21.15.160:0/1343196932 shutdown_connections 2023-12-25T20:15:16.020 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:16.016+0000 7f5dee252700 1 -- 172.21.15.160:0/1343196932 wait complete. 2023-12-25T20:15:16.020 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:16.017+0000 7f5dee252700 1 Processor -- start 2023-12-25T20:15:16.020 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:16.017+0000 7f5dee252700 1 -- start start 2023-12-25T20:15:16.020 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:16.017+0000 7f5dee252700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5de80f5410 0x7f5de8108910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:15:16.021 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:16.017+0000 7f5dee252700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5de8108e00 con 0x7f5de80f5410 2023-12-25T20:15:16.021 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:16.017+0000 7f5de77fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5de80f5410 0x7f5de8108910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:15:16.021 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:16.017+0000 7f5de77fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5de80f5410 0x7f5de8108910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:53640/0 (socket says 172.21.15.160:53640) 2023-12-25T20:15:16.021 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:16.017+0000 7f5de77fe700 1 -- 172.21.15.160:0/933014963 learned_addr learned my addr 172.21.15.160:0/933014963 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:15:16.021 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:16.018+0000 7f5de77fe700 1 -- 172.21.15.160:0/933014963 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ddc005040 con 0x7f5de80f5410 2023-12-25T20:15:16.021 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:16.018+0000 7f5de77fe700 1 --2- 172.21.15.160:0/933014963 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5de80f5410 0x7f5de8108910 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7f5ddc013510 tx=0x7f5ddc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:15:16.021 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:16.018+0000 7f5dcffff700 1 -- 172.21.15.160:0/933014963 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5ddc009320 con 0x7f5de80f5410 2023-12-25T20:15:16.021 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:16.018+0000 7f5dee252700 1 -- 172.21.15.160:0/933014963 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5de8109000 con 0x7f5de80f5410 2023-12-25T20:15:16.021 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:16.018+0000 7f5dcffff700 1 -- 172.21.15.160:0/933014963 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5ddc013f00 con 0x7f5de80f5410 2023-12-25T20:15:16.021 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:16.018+0000 7f5dcffff700 1 -- 172.21.15.160:0/933014963 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5ddc014d90 con 0x7f5de80f5410 2023-12-25T20:15:16.021 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:16.018+0000 7f5dee252700 1 -- 172.21.15.160:0/933014963 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5de8109490 con 0x7f5de80f5410 2023-12-25T20:15:16.021 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:16.019+0000 7f5dcffff700 1 -- 172.21.15.160:0/933014963 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5ddc014420 con 0x7f5de80f5410 2023-12-25T20:15:16.021 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:16.019+0000 7f5dee252700 1 -- 172.21.15.160:0/933014963 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5de8045ad0 con 0x7f5de80f5410 2023-12-25T20:15:16.021 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:16.019+0000 7f5dcffff700 1 --2- 172.21.15.160:0/933014963 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f5dd002ff60 0x7f5dd0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:15:16.022 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:16.019+0000 7f5dcffff700 1 -- 172.21.15.160:0/933014963 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5ddc0410c0 con 0x7f5de80f5410 2023-12-25T20:15:16.022 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:16.019+0000 7f5de6ffd700 1 --2- 172.21.15.160:0/933014963 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f5dd002ff60 0x7f5dd0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:15:16.022 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:16.019+0000 7f5de6ffd700 1 --2- 172.21.15.160:0/933014963 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f5dd002ff60 0x7f5dd0032420 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f5dd80096a0 tx=0x7f5dd8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:15:16.024 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:16.024+0000 7f5dcffff700 1 -- 172.21.15.160:0/933014963 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5ddc0095f0 con 0x7f5de80f5410 2023-12-25T20:15:16.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:15:15 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:15:14.700250+0000 mgr.smithi050.nbzhun (mgr.14162) 248 : cluster [DBG] pgmap v234: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:15:16.229 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:16.229+0000 7f5dee252700 1 -- 172.21.15.160:0/933014963 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5de80fae60 con 0x7f5de80f5410 2023-12-25T20:15:16.233 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:16.230+0000 7f5dcffff700 1 -- 172.21.15.160:0/933014963 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5ddc01db70 con 0x7f5de80f5410 2023-12-25T20:15:16.234 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:15:16.234 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:15:16.234 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:16.231+0000 7f5dee252700 1 -- 172.21.15.160:0/933014963 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f5dd002ff60 msgr2=0x7f5dd0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:15:16.234 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:16.231+0000 7f5dee252700 1 --2- 172.21.15.160:0/933014963 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f5dd002ff60 0x7f5dd0032420 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f5dd80096a0 tx=0x7f5dd8006b40).stop 2023-12-25T20:15:16.235 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:16.231+0000 7f5dee252700 1 -- 172.21.15.160:0/933014963 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5de80f5410 msgr2=0x7f5de8108910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:15:16.235 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:16.231+0000 7f5dee252700 1 --2- 172.21.15.160:0/933014963 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5de80f5410 0x7f5de8108910 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7f5ddc013510 tx=0x7f5ddc00b040).stop 2023-12-25T20:15:16.235 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:16.231+0000 7f5dee252700 1 -- 172.21.15.160:0/933014963 shutdown_connections 2023-12-25T20:15:16.235 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:16.231+0000 7f5dee252700 1 --2- 172.21.15.160:0/933014963 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f5dd002ff60 0x7f5dd0032420 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:15:16.235 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:16.231+0000 7f5dee252700 1 --2- 172.21.15.160:0/933014963 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5de80f5410 0x7f5de8108910 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:15:16.235 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:16.231+0000 7f5dee252700 1 -- 172.21.15.160:0/933014963 >> 172.21.15.160:0/933014963 conn(0x7f5de80f0b70 msgr2=0x7f5de805fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:15:16.235 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:16.232+0000 7f5dee252700 1 -- 172.21.15.160:0/933014963 shutdown_connections 2023-12-25T20:15:16.235 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:16.232+0000 7f5dee252700 1 -- 172.21.15.160:0/933014963 wait complete. 2023-12-25T20:15:16.235 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:15:17.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:15:16 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:15:16.231359+0000 mon.smithi050 (mon.0) 301 : audit [DBG] from='client.? 172.21.15.160:0/933014963' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:15:17.620 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:15:17.621 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:15:18.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:15:17 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:15:16.700512+0000 mgr.smithi050.nbzhun (mgr.14162) 249 : cluster [DBG] pgmap v235: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:15:18.777 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:18.773+0000 7f03c7081700 1 -- 172.21.15.160:0/147383757 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f03c00f51a0 msgr2=0x7f03c00f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:15:18.777 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:18.773+0000 7f03c7081700 1 --2- 172.21.15.160:0/147383757 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f03c00f51a0 0x7f03c00f55c0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 rx=0x7f03bc009fb0 tx=0x7f03bc009b30).stop 2023-12-25T20:15:18.777 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:18.773+0000 7f03c7081700 1 -- 172.21.15.160:0/147383757 shutdown_connections 2023-12-25T20:15:18.777 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:18.773+0000 7f03c7081700 1 --2- 172.21.15.160:0/147383757 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f03c00f51a0 0x7f03c00f55c0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:15:18.777 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:18.773+0000 7f03c7081700 1 -- 172.21.15.160:0/147383757 >> 172.21.15.160:0/147383757 conn(0x7f03c00f09a0 msgr2=0x7f03c00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:15:18.778 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:18.773+0000 7f03c7081700 1 -- 172.21.15.160:0/147383757 shutdown_connections 2023-12-25T20:15:18.778 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:18.773+0000 7f03c7081700 1 -- 172.21.15.160:0/147383757 wait complete. 2023-12-25T20:15:18.778 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:18.774+0000 7f03c7081700 1 Processor -- start 2023-12-25T20:15:18.778 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:18.774+0000 7f03c7081700 1 -- start start 2023-12-25T20:15:18.778 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:18.774+0000 7f03c7081700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f03c00f51a0 0x7f03c01053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:15:18.778 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:18.774+0000 7f03c7081700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03c01058a0 con 0x7f03c00f51a0 2023-12-25T20:15:18.779 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:18.774+0000 7f03c607f700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f03c00f51a0 0x7f03c01053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:15:18.779 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:18.774+0000 7f03c607f700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f03c00f51a0 0x7f03c01053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:38822/0 (socket says 172.21.15.160:38822) 2023-12-25T20:15:18.779 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:18.774+0000 7f03c607f700 1 -- 172.21.15.160:0/2127688116 learned_addr learned my addr 172.21.15.160:0/2127688116 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:15:18.779 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:18.774+0000 7f03c607f700 1 -- 172.21.15.160:0/2127688116 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03bc005040 con 0x7f03c00f51a0 2023-12-25T20:15:18.779 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:18.774+0000 7f03c607f700 1 --2- 172.21.15.160:0/2127688116 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f03c00f51a0 0x7f03c01053b0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7f03bc013510 tx=0x7f03bc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:15:18.779 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:18.775+0000 7f03b6ffd700 1 -- 172.21.15.160:0/2127688116 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f03bc009320 con 0x7f03c00f51a0 2023-12-25T20:15:18.779 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:18.775+0000 7f03c7081700 1 -- 172.21.15.160:0/2127688116 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03c0105aa0 con 0x7f03c00f51a0 2023-12-25T20:15:18.780 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:18.775+0000 7f03b6ffd700 1 -- 172.21.15.160:0/2127688116 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f03bc013f00 con 0x7f03c00f51a0 2023-12-25T20:15:18.780 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:18.775+0000 7f03b6ffd700 1 -- 172.21.15.160:0/2127688116 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f03bc014dc0 con 0x7f03c00f51a0 2023-12-25T20:15:18.780 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:18.775+0000 7f03c7081700 1 -- 172.21.15.160:0/2127688116 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03c0105f30 con 0x7f03c00f51a0 2023-12-25T20:15:18.780 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:18.775+0000 7f03b6ffd700 1 -- 172.21.15.160:0/2127688116 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f03bc01d3d0 con 0x7f03c00f51a0 2023-12-25T20:15:18.780 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:18.776+0000 7f03b6ffd700 1 --2- 172.21.15.160:0/2127688116 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f03ac032170 0x7f03ac034630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:15:18.780 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:18.776+0000 7f03b6ffd700 1 -- 172.21.15.160:0/2127688116 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f03bc041110 con 0x7f03c00f51a0 2023-12-25T20:15:18.780 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:18.776+0000 7f03c7081700 1 -- 172.21.15.160:0/2127688116 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03c0045ad0 con 0x7f03c00f51a0 2023-12-25T20:15:18.780 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:18.776+0000 7f03c587e700 1 --2- 172.21.15.160:0/2127688116 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f03ac032170 0x7f03ac034630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:15:18.780 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:18.776+0000 7f03c587e700 1 --2- 172.21.15.160:0/2127688116 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f03ac032170 0x7f03ac034630 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f03b00096a0 tx=0x7f03b0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:15:18.781 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:18.781+0000 7f03b6ffd700 1 -- 172.21.15.160:0/2127688116 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f03bc012920 con 0x7f03c00f51a0 2023-12-25T20:15:18.989 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:18.988+0000 7f03c7081700 1 -- 172.21.15.160:0/2127688116 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f03c0106c50 con 0x7f03c00f51a0 2023-12-25T20:15:18.989 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:18.990+0000 7f03b6ffd700 1 -- 172.21.15.160:0/2127688116 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f03bc02b020 con 0x7f03c00f51a0 2023-12-25T20:15:18.990 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:15:18.990 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:15:18.991 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:18.990+0000 7f03c7081700 1 -- 172.21.15.160:0/2127688116 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f03ac032170 msgr2=0x7f03ac034630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:15:18.991 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:18.990+0000 7f03c7081700 1 --2- 172.21.15.160:0/2127688116 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f03ac032170 0x7f03ac034630 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f03b00096a0 tx=0x7f03b0006b40).stop 2023-12-25T20:15:18.991 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:18.990+0000 7f03c7081700 1 -- 172.21.15.160:0/2127688116 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f03c00f51a0 msgr2=0x7f03c01053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:15:18.991 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:18.990+0000 7f03c7081700 1 --2- 172.21.15.160:0/2127688116 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f03c00f51a0 0x7f03c01053b0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7f03bc013510 tx=0x7f03bc00b040).stop 2023-12-25T20:15:18.991 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:18.990+0000 7f03c7081700 1 -- 172.21.15.160:0/2127688116 shutdown_connections 2023-12-25T20:15:18.991 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:18.990+0000 7f03c7081700 1 --2- 172.21.15.160:0/2127688116 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f03ac032170 0x7f03ac034630 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:15:18.991 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:18.990+0000 7f03c7081700 1 --2- 172.21.15.160:0/2127688116 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f03c00f51a0 0x7f03c01053b0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:15:18.992 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:18.990+0000 7f03c7081700 1 -- 172.21.15.160:0/2127688116 >> 172.21.15.160:0/2127688116 conn(0x7f03c00f09a0 msgr2=0x7f03c00f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:15:18.992 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:18.990+0000 7f03c7081700 1 -- 172.21.15.160:0/2127688116 shutdown_connections 2023-12-25T20:15:18.992 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:18.990+0000 7f03c7081700 1 -- 172.21.15.160:0/2127688116 wait complete. 2023-12-25T20:15:18.992 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:15:20.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:15:19 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:15:18.700832+0000 mgr.smithi050.nbzhun (mgr.14162) 250 : cluster [DBG] pgmap v236: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:15:20.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:15:19 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:15:18.990653+0000 mon.smithi050 (mon.0) 302 : audit [DBG] from='client.? 172.21.15.160:0/2127688116' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:15:20.428 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:15:20.428 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:15:21.561 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:21.560+0000 7f2fa109c700 1 -- 172.21.15.160:0/315095876 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2f9c0f51a0 msgr2=0x7f2f9c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:15:21.561 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:21.560+0000 7f2fa109c700 1 --2- 172.21.15.160:0/315095876 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2f9c0f51a0 0x7f2f9c0f55c0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 rx=0x7f2f8c009fb0 tx=0x7f2f8c009b30).stop 2023-12-25T20:15:21.561 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:21.560+0000 7f2fa109c700 1 -- 172.21.15.160:0/315095876 shutdown_connections 2023-12-25T20:15:21.562 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:21.560+0000 7f2fa109c700 1 --2- 172.21.15.160:0/315095876 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2f9c0f51a0 0x7f2f9c0f55c0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:15:21.562 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:21.560+0000 7f2fa109c700 1 -- 172.21.15.160:0/315095876 >> 172.21.15.160:0/315095876 conn(0x7f2f9c0f09a0 msgr2=0x7f2f9c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:15:21.562 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:21.560+0000 7f2fa109c700 1 -- 172.21.15.160:0/315095876 shutdown_connections 2023-12-25T20:15:21.562 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:21.560+0000 7f2fa109c700 1 -- 172.21.15.160:0/315095876 wait complete. 2023-12-25T20:15:21.562 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:21.560+0000 7f2fa109c700 1 Processor -- start 2023-12-25T20:15:21.562 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:21.561+0000 7f2fa109c700 1 -- start start 2023-12-25T20:15:21.564 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:21.561+0000 7f2fa109c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2f9c0f51a0 0x7f2f9c1091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:15:21.564 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:21.561+0000 7f2fa109c700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f9c060240 con 0x7f2f9c0f51a0 2023-12-25T20:15:21.564 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:21.561+0000 7f2f9bfff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2f9c0f51a0 0x7f2f9c1091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:15:21.564 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:21.561+0000 7f2f9bfff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2f9c0f51a0 0x7f2f9c1091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:38838/0 (socket says 172.21.15.160:38838) 2023-12-25T20:15:21.564 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:21.561+0000 7f2f9bfff700 1 -- 172.21.15.160:0/181327195 learned_addr learned my addr 172.21.15.160:0/181327195 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:15:21.565 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:21.562+0000 7f2f9bfff700 1 -- 172.21.15.160:0/181327195 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f8c005040 con 0x7f2f9c0f51a0 2023-12-25T20:15:21.565 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:21.562+0000 7f2f9bfff700 1 --2- 172.21.15.160:0/181327195 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2f9c0f51a0 0x7f2f9c1091b0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7f2f8c013510 tx=0x7f2f8c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:15:21.565 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:21.562+0000 7f2f98ff9700 1 -- 172.21.15.160:0/181327195 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2f8c009320 con 0x7f2f9c0f51a0 2023-12-25T20:15:21.565 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:21.562+0000 7f2fa109c700 1 -- 172.21.15.160:0/181327195 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f9c060440 con 0x7f2f9c0f51a0 2023-12-25T20:15:21.565 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:21.562+0000 7f2f98ff9700 1 -- 172.21.15.160:0/181327195 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2f8c013f00 con 0x7f2f9c0f51a0 2023-12-25T20:15:21.565 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:21.562+0000 7f2f98ff9700 1 -- 172.21.15.160:0/181327195 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2f8c014dc0 con 0x7f2f9c0f51a0 2023-12-25T20:15:21.565 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:21.562+0000 7f2fa109c700 1 -- 172.21.15.160:0/181327195 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f9c109860 con 0x7f2f9c0f51a0 2023-12-25T20:15:21.565 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:21.563+0000 7f2f98ff9700 1 -- 172.21.15.160:0/181327195 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2f8c0095f0 con 0x7f2f9c0f51a0 2023-12-25T20:15:21.565 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:21.563+0000 7f2fa109c700 1 -- 172.21.15.160:0/181327195 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f9c045ad0 con 0x7f2f9c0f51a0 2023-12-25T20:15:21.565 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:21.563+0000 7f2f98ff9700 1 --2- 172.21.15.160:0/181327195 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f2f8402ff60 0x7f2f84032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:15:21.565 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:21.563+0000 7f2f98ff9700 1 -- 172.21.15.160:0/181327195 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2f8c041100 con 0x7f2f9c0f51a0 2023-12-25T20:15:21.565 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:21.563+0000 7f2f9b7fe700 1 --2- 172.21.15.160:0/181327195 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f2f8402ff60 0x7f2f84032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:15:21.566 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:21.563+0000 7f2f9b7fe700 1 --2- 172.21.15.160:0/181327195 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f2f8402ff60 0x7f2f84032420 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f2f900096a0 tx=0x7f2f90006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:15:21.568 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:21.568+0000 7f2f98ff9700 1 -- 172.21.15.160:0/181327195 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2f8c014420 con 0x7f2f9c0f51a0 2023-12-25T20:15:21.774 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:21.773+0000 7f2fa109c700 1 -- 172.21.15.160:0/181327195 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2f9c10a080 con 0x7f2f9c0f51a0 2023-12-25T20:15:21.774 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:21.774+0000 7f2f98ff9700 1 -- 172.21.15.160:0/181327195 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2f8c010230 con 0x7f2f9c0f51a0 2023-12-25T20:15:21.775 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:15:21.775 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:15:21.776 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:21.775+0000 7f2fa109c700 1 -- 172.21.15.160:0/181327195 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f2f8402ff60 msgr2=0x7f2f84032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:15:21.776 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:21.775+0000 7f2fa109c700 1 --2- 172.21.15.160:0/181327195 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f2f8402ff60 0x7f2f84032420 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f2f900096a0 tx=0x7f2f90006b40).stop 2023-12-25T20:15:21.776 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:21.775+0000 7f2fa109c700 1 -- 172.21.15.160:0/181327195 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2f9c0f51a0 msgr2=0x7f2f9c1091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:15:21.776 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:21.775+0000 7f2fa109c700 1 --2- 172.21.15.160:0/181327195 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2f9c0f51a0 0x7f2f9c1091b0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7f2f8c013510 tx=0x7f2f8c00b040).stop 2023-12-25T20:15:21.776 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:21.775+0000 7f2fa109c700 1 -- 172.21.15.160:0/181327195 shutdown_connections 2023-12-25T20:15:21.777 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:21.775+0000 7f2fa109c700 1 --2- 172.21.15.160:0/181327195 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f2f8402ff60 0x7f2f84032420 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:15:21.777 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:21.775+0000 7f2fa109c700 1 --2- 172.21.15.160:0/181327195 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2f9c0f51a0 0x7f2f9c1091b0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:15:21.777 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:21.775+0000 7f2fa109c700 1 -- 172.21.15.160:0/181327195 >> 172.21.15.160:0/181327195 conn(0x7f2f9c0f09a0 msgr2=0x7f2f9c0f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:15:21.777 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:21.775+0000 7f2fa109c700 1 -- 172.21.15.160:0/181327195 shutdown_connections 2023-12-25T20:15:21.777 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:21.775+0000 7f2fa109c700 1 -- 172.21.15.160:0/181327195 wait complete. 2023-12-25T20:15:21.777 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:15:22.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:15:21 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:15:20.701230+0000 mgr.smithi050.nbzhun (mgr.14162) 251 : cluster [DBG] pgmap v237: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:15:22.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:15:21 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:15:21.775492+0000 mon.smithi050 (mon.0) 303 : audit [DBG] from='client.? 172.21.15.160:0/181327195' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:15:23.152 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:15:23.152 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:15:24.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:15:23 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:15:22.701565+0000 mgr.smithi050.nbzhun (mgr.14162) 252 : cluster [DBG] pgmap v238: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:15:24.314 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:24.313+0000 7fb2f105d700 1 -- 172.21.15.160:0/2925746624 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb2ec0f5410 msgr2=0x7fb2ec0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:15:24.314 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:24.313+0000 7fb2f105d700 1 --2- 172.21.15.160:0/2925746624 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb2ec0f5410 0x7fb2ec0f5830 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 rx=0x7fb2d4009fb0 tx=0x7fb2d4009b30).stop 2023-12-25T20:15:24.314 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:24.313+0000 7fb2f105d700 1 -- 172.21.15.160:0/2925746624 shutdown_connections 2023-12-25T20:15:24.314 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:24.313+0000 7fb2f105d700 1 --2- 172.21.15.160:0/2925746624 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb2ec0f5410 0x7fb2ec0f5830 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:15:24.314 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:24.313+0000 7fb2f105d700 1 -- 172.21.15.160:0/2925746624 >> 172.21.15.160:0/2925746624 conn(0x7fb2ec0f0b70 msgr2=0x7fb2ec0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:15:24.314 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:24.313+0000 7fb2f105d700 1 -- 172.21.15.160:0/2925746624 shutdown_connections 2023-12-25T20:15:24.314 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:24.313+0000 7fb2f105d700 1 -- 172.21.15.160:0/2925746624 wait complete. 2023-12-25T20:15:24.316 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:24.314+0000 7fb2f105d700 1 Processor -- start 2023-12-25T20:15:24.316 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:24.314+0000 7fb2f105d700 1 -- start start 2023-12-25T20:15:24.316 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:24.314+0000 7fb2f105d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb2ec0f5410 0x7fb2ec105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:15:24.316 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:24.314+0000 7fb2f105d700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2ec105b20 con 0x7fb2ec0f5410 2023-12-25T20:15:24.316 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:24.314+0000 7fb2ead9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb2ec0f5410 0x7fb2ec105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:15:24.316 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:24.314+0000 7fb2ead9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb2ec0f5410 0x7fb2ec105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:38858/0 (socket says 172.21.15.160:38858) 2023-12-25T20:15:24.316 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:24.314+0000 7fb2ead9d700 1 -- 172.21.15.160:0/2954982572 learned_addr learned my addr 172.21.15.160:0/2954982572 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:15:24.316 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:24.315+0000 7fb2ead9d700 1 -- 172.21.15.160:0/2954982572 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2d4005040 con 0x7fb2ec0f5410 2023-12-25T20:15:24.316 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:24.315+0000 7fb2ead9d700 1 --2- 172.21.15.160:0/2954982572 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb2ec0f5410 0x7fb2ec105630 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7fb2d4013510 tx=0x7fb2d400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:15:24.316 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:24.315+0000 7fb2e37fe700 1 -- 172.21.15.160:0/2954982572 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb2d4009320 con 0x7fb2ec0f5410 2023-12-25T20:15:24.317 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:24.315+0000 7fb2f105d700 1 -- 172.21.15.160:0/2954982572 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2ec105d20 con 0x7fb2ec0f5410 2023-12-25T20:15:24.317 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:24.315+0000 7fb2e37fe700 1 -- 172.21.15.160:0/2954982572 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb2d4013f00 con 0x7fb2ec0f5410 2023-12-25T20:15:24.317 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:24.315+0000 7fb2e37fe700 1 -- 172.21.15.160:0/2954982572 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb2d4014d90 con 0x7fb2ec0f5410 2023-12-25T20:15:24.317 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:24.315+0000 7fb2f105d700 1 -- 172.21.15.160:0/2954982572 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2ec1061b0 con 0x7fb2ec0f5410 2023-12-25T20:15:24.318 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:24.316+0000 7fb2e37fe700 1 -- 172.21.15.160:0/2954982572 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb2d40095f0 con 0x7fb2ec0f5410 2023-12-25T20:15:24.318 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:24.316+0000 7fb2f105d700 1 -- 172.21.15.160:0/2954982572 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2ec045ad0 con 0x7fb2ec0f5410 2023-12-25T20:15:24.318 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:24.316+0000 7fb2e37fe700 1 --2- 172.21.15.160:0/2954982572 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fb2d802ff60 0x7fb2d8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:15:24.318 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:24.316+0000 7fb2e37fe700 1 -- 172.21.15.160:0/2954982572 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb2d4040fe0 con 0x7fb2ec0f5410 2023-12-25T20:15:24.318 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:24.316+0000 7fb2ea59c700 1 --2- 172.21.15.160:0/2954982572 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fb2d802ff60 0x7fb2d8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:15:24.318 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:24.317+0000 7fb2ea59c700 1 --2- 172.21.15.160:0/2954982572 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fb2d802ff60 0x7fb2d8032420 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7fb2dc0096a0 tx=0x7fb2dc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:15:24.321 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:24.321+0000 7fb2e37fe700 1 -- 172.21.15.160:0/2954982572 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb2d401d7f0 con 0x7fb2ec0f5410 2023-12-25T20:15:24.528 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:24.527+0000 7fb2f105d700 1 -- 172.21.15.160:0/2954982572 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb2ec0fae60 con 0x7fb2ec0f5410 2023-12-25T20:15:24.528 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:24.529+0000 7fb2e37fe700 1 -- 172.21.15.160:0/2954982572 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb2d4010230 con 0x7fb2ec0f5410 2023-12-25T20:15:24.529 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:15:24.529 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:15:24.530 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:24.529+0000 7fb2f105d700 1 -- 172.21.15.160:0/2954982572 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fb2d802ff60 msgr2=0x7fb2d8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:15:24.530 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:24.529+0000 7fb2f105d700 1 --2- 172.21.15.160:0/2954982572 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fb2d802ff60 0x7fb2d8032420 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7fb2dc0096a0 tx=0x7fb2dc006b40).stop 2023-12-25T20:15:24.530 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:24.529+0000 7fb2f105d700 1 -- 172.21.15.160:0/2954982572 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb2ec0f5410 msgr2=0x7fb2ec105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:15:24.530 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:24.529+0000 7fb2f105d700 1 --2- 172.21.15.160:0/2954982572 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb2ec0f5410 0x7fb2ec105630 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7fb2d4013510 tx=0x7fb2d400b040).stop 2023-12-25T20:15:24.531 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:24.530+0000 7fb2f105d700 1 -- 172.21.15.160:0/2954982572 shutdown_connections 2023-12-25T20:15:24.531 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:24.530+0000 7fb2f105d700 1 --2- 172.21.15.160:0/2954982572 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fb2d802ff60 0x7fb2d8032420 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:15:24.531 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:24.530+0000 7fb2f105d700 1 --2- 172.21.15.160:0/2954982572 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb2ec0f5410 0x7fb2ec105630 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:15:24.531 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:24.530+0000 7fb2f105d700 1 -- 172.21.15.160:0/2954982572 >> 172.21.15.160:0/2954982572 conn(0x7fb2ec0f0b70 msgr2=0x7fb2ec0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:15:24.531 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:24.530+0000 7fb2f105d700 1 -- 172.21.15.160:0/2954982572 shutdown_connections 2023-12-25T20:15:24.531 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:24.530+0000 7fb2f105d700 1 -- 172.21.15.160:0/2954982572 wait complete. 2023-12-25T20:15:24.531 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:15:25.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:15:24 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:15:24.529748+0000 mon.smithi050 (mon.0) 304 : audit [DBG] from='client.? 172.21.15.160:0/2954982572' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:15:25.907 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:15:25.907 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:15:26.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:15:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:15:24.701858+0000 mgr.smithi050.nbzhun (mgr.14162) 253 : cluster [DBG] pgmap v239: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:15:27.060 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:27.058+0000 7f26c200f700 1 -- 172.21.15.160:0/2551520003 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f26bc0f5410 msgr2=0x7f26bc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:15:27.060 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:27.058+0000 7f26c200f700 1 --2- 172.21.15.160:0/2551520003 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f26bc0f5410 0x7f26bc0f5830 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 rx=0x7f26b0009fb0 tx=0x7f26b0009b30).stop 2023-12-25T20:15:27.060 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:27.058+0000 7f26c200f700 1 -- 172.21.15.160:0/2551520003 shutdown_connections 2023-12-25T20:15:27.061 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:27.058+0000 7f26c200f700 1 --2- 172.21.15.160:0/2551520003 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f26bc0f5410 0x7f26bc0f5830 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:15:27.061 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:27.058+0000 7f26c200f700 1 -- 172.21.15.160:0/2551520003 >> 172.21.15.160:0/2551520003 conn(0x7f26bc0f0b70 msgr2=0x7f26bc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:15:27.061 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:27.058+0000 7f26c200f700 1 -- 172.21.15.160:0/2551520003 shutdown_connections 2023-12-25T20:15:27.061 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:27.058+0000 7f26c200f700 1 -- 172.21.15.160:0/2551520003 wait complete. 2023-12-25T20:15:27.061 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:27.059+0000 7f26c200f700 1 Processor -- start 2023-12-25T20:15:27.061 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:27.059+0000 7f26c200f700 1 -- start start 2023-12-25T20:15:27.061 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:27.059+0000 7f26c200f700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f26bc0f5410 0x7f26bc10b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:15:27.061 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:27.059+0000 7f26c200f700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26bc107240 con 0x7f26bc0f5410 2023-12-25T20:15:27.061 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:27.059+0000 7f26bb7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f26bc0f5410 0x7f26bc10b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:15:27.061 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:27.059+0000 7f26bb7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f26bc0f5410 0x7f26bc10b850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:38880/0 (socket says 172.21.15.160:38880) 2023-12-25T20:15:27.061 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:27.059+0000 7f26bb7fe700 1 -- 172.21.15.160:0/3478509729 learned_addr learned my addr 172.21.15.160:0/3478509729 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:15:27.061 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:27.060+0000 7f26bb7fe700 1 -- 172.21.15.160:0/3478509729 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26b0005040 con 0x7f26bc0f5410 2023-12-25T20:15:27.061 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:27.060+0000 7f26bb7fe700 1 --2- 172.21.15.160:0/3478509729 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f26bc0f5410 0x7f26bc10b850 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7f26b0014000 tx=0x7f26b000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:15:27.063 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:27.060+0000 7f26a3fff700 1 -- 172.21.15.160:0/3478509729 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f26b00095f0 con 0x7f26bc0f5410 2023-12-25T20:15:27.063 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:27.060+0000 7f26c200f700 1 -- 172.21.15.160:0/3478509729 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26bc060220 con 0x7f26bc0f5410 2023-12-25T20:15:27.063 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:27.060+0000 7f26a3fff700 1 -- 172.21.15.160:0/3478509729 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f26b0007e70 con 0x7f26bc0f5410 2023-12-25T20:15:27.063 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:27.060+0000 7f26a3fff700 1 -- 172.21.15.160:0/3478509729 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f26b0014ea0 con 0x7f26bc0f5410 2023-12-25T20:15:27.063 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:27.060+0000 7f26c200f700 1 -- 172.21.15.160:0/3478509729 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26bc10bd40 con 0x7f26bc0f5410 2023-12-25T20:15:27.063 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:27.061+0000 7f26c200f700 1 -- 172.21.15.160:0/3478509729 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26bc045ad0 con 0x7f26bc0f5410 2023-12-25T20:15:27.063 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:27.061+0000 7f26a3fff700 1 -- 172.21.15.160:0/3478509729 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f26b0025070 con 0x7f26bc0f5410 2023-12-25T20:15:27.063 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:27.061+0000 7f26a3fff700 1 --2- 172.21.15.160:0/3478509729 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f26a402ff60 0x7f26a4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:15:27.063 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:27.061+0000 7f26a3fff700 1 -- 172.21.15.160:0/3478509729 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f26b00409b0 con 0x7f26bc0f5410 2023-12-25T20:15:27.063 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:27.061+0000 7f26baffd700 1 --2- 172.21.15.160:0/3478509729 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f26a402ff60 0x7f26a4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:15:27.063 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:27.062+0000 7f26baffd700 1 --2- 172.21.15.160:0/3478509729 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f26a402ff60 0x7f26a4032420 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f26ac0096a0 tx=0x7f26ac006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:15:27.066 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:27.066+0000 7f26a3fff700 1 -- 172.21.15.160:0/3478509729 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f26b00079c0 con 0x7f26bc0f5410 2023-12-25T20:15:27.272 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:27.271+0000 7f26c200f700 1 -- 172.21.15.160:0/3478509729 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f26bc0fae60 con 0x7f26bc0f5410 2023-12-25T20:15:27.272 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:27.272+0000 7f26a3fff700 1 -- 172.21.15.160:0/3478509729 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f26b000f300 con 0x7f26bc0f5410 2023-12-25T20:15:27.272 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:15:27.272 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:15:27.274 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:27.273+0000 7f26c200f700 1 -- 172.21.15.160:0/3478509729 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f26a402ff60 msgr2=0x7f26a4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:15:27.274 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:27.273+0000 7f26c200f700 1 --2- 172.21.15.160:0/3478509729 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f26a402ff60 0x7f26a4032420 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f26ac0096a0 tx=0x7f26ac006b40).stop 2023-12-25T20:15:27.274 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:27.273+0000 7f26c200f700 1 -- 172.21.15.160:0/3478509729 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f26bc0f5410 msgr2=0x7f26bc10b850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:15:27.274 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:27.273+0000 7f26c200f700 1 --2- 172.21.15.160:0/3478509729 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f26bc0f5410 0x7f26bc10b850 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7f26b0014000 tx=0x7f26b000b040).stop 2023-12-25T20:15:27.274 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:27.273+0000 7f26c200f700 1 -- 172.21.15.160:0/3478509729 shutdown_connections 2023-12-25T20:15:27.274 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:27.273+0000 7f26c200f700 1 --2- 172.21.15.160:0/3478509729 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f26a402ff60 0x7f26a4032420 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:15:27.274 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:27.273+0000 7f26c200f700 1 --2- 172.21.15.160:0/3478509729 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f26bc0f5410 0x7f26bc10b850 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:15:27.274 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:27.273+0000 7f26c200f700 1 -- 172.21.15.160:0/3478509729 >> 172.21.15.160:0/3478509729 conn(0x7f26bc0f0b70 msgr2=0x7f26bc0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:15:27.275 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:27.273+0000 7f26c200f700 1 -- 172.21.15.160:0/3478509729 shutdown_connections 2023-12-25T20:15:27.275 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:27.273+0000 7f26c200f700 1 -- 172.21.15.160:0/3478509729 wait complete. 2023-12-25T20:15:27.275 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:15:28.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:15:27 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:15:26.702221+0000 mgr.smithi050.nbzhun (mgr.14162) 254 : cluster [DBG] pgmap v240: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:15:28.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:15:27 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:15:27.273360+0000 mon.smithi050 (mon.0) 305 : audit [DBG] from='client.? 172.21.15.160:0/3478509729' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:15:28.659 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:15:28.659 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:15:29.772 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:29.770+0000 7ff4dfa17700 1 -- 172.21.15.160:0/1807636954 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff4d80f6510 msgr2=0x7ff4d80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:15:29.772 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:29.770+0000 7ff4dfa17700 1 --2- 172.21.15.160:0/1807636954 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff4d80f6510 0x7ff4d80f6930 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 rx=0x7ff4cc009fb0 tx=0x7ff4cc009b30).stop 2023-12-25T20:15:29.772 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:29.770+0000 7ff4dfa17700 1 -- 172.21.15.160:0/1807636954 shutdown_connections 2023-12-25T20:15:29.772 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:29.770+0000 7ff4dfa17700 1 --2- 172.21.15.160:0/1807636954 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff4d80f6510 0x7ff4d80f6930 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:15:29.772 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:29.770+0000 7ff4dfa17700 1 -- 172.21.15.160:0/1807636954 >> 172.21.15.160:0/1807636954 conn(0x7ff4d80f1c70 msgr2=0x7ff4d80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:15:29.772 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:29.770+0000 7ff4dfa17700 1 -- 172.21.15.160:0/1807636954 shutdown_connections 2023-12-25T20:15:29.772 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:29.770+0000 7ff4dfa17700 1 -- 172.21.15.160:0/1807636954 wait complete. 2023-12-25T20:15:29.772 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:29.771+0000 7ff4dfa17700 1 Processor -- start 2023-12-25T20:15:29.772 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:29.771+0000 7ff4dfa17700 1 -- start start 2023-12-25T20:15:29.772 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:29.771+0000 7ff4dfa17700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff4d80f6510 0x7ff4d8106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:15:29.772 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:29.771+0000 7ff4dfa17700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4d8106c20 con 0x7ff4d80f6510 2023-12-25T20:15:29.772 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:29.771+0000 7ff4dd7b3700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff4d80f6510 0x7ff4d8106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:15:29.773 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:29.771+0000 7ff4dd7b3700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff4d80f6510 0x7ff4d8106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:53256/0 (socket says 172.21.15.160:53256) 2023-12-25T20:15:29.773 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:29.771+0000 7ff4dd7b3700 1 -- 172.21.15.160:0/613335399 learned_addr learned my addr 172.21.15.160:0/613335399 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:15:29.774 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:29.772+0000 7ff4dd7b3700 1 -- 172.21.15.160:0/613335399 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4cc005040 con 0x7ff4d80f6510 2023-12-25T20:15:29.774 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:29.772+0000 7ff4dd7b3700 1 --2- 172.21.15.160:0/613335399 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff4d80f6510 0x7ff4d8106730 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7ff4cc013510 tx=0x7ff4cc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:15:29.774 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:29.772+0000 7ff4c9ffb700 1 -- 172.21.15.160:0/613335399 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff4cc009320 con 0x7ff4d80f6510 2023-12-25T20:15:29.774 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:29.772+0000 7ff4dfa17700 1 -- 172.21.15.160:0/613335399 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4d8106e20 con 0x7ff4d80f6510 2023-12-25T20:15:29.774 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:29.772+0000 7ff4c9ffb700 1 -- 172.21.15.160:0/613335399 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff4cc013f00 con 0x7ff4d80f6510 2023-12-25T20:15:29.775 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:29.772+0000 7ff4c9ffb700 1 -- 172.21.15.160:0/613335399 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff4cc014d90 con 0x7ff4d80f6510 2023-12-25T20:15:29.775 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:29.772+0000 7ff4dfa17700 1 -- 172.21.15.160:0/613335399 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4d81072b0 con 0x7ff4d80f6510 2023-12-25T20:15:29.775 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:29.773+0000 7ff4c9ffb700 1 -- 172.21.15.160:0/613335399 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff4cc0095f0 con 0x7ff4d80f6510 2023-12-25T20:15:29.775 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:29.773+0000 7ff4dfa17700 1 -- 172.21.15.160:0/613335399 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4d8045ad0 con 0x7ff4d80f6510 2023-12-25T20:15:29.775 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:29.773+0000 7ff4c9ffb700 1 --2- 172.21.15.160:0/613335399 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7ff4c402ff60 0x7ff4c4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:15:29.775 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:29.773+0000 7ff4c9ffb700 1 -- 172.21.15.160:0/613335399 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff4cc0410c0 con 0x7ff4d80f6510 2023-12-25T20:15:29.775 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:29.773+0000 7ff4dcfb2700 1 --2- 172.21.15.160:0/613335399 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7ff4c402ff60 0x7ff4c4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:15:29.775 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:29.774+0000 7ff4dcfb2700 1 --2- 172.21.15.160:0/613335399 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7ff4c402ff60 0x7ff4c4032420 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7ff4d40096a0 tx=0x7ff4d4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:15:29.778 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:29.778+0000 7ff4c9ffb700 1 -- 172.21.15.160:0/613335399 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff4cc01d7f0 con 0x7ff4d80f6510 2023-12-25T20:15:29.987 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:29.987+0000 7ff4dfa17700 1 -- 172.21.15.160:0/613335399 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff4d80fbf60 con 0x7ff4d80f6510 2023-12-25T20:15:29.988 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:29.988+0000 7ff4c9ffb700 1 -- 172.21.15.160:0/613335399 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff4cc010230 con 0x7ff4d80f6510 2023-12-25T20:15:29.988 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:15:29.989 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:15:29.990 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:29.989+0000 7ff4dfa17700 1 -- 172.21.15.160:0/613335399 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7ff4c402ff60 msgr2=0x7ff4c4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:15:29.990 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:29.989+0000 7ff4dfa17700 1 --2- 172.21.15.160:0/613335399 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7ff4c402ff60 0x7ff4c4032420 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7ff4d40096a0 tx=0x7ff4d4006b40).stop 2023-12-25T20:15:29.990 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:29.989+0000 7ff4dfa17700 1 -- 172.21.15.160:0/613335399 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff4d80f6510 msgr2=0x7ff4d8106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:15:29.990 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:29.989+0000 7ff4dfa17700 1 --2- 172.21.15.160:0/613335399 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff4d80f6510 0x7ff4d8106730 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7ff4cc013510 tx=0x7ff4cc00b040).stop 2023-12-25T20:15:29.990 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:29.989+0000 7ff4dfa17700 1 -- 172.21.15.160:0/613335399 shutdown_connections 2023-12-25T20:15:29.990 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:29.989+0000 7ff4dfa17700 1 --2- 172.21.15.160:0/613335399 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7ff4c402ff60 0x7ff4c4032420 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:15:29.991 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:29.989+0000 7ff4dfa17700 1 --2- 172.21.15.160:0/613335399 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff4d80f6510 0x7ff4d8106730 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:15:29.991 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:29.989+0000 7ff4dfa17700 1 -- 172.21.15.160:0/613335399 >> 172.21.15.160:0/613335399 conn(0x7ff4d80f1c70 msgr2=0x7ff4d80f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:15:29.991 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:29.989+0000 7ff4dfa17700 1 -- 172.21.15.160:0/613335399 shutdown_connections 2023-12-25T20:15:29.991 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:29.990+0000 7ff4dfa17700 1 -- 172.21.15.160:0/613335399 wait complete. 2023-12-25T20:15:29.991 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:15:30.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:15:29 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:15:28.702603+0000 mgr.smithi050.nbzhun (mgr.14162) 255 : cluster [DBG] pgmap v241: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:15:31.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:15:30 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:15:29.989331+0000 mon.smithi050 (mon.0) 306 : audit [DBG] from='client.? 172.21.15.160:0/613335399' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:15:31.402 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:15:31.403 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:15:32.220 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:15:31 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:15:30.702893+0000 mgr.smithi050.nbzhun (mgr.14162) 256 : cluster [DBG] pgmap v242: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:15:32.567 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:32.566+0000 7f1e0ce3d700 1 -- 172.21.15.160:0/1391633641 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1e080f5410 msgr2=0x7f1e080f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:15:32.567 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:32.566+0000 7f1e0ce3d700 1 --2- 172.21.15.160:0/1391633641 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1e080f5410 0x7f1e080f5830 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 rx=0x7f1df0009fb0 tx=0x7f1df0009b30).stop 2023-12-25T20:15:32.567 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:32.566+0000 7f1e0ce3d700 1 -- 172.21.15.160:0/1391633641 shutdown_connections 2023-12-25T20:15:32.567 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:32.566+0000 7f1e0ce3d700 1 --2- 172.21.15.160:0/1391633641 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1e080f5410 0x7f1e080f5830 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:15:32.567 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:32.566+0000 7f1e0ce3d700 1 -- 172.21.15.160:0/1391633641 >> 172.21.15.160:0/1391633641 conn(0x7f1e080f0b70 msgr2=0x7f1e080f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:15:32.567 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:32.566+0000 7f1e0ce3d700 1 -- 172.21.15.160:0/1391633641 shutdown_connections 2023-12-25T20:15:32.567 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:32.567+0000 7f1e0ce3d700 1 -- 172.21.15.160:0/1391633641 wait complete. 2023-12-25T20:15:32.567 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:32.567+0000 7f1e0ce3d700 1 Processor -- start 2023-12-25T20:15:32.567 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:32.567+0000 7f1e0ce3d700 1 -- start start 2023-12-25T20:15:32.567 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:32.567+0000 7f1e0ce3d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1e080f5410 0x7f1e08109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:15:32.568 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:32.567+0000 7f1e0ce3d700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e08060240 con 0x7f1e080f5410 2023-12-25T20:15:32.569 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:32.567+0000 7f1e0659c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1e080f5410 0x7f1e08109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:15:32.569 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:32.567+0000 7f1e0659c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1e080f5410 0x7f1e08109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:53276/0 (socket says 172.21.15.160:53276) 2023-12-25T20:15:32.569 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:32.567+0000 7f1e0659c700 1 -- 172.21.15.160:0/4047586419 learned_addr learned my addr 172.21.15.160:0/4047586419 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:15:32.570 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:32.568+0000 7f1e0659c700 1 -- 172.21.15.160:0/4047586419 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1df0005040 con 0x7f1e080f5410 2023-12-25T20:15:32.570 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:32.568+0000 7f1e0659c700 1 --2- 172.21.15.160:0/4047586419 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1e080f5410 0x7f1e08109430 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7f1df0013510 tx=0x7f1df000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:15:32.570 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:32.568+0000 7f1dfeffd700 1 -- 172.21.15.160:0/4047586419 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1df0009320 con 0x7f1e080f5410 2023-12-25T20:15:32.570 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:32.568+0000 7f1e0ce3d700 1 -- 172.21.15.160:0/4047586419 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e08060440 con 0x7f1e080f5410 2023-12-25T20:15:32.570 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:32.568+0000 7f1e0ce3d700 1 -- 172.21.15.160:0/4047586419 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e08109ae0 con 0x7f1e080f5410 2023-12-25T20:15:32.570 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:32.568+0000 7f1dfeffd700 1 -- 172.21.15.160:0/4047586419 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1df0013f00 con 0x7f1e080f5410 2023-12-25T20:15:32.570 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:32.568+0000 7f1dfeffd700 1 -- 172.21.15.160:0/4047586419 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1df0014d90 con 0x7f1e080f5410 2023-12-25T20:15:32.570 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:32.569+0000 7f1e0ce3d700 1 -- 172.21.15.160:0/4047586419 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e08045ad0 con 0x7f1e080f5410 2023-12-25T20:15:32.570 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:32.569+0000 7f1dfeffd700 1 -- 172.21.15.160:0/4047586419 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1df00095f0 con 0x7f1e080f5410 2023-12-25T20:15:32.570 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:32.569+0000 7f1dfeffd700 1 --2- 172.21.15.160:0/4047586419 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f1df402ffb0 0x7f1df4032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:15:32.570 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:32.569+0000 7f1dfeffd700 1 -- 172.21.15.160:0/4047586419 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1df0041020 con 0x7f1e080f5410 2023-12-25T20:15:32.570 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:32.569+0000 7f1e05d9b700 1 --2- 172.21.15.160:0/4047586419 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f1df402ffb0 0x7f1df4032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:15:32.571 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:32.570+0000 7f1e05d9b700 1 --2- 172.21.15.160:0/4047586419 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f1df402ffb0 0x7f1df4032470 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f1df80096a0 tx=0x7f1df8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:15:32.573 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:32.574+0000 7f1dfeffd700 1 -- 172.21.15.160:0/4047586419 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1df0014420 con 0x7f1e080f5410 2023-12-25T20:15:32.780 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:32.780+0000 7f1e0ce3d700 1 -- 172.21.15.160:0/4047586419 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1e080fae60 con 0x7f1e080f5410 2023-12-25T20:15:32.780 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:32.781+0000 7f1dfeffd700 1 -- 172.21.15.160:0/4047586419 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1df0010230 con 0x7f1e080f5410 2023-12-25T20:15:32.781 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:15:32.781 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:15:32.782 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:32.783+0000 7f1e0ce3d700 1 -- 172.21.15.160:0/4047586419 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f1df402ffb0 msgr2=0x7f1df4032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:15:32.782 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:32.783+0000 7f1e0ce3d700 1 --2- 172.21.15.160:0/4047586419 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f1df402ffb0 0x7f1df4032470 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f1df80096a0 tx=0x7f1df8006b40).stop 2023-12-25T20:15:32.783 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:32.783+0000 7f1e0ce3d700 1 -- 172.21.15.160:0/4047586419 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1e080f5410 msgr2=0x7f1e08109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:15:32.783 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:32.783+0000 7f1e0ce3d700 1 --2- 172.21.15.160:0/4047586419 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1e080f5410 0x7f1e08109430 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7f1df0013510 tx=0x7f1df000b040).stop 2023-12-25T20:15:32.783 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:32.783+0000 7f1e0ce3d700 1 -- 172.21.15.160:0/4047586419 shutdown_connections 2023-12-25T20:15:32.783 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:32.783+0000 7f1e0ce3d700 1 --2- 172.21.15.160:0/4047586419 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f1df402ffb0 0x7f1df4032470 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:15:32.783 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:32.783+0000 7f1e0ce3d700 1 --2- 172.21.15.160:0/4047586419 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1e080f5410 0x7f1e08109430 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:15:32.783 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:32.783+0000 7f1e0ce3d700 1 -- 172.21.15.160:0/4047586419 >> 172.21.15.160:0/4047586419 conn(0x7f1e080f0b70 msgr2=0x7f1e080f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:15:32.783 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:32.783+0000 7f1e0ce3d700 1 -- 172.21.15.160:0/4047586419 shutdown_connections 2023-12-25T20:15:32.783 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:32.783+0000 7f1e0ce3d700 1 -- 172.21.15.160:0/4047586419 wait complete. 2023-12-25T20:15:32.783 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:15:33.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:15:32 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:15:32.781774+0000 mon.smithi050 (mon.0) 307 : audit [DBG] from='client.? 172.21.15.160:0/4047586419' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:15:34.162 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:15:34.162 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:15:34.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:15:33 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:15:32.703282+0000 mgr.smithi050.nbzhun (mgr.14162) 257 : cluster [DBG] pgmap v243: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:15:35.339 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:35.337+0000 7f1e5a3e8700 1 -- 172.21.15.160:0/272527978 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1e540f5410 msgr2=0x7f1e540f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:15:35.339 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:35.337+0000 7f1e5a3e8700 1 --2- 172.21.15.160:0/272527978 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1e540f5410 0x7f1e540f5830 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 rx=0x7f1e44009fb0 tx=0x7f1e44009b30).stop 2023-12-25T20:15:35.339 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:35.338+0000 7f1e5a3e8700 1 -- 172.21.15.160:0/272527978 shutdown_connections 2023-12-25T20:15:35.339 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:35.338+0000 7f1e5a3e8700 1 --2- 172.21.15.160:0/272527978 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1e540f5410 0x7f1e540f5830 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:15:35.339 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:35.338+0000 7f1e5a3e8700 1 -- 172.21.15.160:0/272527978 >> 172.21.15.160:0/272527978 conn(0x7f1e540f0b70 msgr2=0x7f1e540f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:15:35.339 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:35.338+0000 7f1e5a3e8700 1 -- 172.21.15.160:0/272527978 shutdown_connections 2023-12-25T20:15:35.340 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:35.338+0000 7f1e5a3e8700 1 -- 172.21.15.160:0/272527978 wait complete. 2023-12-25T20:15:35.340 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:35.338+0000 7f1e5a3e8700 1 Processor -- start 2023-12-25T20:15:35.340 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:35.338+0000 7f1e5a3e8700 1 -- start start 2023-12-25T20:15:35.340 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:35.338+0000 7f1e5a3e8700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1e540f5410 0x7f1e541066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:15:35.340 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:35.338+0000 7f1e5a3e8700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e54106bd0 con 0x7f1e540f5410 2023-12-25T20:15:35.340 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:35.338+0000 7f1e53fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1e540f5410 0x7f1e541066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:15:35.340 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:35.339+0000 7f1e53fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1e540f5410 0x7f1e541066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:53292/0 (socket says 172.21.15.160:53292) 2023-12-25T20:15:35.340 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:35.339+0000 7f1e53fff700 1 -- 172.21.15.160:0/3867578839 learned_addr learned my addr 172.21.15.160:0/3867578839 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:15:35.340 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:35.339+0000 7f1e53fff700 1 -- 172.21.15.160:0/3867578839 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e44005040 con 0x7f1e540f5410 2023-12-25T20:15:35.340 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:35.339+0000 7f1e53fff700 1 --2- 172.21.15.160:0/3867578839 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1e540f5410 0x7f1e541066e0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7f1e44014000 tx=0x7f1e4400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:15:35.340 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:35.339+0000 7f1e517fa700 1 -- 172.21.15.160:0/3867578839 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1e440095f0 con 0x7f1e540f5410 2023-12-25T20:15:35.340 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:35.339+0000 7f1e517fa700 1 -- 172.21.15.160:0/3867578839 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1e44007e70 con 0x7f1e540f5410 2023-12-25T20:15:35.340 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:35.339+0000 7f1e517fa700 1 -- 172.21.15.160:0/3867578839 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1e44014ea0 con 0x7f1e540f5410 2023-12-25T20:15:35.340 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:35.339+0000 7f1e5a3e8700 1 -- 172.21.15.160:0/3867578839 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e54106dd0 con 0x7f1e540f5410 2023-12-25T20:15:35.340 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:35.339+0000 7f1e5a3e8700 1 -- 172.21.15.160:0/3867578839 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e54107260 con 0x7f1e540f5410 2023-12-25T20:15:35.341 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:35.340+0000 7f1e517fa700 1 -- 172.21.15.160:0/3867578839 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1e44025070 con 0x7f1e540f5410 2023-12-25T20:15:35.341 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:35.340+0000 7f1e517fa700 1 --2- 172.21.15.160:0/3867578839 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f1e4002ffb0 0x7f1e40032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:15:35.342 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:35.340+0000 7f1e517fa700 1 -- 172.21.15.160:0/3867578839 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1e44040a30 con 0x7f1e540f5410 2023-12-25T20:15:35.342 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:35.340+0000 7f1e5a3e8700 1 -- 172.21.15.160:0/3867578839 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e54045ad0 con 0x7f1e540f5410 2023-12-25T20:15:35.342 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:35.340+0000 7f1e4bfff700 1 --2- 172.21.15.160:0/3867578839 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f1e4002ffb0 0x7f1e40032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:15:35.342 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:35.341+0000 7f1e4bfff700 1 --2- 172.21.15.160:0/3867578839 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f1e4002ffb0 0x7f1e40032470 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f1e3c0096a0 tx=0x7f1e3c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:15:35.344 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:35.345+0000 7f1e517fa700 1 -- 172.21.15.160:0/3867578839 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1e4400f4e0 con 0x7f1e540f5410 2023-12-25T20:15:35.550 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:35.551+0000 7f1e5a3e8700 1 -- 172.21.15.160:0/3867578839 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1e540faf50 con 0x7f1e540f5410 2023-12-25T20:15:35.551 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:35.551+0000 7f1e517fa700 1 -- 172.21.15.160:0/3867578839 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1e4402d090 con 0x7f1e540f5410 2023-12-25T20:15:35.551 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:15:35.551 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:15:35.553 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:35.553+0000 7f1e5a3e8700 1 -- 172.21.15.160:0/3867578839 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f1e4002ffb0 msgr2=0x7f1e40032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:15:35.553 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:35.553+0000 7f1e5a3e8700 1 --2- 172.21.15.160:0/3867578839 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f1e4002ffb0 0x7f1e40032470 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f1e3c0096a0 tx=0x7f1e3c006b40).stop 2023-12-25T20:15:35.553 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:35.553+0000 7f1e5a3e8700 1 -- 172.21.15.160:0/3867578839 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1e540f5410 msgr2=0x7f1e541066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:15:35.553 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:35.553+0000 7f1e5a3e8700 1 --2- 172.21.15.160:0/3867578839 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1e540f5410 0x7f1e541066e0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7f1e44014000 tx=0x7f1e4400b040).stop 2023-12-25T20:15:35.553 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:35.553+0000 7f1e5a3e8700 1 -- 172.21.15.160:0/3867578839 shutdown_connections 2023-12-25T20:15:35.553 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:35.553+0000 7f1e5a3e8700 1 --2- 172.21.15.160:0/3867578839 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f1e4002ffb0 0x7f1e40032470 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:15:35.553 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:35.553+0000 7f1e5a3e8700 1 --2- 172.21.15.160:0/3867578839 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1e540f5410 0x7f1e541066e0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:15:35.553 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:35.553+0000 7f1e5a3e8700 1 -- 172.21.15.160:0/3867578839 >> 172.21.15.160:0/3867578839 conn(0x7f1e540f0b70 msgr2=0x7f1e5405fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:15:35.553 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:35.553+0000 7f1e5a3e8700 1 -- 172.21.15.160:0/3867578839 shutdown_connections 2023-12-25T20:15:35.554 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:35.553+0000 7f1e5a3e8700 1 -- 172.21.15.160:0/3867578839 wait complete. 2023-12-25T20:15:35.554 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:15:36.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:15:35 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:15:34.703659+0000 mgr.smithi050.nbzhun (mgr.14162) 258 : cluster [DBG] pgmap v244: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:15:36.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:15:35 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:15:35.552256+0000 mon.smithi050 (mon.0) 308 : audit [DBG] from='client.? 172.21.15.160:0/3867578839' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:15:36.950 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:15:36.951 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:15:38.099 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:38.099+0000 7fb6463eb700 1 -- 172.21.15.160:0/1054890719 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb6400f5410 msgr2=0x7fb6400f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:15:38.099 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:38.099+0000 7fb6463eb700 1 --2- 172.21.15.160:0/1054890719 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb6400f5410 0x7fb6400f5830 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 rx=0x7fb630009fb0 tx=0x7fb630009b30).stop 2023-12-25T20:15:38.099 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:38.099+0000 7fb6463eb700 1 -- 172.21.15.160:0/1054890719 shutdown_connections 2023-12-25T20:15:38.099 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:38.099+0000 7fb6463eb700 1 --2- 172.21.15.160:0/1054890719 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb6400f5410 0x7fb6400f5830 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:15:38.100 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:38.099+0000 7fb6463eb700 1 -- 172.21.15.160:0/1054890719 >> 172.21.15.160:0/1054890719 conn(0x7fb6400f0b70 msgr2=0x7fb6400f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:15:38.100 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:38.099+0000 7fb6463eb700 1 -- 172.21.15.160:0/1054890719 shutdown_connections 2023-12-25T20:15:38.100 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:38.099+0000 7fb6463eb700 1 -- 172.21.15.160:0/1054890719 wait complete. 2023-12-25T20:15:38.100 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:38.099+0000 7fb6463eb700 1 Processor -- start 2023-12-25T20:15:38.100 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:38.099+0000 7fb6463eb700 1 -- start start 2023-12-25T20:15:38.100 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:38.100+0000 7fb6463eb700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb6400f5410 0x7fb6400fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:15:38.100 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:38.100+0000 7fb6463eb700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6400fb820 con 0x7fb6400f5410 2023-12-25T20:15:38.102 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:38.100+0000 7fb63ffff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb6400f5410 0x7fb6400fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:15:38.102 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:38.100+0000 7fb63ffff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb6400f5410 0x7fb6400fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:40012/0 (socket says 172.21.15.160:40012) 2023-12-25T20:15:38.102 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:38.100+0000 7fb63ffff700 1 -- 172.21.15.160:0/641854342 learned_addr learned my addr 172.21.15.160:0/641854342 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:15:38.102 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:38.100+0000 7fb63ffff700 1 -- 172.21.15.160:0/641854342 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb630005040 con 0x7fb6400f5410 2023-12-25T20:15:38.102 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:38.101+0000 7fb63ffff700 1 --2- 172.21.15.160:0/641854342 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb6400f5410 0x7fb6400fd3a0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7fb6300089f0 tx=0x7fb63000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:15:38.102 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:38.101+0000 7fb63cff9700 1 -- 172.21.15.160:0/641854342 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb630009320 con 0x7fb6400f5410 2023-12-25T20:15:38.102 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:38.101+0000 7fb6463eb700 1 -- 172.21.15.160:0/641854342 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6400fb9c0 con 0x7fb6400f5410 2023-12-25T20:15:38.102 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:38.101+0000 7fb63cff9700 1 -- 172.21.15.160:0/641854342 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb6300136c0 con 0x7fb6400f5410 2023-12-25T20:15:38.102 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:38.101+0000 7fb63cff9700 1 -- 172.21.15.160:0/641854342 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb63001ecb0 con 0x7fb6400f5410 2023-12-25T20:15:38.102 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:38.101+0000 7fb6463eb700 1 -- 172.21.15.160:0/641854342 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6400fbe50 con 0x7fb6400f5410 2023-12-25T20:15:38.102 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:38.102+0000 7fb63cff9700 1 -- 172.21.15.160:0/641854342 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb63001e420 con 0x7fb6400f5410 2023-12-25T20:15:38.102 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:38.102+0000 7fb6463eb700 1 -- 172.21.15.160:0/641854342 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb640059100 con 0x7fb6400f5410 2023-12-25T20:15:38.102 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:38.102+0000 7fb63cff9700 1 --2- 172.21.15.160:0/641854342 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fb62802ff60 0x7fb628032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:15:38.102 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:38.102+0000 7fb63cff9700 1 -- 172.21.15.160:0/641854342 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb630040eb0 con 0x7fb6400f5410 2023-12-25T20:15:38.103 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:38.102+0000 7fb63f7fe700 1 --2- 172.21.15.160:0/641854342 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fb62802ff60 0x7fb628032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:15:38.103 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:38.103+0000 7fb63f7fe700 1 --2- 172.21.15.160:0/641854342 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fb62802ff60 0x7fb628032420 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7fb6340096a0 tx=0x7fb634006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:15:38.106 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:38.107+0000 7fb63cff9700 1 -- 172.21.15.160:0/641854342 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb63001cb50 con 0x7fb6400f5410 2023-12-25T20:15:38.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:15:37 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:15:36.704010+0000 mgr.smithi050.nbzhun (mgr.14162) 259 : cluster [DBG] pgmap v245: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:15:38.311 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:38.311+0000 7fb6463eb700 1 -- 172.21.15.160:0/641854342 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb6400fae60 con 0x7fb6400f5410 2023-12-25T20:15:38.312 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:38.312+0000 7fb63cff9700 1 -- 172.21.15.160:0/641854342 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb630004490 con 0x7fb6400f5410 2023-12-25T20:15:38.312 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:15:38.312 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:15:38.314 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:38.314+0000 7fb6463eb700 1 -- 172.21.15.160:0/641854342 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fb62802ff60 msgr2=0x7fb628032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:15:38.314 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:38.314+0000 7fb6463eb700 1 --2- 172.21.15.160:0/641854342 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fb62802ff60 0x7fb628032420 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7fb6340096a0 tx=0x7fb634006b40).stop 2023-12-25T20:15:38.314 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:38.314+0000 7fb6463eb700 1 -- 172.21.15.160:0/641854342 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb6400f5410 msgr2=0x7fb6400fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:15:38.314 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:38.314+0000 7fb6463eb700 1 --2- 172.21.15.160:0/641854342 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb6400f5410 0x7fb6400fd3a0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7fb6300089f0 tx=0x7fb63000b040).stop 2023-12-25T20:15:38.314 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:38.314+0000 7fb6463eb700 1 -- 172.21.15.160:0/641854342 shutdown_connections 2023-12-25T20:15:38.314 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:38.314+0000 7fb6463eb700 1 --2- 172.21.15.160:0/641854342 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fb62802ff60 0x7fb628032420 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:15:38.314 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:38.314+0000 7fb6463eb700 1 --2- 172.21.15.160:0/641854342 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb6400f5410 0x7fb6400fd3a0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:15:38.314 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:38.314+0000 7fb6463eb700 1 -- 172.21.15.160:0/641854342 >> 172.21.15.160:0/641854342 conn(0x7fb6400f0b70 msgr2=0x7fb64005fe40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:15:38.314 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:38.314+0000 7fb6463eb700 1 -- 172.21.15.160:0/641854342 shutdown_connections 2023-12-25T20:15:38.314 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:38.314+0000 7fb6463eb700 1 -- 172.21.15.160:0/641854342 wait complete. 2023-12-25T20:15:38.315 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:15:39.220 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:15:38 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:15:38.312984+0000 mon.smithi050 (mon.0) 309 : audit [DBG] from='client.? 172.21.15.160:0/641854342' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:15:39.692 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:15:39.692 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:15:40.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:15:39 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:15:38.704377+0000 mgr.smithi050.nbzhun (mgr.14162) 260 : cluster [DBG] pgmap v246: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:15:40.806 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:40.805+0000 7ff7c4518700 1 -- 172.21.15.160:0/2882463043 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff7bc0f47c0 msgr2=0x7ff7bc0f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:15:40.806 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:40.805+0000 7ff7c4518700 1 --2- 172.21.15.160:0/2882463043 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff7bc0f47c0 0x7ff7bc0f4be0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 rx=0x7ff7ac009fb0 tx=0x7ff7ac009b30).stop 2023-12-25T20:15:40.806 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:40.805+0000 7ff7c4518700 1 -- 172.21.15.160:0/2882463043 shutdown_connections 2023-12-25T20:15:40.807 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:40.805+0000 7ff7c4518700 1 --2- 172.21.15.160:0/2882463043 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff7bc0f47c0 0x7ff7bc0f4be0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:15:40.807 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:40.805+0000 7ff7c4518700 1 -- 172.21.15.160:0/2882463043 >> 172.21.15.160:0/2882463043 conn(0x7ff7bc0f02e0 msgr2=0x7ff7bc0f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:15:40.807 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:40.805+0000 7ff7c4518700 1 -- 172.21.15.160:0/2882463043 shutdown_connections 2023-12-25T20:15:40.807 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:40.805+0000 7ff7c4518700 1 -- 172.21.15.160:0/2882463043 wait complete. 2023-12-25T20:15:40.807 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:40.806+0000 7ff7c4518700 1 Processor -- start 2023-12-25T20:15:40.807 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:40.806+0000 7ff7c4518700 1 -- start start 2023-12-25T20:15:40.807 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:40.806+0000 7ff7c4518700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff7bc0f47c0 0x7ff7bc0f7370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:15:40.807 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:40.806+0000 7ff7c4518700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7bc0f57f0 con 0x7ff7bc0f47c0 2023-12-25T20:15:40.807 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:40.806+0000 7ff7c22b4700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff7bc0f47c0 0x7ff7bc0f7370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:15:40.807 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:40.806+0000 7ff7c22b4700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff7bc0f47c0 0x7ff7bc0f7370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:40022/0 (socket says 172.21.15.160:40022) 2023-12-25T20:15:40.807 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:40.806+0000 7ff7c22b4700 1 -- 172.21.15.160:0/116984067 learned_addr learned my addr 172.21.15.160:0/116984067 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:15:40.807 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:40.806+0000 7ff7c22b4700 1 -- 172.21.15.160:0/116984067 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7ac005040 con 0x7ff7bc0f47c0 2023-12-25T20:15:40.807 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:40.806+0000 7ff7c22b4700 1 --2- 172.21.15.160:0/116984067 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff7bc0f47c0 0x7ff7bc0f7370 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7ff7ac013510 tx=0x7ff7ac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:15:40.807 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:40.807+0000 7ff7b2ffd700 1 -- 172.21.15.160:0/116984067 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff7ac009320 con 0x7ff7bc0f47c0 2023-12-25T20:15:40.808 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:40.807+0000 7ff7c4518700 1 -- 172.21.15.160:0/116984067 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7bc0f5990 con 0x7ff7bc0f47c0 2023-12-25T20:15:40.808 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:40.807+0000 7ff7b2ffd700 1 -- 172.21.15.160:0/116984067 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff7ac013f00 con 0x7ff7bc0f47c0 2023-12-25T20:15:40.808 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:40.807+0000 7ff7b2ffd700 1 -- 172.21.15.160:0/116984067 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff7ac014d90 con 0x7ff7bc0f47c0 2023-12-25T20:15:40.808 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:40.807+0000 7ff7c4518700 1 -- 172.21.15.160:0/116984067 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7bc0f5e20 con 0x7ff7bc0f47c0 2023-12-25T20:15:40.808 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:40.807+0000 7ff7c4518700 1 -- 172.21.15.160:0/116984067 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7bc059100 con 0x7ff7bc0f47c0 2023-12-25T20:15:40.809 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:40.808+0000 7ff7b2ffd700 1 -- 172.21.15.160:0/116984067 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff7ac0095f0 con 0x7ff7bc0f47c0 2023-12-25T20:15:40.809 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:40.808+0000 7ff7b2ffd700 1 --2- 172.21.15.160:0/116984067 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7ff7a802ff60 0x7ff7a8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:15:40.809 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:40.808+0000 7ff7b2ffd700 1 -- 172.21.15.160:0/116984067 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff7ac040fe0 con 0x7ff7bc0f47c0 2023-12-25T20:15:40.809 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:40.808+0000 7ff7c1ab3700 1 --2- 172.21.15.160:0/116984067 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7ff7a802ff60 0x7ff7a8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:15:40.809 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:40.808+0000 7ff7c1ab3700 1 --2- 172.21.15.160:0/116984067 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7ff7a802ff60 0x7ff7a8032420 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7ff7b80096a0 tx=0x7ff7b8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:15:40.812 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:40.812+0000 7ff7b2ffd700 1 -- 172.21.15.160:0/116984067 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff7ac010230 con 0x7ff7bc0f47c0 2023-12-25T20:15:41.019 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:41.020+0000 7ff7c4518700 1 -- 172.21.15.160:0/116984067 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff7bc045ad0 con 0x7ff7bc0f47c0 2023-12-25T20:15:41.020 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:41.021+0000 7ff7b2ffd700 1 -- 172.21.15.160:0/116984067 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff7ac010230 con 0x7ff7bc0f47c0 2023-12-25T20:15:41.021 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:15:41.021 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:15:41.022 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:41.022+0000 7ff7c4518700 1 -- 172.21.15.160:0/116984067 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7ff7a802ff60 msgr2=0x7ff7a8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:15:41.022 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:41.022+0000 7ff7c4518700 1 --2- 172.21.15.160:0/116984067 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7ff7a802ff60 0x7ff7a8032420 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7ff7b80096a0 tx=0x7ff7b8006b40).stop 2023-12-25T20:15:41.022 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:41.022+0000 7ff7c4518700 1 -- 172.21.15.160:0/116984067 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff7bc0f47c0 msgr2=0x7ff7bc0f7370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:15:41.022 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:41.022+0000 7ff7c4518700 1 --2- 172.21.15.160:0/116984067 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff7bc0f47c0 0x7ff7bc0f7370 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7ff7ac013510 tx=0x7ff7ac00b040).stop 2023-12-25T20:15:41.022 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:41.022+0000 7ff7c4518700 1 -- 172.21.15.160:0/116984067 shutdown_connections 2023-12-25T20:15:41.022 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:41.022+0000 7ff7c4518700 1 --2- 172.21.15.160:0/116984067 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7ff7a802ff60 0x7ff7a8032420 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:15:41.023 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:41.022+0000 7ff7c4518700 1 --2- 172.21.15.160:0/116984067 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff7bc0f47c0 0x7ff7bc0f7370 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:15:41.023 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:41.022+0000 7ff7c4518700 1 -- 172.21.15.160:0/116984067 >> 172.21.15.160:0/116984067 conn(0x7ff7bc0f02e0 msgr2=0x7ff7bc0f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:15:41.023 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:41.022+0000 7ff7c4518700 1 -- 172.21.15.160:0/116984067 shutdown_connections 2023-12-25T20:15:41.023 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:41.022+0000 7ff7c4518700 1 -- 172.21.15.160:0/116984067 wait complete. 2023-12-25T20:15:41.024 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:15:42.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:15:41 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:15:40.704733+0000 mgr.smithi050.nbzhun (mgr.14162) 261 : cluster [DBG] pgmap v247: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:15:42.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:15:41 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:15:41.021298+0000 mon.smithi050 (mon.0) 310 : audit [DBG] from='client.? 172.21.15.160:0/116984067' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:15:42.450 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:15:42.450 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:15:43.657 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:43.657+0000 7ff2f8cfd700 1 -- 172.21.15.160:0/1647423930 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff2f40f5410 msgr2=0x7ff2f40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:15:43.657 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:43.657+0000 7ff2f8cfd700 1 --2- 172.21.15.160:0/1647423930 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff2f40f5410 0x7ff2f40f5830 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 rx=0x7ff2e4009fb0 tx=0x7ff2e4009b30).stop 2023-12-25T20:15:43.657 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:43.657+0000 7ff2f8cfd700 1 -- 172.21.15.160:0/1647423930 shutdown_connections 2023-12-25T20:15:43.658 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:43.657+0000 7ff2f8cfd700 1 --2- 172.21.15.160:0/1647423930 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff2f40f5410 0x7ff2f40f5830 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:15:43.658 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:43.657+0000 7ff2f8cfd700 1 -- 172.21.15.160:0/1647423930 >> 172.21.15.160:0/1647423930 conn(0x7ff2f40f0b70 msgr2=0x7ff2f40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:15:43.658 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:43.657+0000 7ff2f8cfd700 1 -- 172.21.15.160:0/1647423930 shutdown_connections 2023-12-25T20:15:43.658 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:43.657+0000 7ff2f8cfd700 1 -- 172.21.15.160:0/1647423930 wait complete. 2023-12-25T20:15:43.658 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:43.658+0000 7ff2f8cfd700 1 Processor -- start 2023-12-25T20:15:43.658 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:43.658+0000 7ff2f8cfd700 1 -- start start 2023-12-25T20:15:43.659 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:43.658+0000 7ff2f8cfd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff2f40f5410 0x7ff2f41055d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:15:43.659 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:43.658+0000 7ff2f8cfd700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2f4105ac0 con 0x7ff2f40f5410 2023-12-25T20:15:43.659 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:43.658+0000 7ff2f259c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff2f40f5410 0x7ff2f41055d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:15:43.660 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:43.658+0000 7ff2f259c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff2f40f5410 0x7ff2f41055d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:40032/0 (socket says 172.21.15.160:40032) 2023-12-25T20:15:43.660 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:43.658+0000 7ff2f259c700 1 -- 172.21.15.160:0/1950446065 learned_addr learned my addr 172.21.15.160:0/1950446065 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:15:43.660 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:43.659+0000 7ff2f259c700 1 -- 172.21.15.160:0/1950446065 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2e4005040 con 0x7ff2f40f5410 2023-12-25T20:15:43.660 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:43.659+0000 7ff2f259c700 1 --2- 172.21.15.160:0/1950446065 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff2f40f5410 0x7ff2f41055d0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7ff2e4014000 tx=0x7ff2e400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:15:43.660 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:43.659+0000 7ff2e2ffd700 1 -- 172.21.15.160:0/1950446065 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff2e40095f0 con 0x7ff2f40f5410 2023-12-25T20:15:43.660 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:43.659+0000 7ff2f8cfd700 1 -- 172.21.15.160:0/1950446065 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2f4105cc0 con 0x7ff2f40f5410 2023-12-25T20:15:43.660 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:43.659+0000 7ff2e2ffd700 1 -- 172.21.15.160:0/1950446065 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff2e4007e70 con 0x7ff2f40f5410 2023-12-25T20:15:43.660 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:43.659+0000 7ff2e2ffd700 1 -- 172.21.15.160:0/1950446065 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff2e4014ea0 con 0x7ff2f40f5410 2023-12-25T20:15:43.660 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:43.659+0000 7ff2f8cfd700 1 -- 172.21.15.160:0/1950446065 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2f4106150 con 0x7ff2f40f5410 2023-12-25T20:15:43.660 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:43.660+0000 7ff2e2ffd700 1 -- 172.21.15.160:0/1950446065 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff2e4014690 con 0x7ff2f40f5410 2023-12-25T20:15:43.660 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:43.660+0000 7ff2f8cfd700 1 -- 172.21.15.160:0/1950446065 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2f4045ad0 con 0x7ff2f40f5410 2023-12-25T20:15:43.660 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:43.660+0000 7ff2e2ffd700 1 --2- 172.21.15.160:0/1950446065 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7ff2dc02ff60 0x7ff2dc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:15:43.660 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:43.660+0000 7ff2e2ffd700 1 -- 172.21.15.160:0/1950446065 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff2e4040870 con 0x7ff2f40f5410 2023-12-25T20:15:43.660 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:43.660+0000 7ff2f1d9b700 1 --2- 172.21.15.160:0/1950446065 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7ff2dc02ff60 0x7ff2dc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:15:43.661 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:43.660+0000 7ff2f1d9b700 1 --2- 172.21.15.160:0/1950446065 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7ff2dc02ff60 0x7ff2dc032420 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7ff2e80096a0 tx=0x7ff2e8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:15:43.664 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:43.665+0000 7ff2e2ffd700 1 -- 172.21.15.160:0/1950446065 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff2e400f300 con 0x7ff2f40f5410 2023-12-25T20:15:43.868 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:43.868+0000 7ff2f8cfd700 1 -- 172.21.15.160:0/1950446065 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff2f4107000 con 0x7ff2f40f5410 2023-12-25T20:15:43.868 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:43.869+0000 7ff2e2ffd700 1 -- 172.21.15.160:0/1950446065 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff2e400f300 con 0x7ff2f40f5410 2023-12-25T20:15:43.869 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:15:43.869 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:15:43.870 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:43.870+0000 7ff2f8cfd700 1 -- 172.21.15.160:0/1950446065 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7ff2dc02ff60 msgr2=0x7ff2dc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:15:43.871 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:43.870+0000 7ff2f8cfd700 1 --2- 172.21.15.160:0/1950446065 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7ff2dc02ff60 0x7ff2dc032420 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7ff2e80096a0 tx=0x7ff2e8006b40).stop 2023-12-25T20:15:43.871 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:43.870+0000 7ff2f8cfd700 1 -- 172.21.15.160:0/1950446065 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff2f40f5410 msgr2=0x7ff2f41055d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:15:43.871 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:43.870+0000 7ff2f8cfd700 1 --2- 172.21.15.160:0/1950446065 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff2f40f5410 0x7ff2f41055d0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7ff2e4014000 tx=0x7ff2e400b040).stop 2023-12-25T20:15:43.871 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:43.870+0000 7ff2f8cfd700 1 -- 172.21.15.160:0/1950446065 shutdown_connections 2023-12-25T20:15:43.871 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:43.870+0000 7ff2f8cfd700 1 --2- 172.21.15.160:0/1950446065 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7ff2dc02ff60 0x7ff2dc032420 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:15:43.871 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:43.870+0000 7ff2f8cfd700 1 --2- 172.21.15.160:0/1950446065 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff2f40f5410 0x7ff2f41055d0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:15:43.871 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:43.871+0000 7ff2f8cfd700 1 -- 172.21.15.160:0/1950446065 >> 172.21.15.160:0/1950446065 conn(0x7ff2f40f0b70 msgr2=0x7ff2f40f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:15:43.872 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:43.871+0000 7ff2f8cfd700 1 -- 172.21.15.160:0/1950446065 shutdown_connections 2023-12-25T20:15:43.872 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:43.871+0000 7ff2f8cfd700 1 -- 172.21.15.160:0/1950446065 wait complete. 2023-12-25T20:15:43.872 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:15:44.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:15:43 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:15:42.705056+0000 mgr.smithi050.nbzhun (mgr.14162) 262 : cluster [DBG] pgmap v248: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:15:44.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:15:43 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:15:43.869652+0000 mon.smithi050 (mon.0) 311 : audit [DBG] from='client.? 172.21.15.160:0/1950446065' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:15:45.273 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:15:45.274 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:15:46.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:15:45 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:15:44.705416+0000 mgr.smithi050.nbzhun (mgr.14162) 263 : cluster [DBG] pgmap v249: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:15:46.408 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:46.407+0000 7f1d2fa76700 1 -- 172.21.15.160:0/2484322578 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1d280f5410 msgr2=0x7f1d280f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:15:46.408 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:46.407+0000 7f1d2fa76700 1 --2- 172.21.15.160:0/2484322578 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1d280f5410 0x7f1d280f5830 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 rx=0x7f1d24009fb0 tx=0x7f1d24009b30).stop 2023-12-25T20:15:46.408 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:46.407+0000 7f1d2fa76700 1 -- 172.21.15.160:0/2484322578 shutdown_connections 2023-12-25T20:15:46.408 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:46.407+0000 7f1d2fa76700 1 --2- 172.21.15.160:0/2484322578 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1d280f5410 0x7f1d280f5830 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:15:46.408 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:46.407+0000 7f1d2fa76700 1 -- 172.21.15.160:0/2484322578 >> 172.21.15.160:0/2484322578 conn(0x7f1d280f0b70 msgr2=0x7f1d280f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:15:46.408 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:46.407+0000 7f1d2fa76700 1 -- 172.21.15.160:0/2484322578 shutdown_connections 2023-12-25T20:15:46.408 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:46.407+0000 7f1d2fa76700 1 -- 172.21.15.160:0/2484322578 wait complete. 2023-12-25T20:15:46.408 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:46.408+0000 7f1d2fa76700 1 Processor -- start 2023-12-25T20:15:46.408 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:46.408+0000 7f1d2fa76700 1 -- start start 2023-12-25T20:15:46.408 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:46.408+0000 7f1d2fa76700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1d280f5410 0x7f1d28106700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:15:46.409 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:46.408+0000 7f1d2fa76700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d28106bf0 con 0x7f1d280f5410 2023-12-25T20:15:46.409 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:46.408+0000 7f1d2d812700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1d280f5410 0x7f1d28106700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:15:46.409 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:46.408+0000 7f1d2d812700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1d280f5410 0x7f1d28106700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:40038/0 (socket says 172.21.15.160:40038) 2023-12-25T20:15:46.409 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:46.408+0000 7f1d2d812700 1 -- 172.21.15.160:0/1496470361 learned_addr learned my addr 172.21.15.160:0/1496470361 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:15:46.409 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:46.408+0000 7f1d2d812700 1 -- 172.21.15.160:0/1496470361 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d24005040 con 0x7f1d280f5410 2023-12-25T20:15:46.409 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:46.408+0000 7f1d2d812700 1 --2- 172.21.15.160:0/1496470361 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1d280f5410 0x7f1d28106700 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7f1d24013510 tx=0x7f1d2400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:15:46.410 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:46.409+0000 7f1d1e7fc700 1 -- 172.21.15.160:0/1496470361 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1d24009320 con 0x7f1d280f5410 2023-12-25T20:15:46.410 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:46.409+0000 7f1d2fa76700 1 -- 172.21.15.160:0/1496470361 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d28106df0 con 0x7f1d280f5410 2023-12-25T20:15:46.410 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:46.409+0000 7f1d1e7fc700 1 -- 172.21.15.160:0/1496470361 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1d24013f00 con 0x7f1d280f5410 2023-12-25T20:15:46.410 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:46.409+0000 7f1d1e7fc700 1 -- 172.21.15.160:0/1496470361 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1d24014d90 con 0x7f1d280f5410 2023-12-25T20:15:46.410 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:46.409+0000 7f1d2fa76700 1 -- 172.21.15.160:0/1496470361 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d28107280 con 0x7f1d280f5410 2023-12-25T20:15:46.410 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:46.409+0000 7f1d1e7fc700 1 -- 172.21.15.160:0/1496470361 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1d240095f0 con 0x7f1d280f5410 2023-12-25T20:15:46.411 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:46.409+0000 7f1d2fa76700 1 -- 172.21.15.160:0/1496470361 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d28045ad0 con 0x7f1d280f5410 2023-12-25T20:15:46.411 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:46.410+0000 7f1d1e7fc700 1 --2- 172.21.15.160:0/1496470361 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f1d1402ff60 0x7f1d14032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:15:46.411 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:46.410+0000 7f1d1e7fc700 1 -- 172.21.15.160:0/1496470361 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1d24040fd0 con 0x7f1d280f5410 2023-12-25T20:15:46.411 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:46.410+0000 7f1d2d011700 1 --2- 172.21.15.160:0/1496470361 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f1d1402ff60 0x7f1d14032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:15:46.411 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:46.410+0000 7f1d2d011700 1 --2- 172.21.15.160:0/1496470361 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f1d1402ff60 0x7f1d14032420 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f1d180096a0 tx=0x7f1d18006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:15:46.414 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:46.414+0000 7f1d1e7fc700 1 -- 172.21.15.160:0/1496470361 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1d24014420 con 0x7f1d280f5410 2023-12-25T20:15:46.621 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:46.621+0000 7f1d2fa76700 1 -- 172.21.15.160:0/1496470361 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1d280fae60 con 0x7f1d280f5410 2023-12-25T20:15:46.621 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:46.622+0000 7f1d1e7fc700 1 -- 172.21.15.160:0/1496470361 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1d2401d7f0 con 0x7f1d280f5410 2023-12-25T20:15:46.621 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:15:46.622 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:15:46.623 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:46.623+0000 7f1d2fa76700 1 -- 172.21.15.160:0/1496470361 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f1d1402ff60 msgr2=0x7f1d14032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:15:46.623 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:46.623+0000 7f1d2fa76700 1 --2- 172.21.15.160:0/1496470361 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f1d1402ff60 0x7f1d14032420 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f1d180096a0 tx=0x7f1d18006b40).stop 2023-12-25T20:15:46.624 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:46.623+0000 7f1d2fa76700 1 -- 172.21.15.160:0/1496470361 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1d280f5410 msgr2=0x7f1d28106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:15:46.624 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:46.623+0000 7f1d2fa76700 1 --2- 172.21.15.160:0/1496470361 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1d280f5410 0x7f1d28106700 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7f1d24013510 tx=0x7f1d2400b040).stop 2023-12-25T20:15:46.624 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:46.623+0000 7f1d2fa76700 1 -- 172.21.15.160:0/1496470361 shutdown_connections 2023-12-25T20:15:46.624 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:46.623+0000 7f1d2fa76700 1 --2- 172.21.15.160:0/1496470361 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f1d1402ff60 0x7f1d14032420 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:15:46.624 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:46.623+0000 7f1d2fa76700 1 --2- 172.21.15.160:0/1496470361 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1d280f5410 0x7f1d28106700 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:15:46.624 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:46.623+0000 7f1d2fa76700 1 -- 172.21.15.160:0/1496470361 >> 172.21.15.160:0/1496470361 conn(0x7f1d280f0b70 msgr2=0x7f1d280f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:15:46.624 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:46.623+0000 7f1d2fa76700 1 -- 172.21.15.160:0/1496470361 shutdown_connections 2023-12-25T20:15:46.624 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:46.623+0000 7f1d2fa76700 1 -- 172.21.15.160:0/1496470361 wait complete. 2023-12-25T20:15:46.624 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:15:47.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:15:46 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:15:46.622320+0000 mon.smithi050 (mon.0) 312 : audit [DBG] from='client.? 172.21.15.160:0/1496470361' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:15:48.017 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:15:48.017 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:15:48.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:15:47 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:15:46.705706+0000 mgr.smithi050.nbzhun (mgr.14162) 264 : cluster [DBG] pgmap v250: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:15:49.195 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:49.193+0000 7fa01ece2700 1 -- 172.21.15.160:0/1103753255 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa0180f5410 msgr2=0x7fa0180f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:15:49.195 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:49.193+0000 7fa01ece2700 1 --2- 172.21.15.160:0/1103753255 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa0180f5410 0x7fa0180f5830 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 rx=0x7fa00c009fb0 tx=0x7fa00c009b30).stop 2023-12-25T20:15:49.196 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:49.193+0000 7fa01ece2700 1 -- 172.21.15.160:0/1103753255 shutdown_connections 2023-12-25T20:15:49.196 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:49.193+0000 7fa01ece2700 1 --2- 172.21.15.160:0/1103753255 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa0180f5410 0x7fa0180f5830 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:15:49.196 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:49.193+0000 7fa01ece2700 1 -- 172.21.15.160:0/1103753255 >> 172.21.15.160:0/1103753255 conn(0x7fa0180f0b70 msgr2=0x7fa0180f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:15:49.196 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:49.194+0000 7fa01ece2700 1 -- 172.21.15.160:0/1103753255 shutdown_connections 2023-12-25T20:15:49.196 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:49.194+0000 7fa01ece2700 1 -- 172.21.15.160:0/1103753255 wait complete. 2023-12-25T20:15:49.196 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:49.194+0000 7fa01ece2700 1 Processor -- start 2023-12-25T20:15:49.196 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:49.194+0000 7fa01ece2700 1 -- start start 2023-12-25T20:15:49.196 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:49.194+0000 7fa01ece2700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa0180f5410 0x7fa018105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:15:49.196 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:49.194+0000 7fa01ece2700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa018105b20 con 0x7fa0180f5410 2023-12-25T20:15:49.196 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:49.195+0000 7fa01ca7e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa0180f5410 0x7fa018105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:15:49.196 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:49.195+0000 7fa01ca7e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa0180f5410 0x7fa018105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:50118/0 (socket says 172.21.15.160:50118) 2023-12-25T20:15:49.197 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:49.195+0000 7fa01ca7e700 1 -- 172.21.15.160:0/1291229742 learned_addr learned my addr 172.21.15.160:0/1291229742 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:15:49.197 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:49.195+0000 7fa01ca7e700 1 -- 172.21.15.160:0/1291229742 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa00c005040 con 0x7fa0180f5410 2023-12-25T20:15:49.197 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:49.195+0000 7fa01ca7e700 1 --2- 172.21.15.160:0/1291229742 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa0180f5410 0x7fa018105630 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7fa00c013510 tx=0x7fa00c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:15:49.197 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:49.195+0000 7fa0157fa700 1 -- 172.21.15.160:0/1291229742 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa00c009320 con 0x7fa0180f5410 2023-12-25T20:15:49.197 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:49.195+0000 7fa01ece2700 1 -- 172.21.15.160:0/1291229742 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa018105d20 con 0x7fa0180f5410 2023-12-25T20:15:49.197 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:49.195+0000 7fa0157fa700 1 -- 172.21.15.160:0/1291229742 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa00c013f00 con 0x7fa0180f5410 2023-12-25T20:15:49.197 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:49.195+0000 7fa0157fa700 1 -- 172.21.15.160:0/1291229742 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa00c014d90 con 0x7fa0180f5410 2023-12-25T20:15:49.197 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:49.195+0000 7fa01ece2700 1 -- 172.21.15.160:0/1291229742 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0181061b0 con 0x7fa0180f5410 2023-12-25T20:15:49.198 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:49.196+0000 7fa0157fa700 1 -- 172.21.15.160:0/1291229742 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa00c0095f0 con 0x7fa0180f5410 2023-12-25T20:15:49.199 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:49.196+0000 7fa01ece2700 1 -- 172.21.15.160:0/1291229742 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa018045ad0 con 0x7fa0180f5410 2023-12-25T20:15:49.199 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:49.196+0000 7fa0157fa700 1 --2- 172.21.15.160:0/1291229742 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fa00802ff60 0x7fa008032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:15:49.199 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:49.196+0000 7fa0157fa700 1 -- 172.21.15.160:0/1291229742 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa00c0410c0 con 0x7fa0180f5410 2023-12-25T20:15:49.199 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:49.197+0000 7fa017fff700 1 --2- 172.21.15.160:0/1291229742 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fa00802ff60 0x7fa008032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:15:49.199 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:49.197+0000 7fa017fff700 1 --2- 172.21.15.160:0/1291229742 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fa00802ff60 0x7fa008032420 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7fa0040096a0 tx=0x7fa004006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:15:49.201 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:49.201+0000 7fa0157fa700 1 -- 172.21.15.160:0/1291229742 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa00c01d7f0 con 0x7fa0180f5410 2023-12-25T20:15:49.405 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:49.406+0000 7fa01ece2700 1 -- 172.21.15.160:0/1291229742 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa0180fae60 con 0x7fa0180f5410 2023-12-25T20:15:49.406 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:49.406+0000 7fa0157fa700 1 -- 172.21.15.160:0/1291229742 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa00c010230 con 0x7fa0180f5410 2023-12-25T20:15:49.406 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:15:49.406 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:15:49.408 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:49.408+0000 7fa01ece2700 1 -- 172.21.15.160:0/1291229742 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fa00802ff60 msgr2=0x7fa008032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:15:49.408 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:49.408+0000 7fa01ece2700 1 --2- 172.21.15.160:0/1291229742 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fa00802ff60 0x7fa008032420 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7fa0040096a0 tx=0x7fa004006b40).stop 2023-12-25T20:15:49.408 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:49.408+0000 7fa01ece2700 1 -- 172.21.15.160:0/1291229742 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa0180f5410 msgr2=0x7fa018105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:15:49.408 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:49.408+0000 7fa01ece2700 1 --2- 172.21.15.160:0/1291229742 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa0180f5410 0x7fa018105630 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7fa00c013510 tx=0x7fa00c00b040).stop 2023-12-25T20:15:49.408 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:49.408+0000 7fa01ece2700 1 -- 172.21.15.160:0/1291229742 shutdown_connections 2023-12-25T20:15:49.408 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:49.408+0000 7fa01ece2700 1 --2- 172.21.15.160:0/1291229742 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fa00802ff60 0x7fa008032420 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:15:49.408 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:49.408+0000 7fa01ece2700 1 --2- 172.21.15.160:0/1291229742 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa0180f5410 0x7fa018105630 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:15:49.408 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:49.408+0000 7fa01ece2700 1 -- 172.21.15.160:0/1291229742 >> 172.21.15.160:0/1291229742 conn(0x7fa0180f0b70 msgr2=0x7fa0180f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:15:49.408 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:49.408+0000 7fa01ece2700 1 -- 172.21.15.160:0/1291229742 shutdown_connections 2023-12-25T20:15:49.408 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:49.408+0000 7fa01ece2700 1 -- 172.21.15.160:0/1291229742 wait complete. 2023-12-25T20:15:49.409 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:15:50.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:15:49 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:15:48.706050+0000 mgr.smithi050.nbzhun (mgr.14162) 265 : cluster [DBG] pgmap v251: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:15:50.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:15:49 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:15:49.407111+0000 mon.smithi050 (mon.0) 313 : audit [DBG] from='client.? 172.21.15.160:0/1291229742' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:15:50.784 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:15:50.785 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:15:52.017 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:52.015+0000 7fa023388700 1 -- 172.21.15.160:0/2048853 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa01c0f6510 msgr2=0x7fa01c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:15:52.017 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:52.015+0000 7fa023388700 1 --2- 172.21.15.160:0/2048853 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa01c0f6510 0x7fa01c0f6930 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 rx=0x7fa010009fb0 tx=0x7fa010009b30).stop 2023-12-25T20:15:52.017 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:52.015+0000 7fa023388700 1 -- 172.21.15.160:0/2048853 shutdown_connections 2023-12-25T20:15:52.017 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:52.015+0000 7fa023388700 1 --2- 172.21.15.160:0/2048853 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa01c0f6510 0x7fa01c0f6930 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:15:52.017 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:52.015+0000 7fa023388700 1 -- 172.21.15.160:0/2048853 >> 172.21.15.160:0/2048853 conn(0x7fa01c0f1c70 msgr2=0x7fa01c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:15:52.017 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:52.015+0000 7fa023388700 1 -- 172.21.15.160:0/2048853 shutdown_connections 2023-12-25T20:15:52.017 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:52.015+0000 7fa023388700 1 -- 172.21.15.160:0/2048853 wait complete. 2023-12-25T20:15:52.018 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:52.016+0000 7fa023388700 1 Processor -- start 2023-12-25T20:15:52.018 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:52.016+0000 7fa023388700 1 -- start start 2023-12-25T20:15:52.018 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:52.016+0000 7fa023388700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa01c0f6510 0x7fa01c106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:15:52.018 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:52.016+0000 7fa023388700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa01c106c20 con 0x7fa01c0f6510 2023-12-25T20:15:52.018 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:52.016+0000 7fa021124700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa01c0f6510 0x7fa01c106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:15:52.018 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:52.016+0000 7fa021124700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa01c0f6510 0x7fa01c106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:50136/0 (socket says 172.21.15.160:50136) 2023-12-25T20:15:52.018 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:52.016+0000 7fa021124700 1 -- 172.21.15.160:0/4061932873 learned_addr learned my addr 172.21.15.160:0/4061932873 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:15:52.018 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:52.017+0000 7fa021124700 1 -- 172.21.15.160:0/4061932873 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa010005040 con 0x7fa01c0f6510 2023-12-25T20:15:52.018 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:52.017+0000 7fa021124700 1 --2- 172.21.15.160:0/4061932873 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa01c0f6510 0x7fa01c106730 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7fa010013510 tx=0x7fa01000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:15:52.018 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:52.017+0000 7fa00dffb700 1 -- 172.21.15.160:0/4061932873 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa010009320 con 0x7fa01c0f6510 2023-12-25T20:15:52.018 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:52.017+0000 7fa023388700 1 -- 172.21.15.160:0/4061932873 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa01c106e20 con 0x7fa01c0f6510 2023-12-25T20:15:52.018 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:52.017+0000 7fa00dffb700 1 -- 172.21.15.160:0/4061932873 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa010013f00 con 0x7fa01c0f6510 2023-12-25T20:15:52.018 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:52.017+0000 7fa00dffb700 1 -- 172.21.15.160:0/4061932873 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa010014d90 con 0x7fa01c0f6510 2023-12-25T20:15:52.019 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:52.017+0000 7fa023388700 1 -- 172.21.15.160:0/4061932873 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa01c1072b0 con 0x7fa01c0f6510 2023-12-25T20:15:52.019 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:52.018+0000 7fa00dffb700 1 -- 172.21.15.160:0/4061932873 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa010014420 con 0x7fa01c0f6510 2023-12-25T20:15:52.019 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:52.018+0000 7fa023388700 1 -- 172.21.15.160:0/4061932873 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa01c045ad0 con 0x7fa01c0f6510 2023-12-25T20:15:52.019 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:52.018+0000 7fa00dffb700 1 --2- 172.21.15.160:0/4061932873 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fa00802ff60 0x7fa008032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:15:52.020 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:52.018+0000 7fa00dffb700 1 -- 172.21.15.160:0/4061932873 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa010041110 con 0x7fa01c0f6510 2023-12-25T20:15:52.020 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:52.018+0000 7fa020923700 1 --2- 172.21.15.160:0/4061932873 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fa00802ff60 0x7fa008032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:15:52.020 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:52.019+0000 7fa020923700 1 --2- 172.21.15.160:0/4061932873 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fa00802ff60 0x7fa008032420 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7fa0180096a0 tx=0x7fa018006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:15:52.022 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:52.023+0000 7fa00dffb700 1 -- 172.21.15.160:0/4061932873 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa0100095f0 con 0x7fa01c0f6510 2023-12-25T20:15:52.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:15:51 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:15:50.706369+0000 mgr.smithi050.nbzhun (mgr.14162) 266 : cluster [DBG] pgmap v252: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:15:52.229 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:52.229+0000 7fa023388700 1 -- 172.21.15.160:0/4061932873 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa01c0fbf60 con 0x7fa01c0f6510 2023-12-25T20:15:52.230 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:52.230+0000 7fa00dffb700 1 -- 172.21.15.160:0/4061932873 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa01001db70 con 0x7fa01c0f6510 2023-12-25T20:15:52.230 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:15:52.231 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:15:52.232 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:52.231+0000 7fa023388700 1 -- 172.21.15.160:0/4061932873 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fa00802ff60 msgr2=0x7fa008032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:15:52.233 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:52.231+0000 7fa023388700 1 --2- 172.21.15.160:0/4061932873 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fa00802ff60 0x7fa008032420 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7fa0180096a0 tx=0x7fa018006b40).stop 2023-12-25T20:15:52.233 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:52.231+0000 7fa023388700 1 -- 172.21.15.160:0/4061932873 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa01c0f6510 msgr2=0x7fa01c106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:15:52.233 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:52.231+0000 7fa023388700 1 --2- 172.21.15.160:0/4061932873 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa01c0f6510 0x7fa01c106730 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7fa010013510 tx=0x7fa01000b040).stop 2023-12-25T20:15:52.233 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:52.232+0000 7fa023388700 1 -- 172.21.15.160:0/4061932873 shutdown_connections 2023-12-25T20:15:52.233 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:52.232+0000 7fa023388700 1 --2- 172.21.15.160:0/4061932873 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fa00802ff60 0x7fa008032420 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:15:52.234 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:52.232+0000 7fa023388700 1 --2- 172.21.15.160:0/4061932873 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa01c0f6510 0x7fa01c106730 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:15:52.234 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:52.232+0000 7fa023388700 1 -- 172.21.15.160:0/4061932873 >> 172.21.15.160:0/4061932873 conn(0x7fa01c0f1c70 msgr2=0x7fa01c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:15:52.234 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:52.232+0000 7fa023388700 1 -- 172.21.15.160:0/4061932873 shutdown_connections 2023-12-25T20:15:52.234 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:52.232+0000 7fa023388700 1 -- 172.21.15.160:0/4061932873 wait complete. 2023-12-25T20:15:52.234 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:15:53.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:15:52 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:15:52.230585+0000 mon.smithi050 (mon.0) 314 : audit [DBG] from='client.? 172.21.15.160:0/4061932873' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:15:53.611 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:15:53.611 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:15:54.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:15:53 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:15:52.706633+0000 mgr.smithi050.nbzhun (mgr.14162) 267 : cluster [DBG] pgmap v253: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:15:54.803 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:54.802+0000 7f1798961700 1 -- 172.21.15.160:0/1028140385 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f17940f5410 msgr2=0x7f17940f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:15:54.803 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:54.802+0000 7f1798961700 1 --2- 172.21.15.160:0/1028140385 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f17940f5410 0x7f17940f5830 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 rx=0x7f1784009fb0 tx=0x7f1784009b30).stop 2023-12-25T20:15:54.803 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:54.802+0000 7f1798961700 1 -- 172.21.15.160:0/1028140385 shutdown_connections 2023-12-25T20:15:54.803 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:54.802+0000 7f1798961700 1 --2- 172.21.15.160:0/1028140385 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f17940f5410 0x7f17940f5830 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:15:54.803 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:54.802+0000 7f1798961700 1 -- 172.21.15.160:0/1028140385 >> 172.21.15.160:0/1028140385 conn(0x7f17940f0b70 msgr2=0x7f17940f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:15:54.804 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:54.802+0000 7f1798961700 1 -- 172.21.15.160:0/1028140385 shutdown_connections 2023-12-25T20:15:54.804 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:54.802+0000 7f1798961700 1 -- 172.21.15.160:0/1028140385 wait complete. 2023-12-25T20:15:54.804 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:54.803+0000 7f1798961700 1 Processor -- start 2023-12-25T20:15:54.804 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:54.803+0000 7f1798961700 1 -- start start 2023-12-25T20:15:54.804 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:54.803+0000 7f1798961700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f17940f5410 0x7f1794109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:15:54.804 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:54.803+0000 7f1798961700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1794060240 con 0x7f17940f5410 2023-12-25T20:15:54.804 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:54.803+0000 7f179259c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f17940f5410 0x7f1794109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:15:54.804 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:54.803+0000 7f179259c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f17940f5410 0x7f1794109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:50162/0 (socket says 172.21.15.160:50162) 2023-12-25T20:15:54.804 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:54.803+0000 7f179259c700 1 -- 172.21.15.160:0/1246427981 learned_addr learned my addr 172.21.15.160:0/1246427981 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:15:54.804 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:54.803+0000 7f179259c700 1 -- 172.21.15.160:0/1246427981 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1784005040 con 0x7f17940f5410 2023-12-25T20:15:54.804 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:54.803+0000 7f179259c700 1 --2- 172.21.15.160:0/1246427981 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f17940f5410 0x7f1794109430 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7f1784013510 tx=0x7f178400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:15:54.804 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:54.804+0000 7f1782ffd700 1 -- 172.21.15.160:0/1246427981 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1784009320 con 0x7f17940f5410 2023-12-25T20:15:54.804 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:54.804+0000 7f1798961700 1 -- 172.21.15.160:0/1246427981 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1794060440 con 0x7f17940f5410 2023-12-25T20:15:54.804 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:54.804+0000 7f1782ffd700 1 -- 172.21.15.160:0/1246427981 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1784013f00 con 0x7f17940f5410 2023-12-25T20:15:54.804 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:54.804+0000 7f1782ffd700 1 -- 172.21.15.160:0/1246427981 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1784014d90 con 0x7f17940f5410 2023-12-25T20:15:54.805 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:54.804+0000 7f1798961700 1 -- 172.21.15.160:0/1246427981 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1794109ae0 con 0x7f17940f5410 2023-12-25T20:15:54.808 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:54.804+0000 7f1782ffd700 1 -- 172.21.15.160:0/1246427981 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f17840095f0 con 0x7f17940f5410 2023-12-25T20:15:54.808 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:54.804+0000 7f1798961700 1 -- 172.21.15.160:0/1246427981 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1794045ad0 con 0x7f17940f5410 2023-12-25T20:15:54.808 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:54.805+0000 7f1782ffd700 1 --2- 172.21.15.160:0/1246427981 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f177c02ff60 0x7f177c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:15:54.808 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:54.805+0000 7f1782ffd700 1 -- 172.21.15.160:0/1246427981 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f17840410a0 con 0x7f17940f5410 2023-12-25T20:15:54.808 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:54.805+0000 7f1791d9b700 1 --2- 172.21.15.160:0/1246427981 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f177c02ff60 0x7f177c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:15:54.808 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:54.805+0000 7f1791d9b700 1 --2- 172.21.15.160:0/1246427981 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f177c02ff60 0x7f177c032420 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f17880096a0 tx=0x7f1788006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:15:54.809 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:54.809+0000 7f1782ffd700 1 -- 172.21.15.160:0/1246427981 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f178401d7f0 con 0x7f17940f5410 2023-12-25T20:15:55.016 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:55.016+0000 7f1798961700 1 -- 172.21.15.160:0/1246427981 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f17940fae60 con 0x7f17940f5410 2023-12-25T20:15:55.017 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:55.017+0000 7f1782ffd700 1 -- 172.21.15.160:0/1246427981 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1784010230 con 0x7f17940f5410 2023-12-25T20:15:55.017 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:15:55.017 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:15:55.018 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:55.019+0000 7f1798961700 1 -- 172.21.15.160:0/1246427981 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f177c02ff60 msgr2=0x7f177c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:15:55.018 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:55.019+0000 7f1798961700 1 --2- 172.21.15.160:0/1246427981 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f177c02ff60 0x7f177c032420 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f17880096a0 tx=0x7f1788006b40).stop 2023-12-25T20:15:55.018 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:55.019+0000 7f1798961700 1 -- 172.21.15.160:0/1246427981 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f17940f5410 msgr2=0x7f1794109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:15:55.019 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:55.019+0000 7f1798961700 1 --2- 172.21.15.160:0/1246427981 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f17940f5410 0x7f1794109430 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7f1784013510 tx=0x7f178400b040).stop 2023-12-25T20:15:55.019 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:55.019+0000 7f1798961700 1 -- 172.21.15.160:0/1246427981 shutdown_connections 2023-12-25T20:15:55.019 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:55.019+0000 7f1798961700 1 --2- 172.21.15.160:0/1246427981 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f177c02ff60 0x7f177c032420 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:15:55.019 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:55.019+0000 7f1798961700 1 --2- 172.21.15.160:0/1246427981 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f17940f5410 0x7f1794109430 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:15:55.019 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:55.019+0000 7f1798961700 1 -- 172.21.15.160:0/1246427981 >> 172.21.15.160:0/1246427981 conn(0x7f17940f0b70 msgr2=0x7f17940f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:15:55.019 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:55.019+0000 7f1798961700 1 -- 172.21.15.160:0/1246427981 shutdown_connections 2023-12-25T20:15:55.019 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:55.019+0000 7f1798961700 1 -- 172.21.15.160:0/1246427981 wait complete. 2023-12-25T20:15:55.019 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:15:56.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:15:55 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:15:54.706941+0000 mgr.smithi050.nbzhun (mgr.14162) 268 : cluster [DBG] pgmap v254: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:15:56.219 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:15:55 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:15:55.017876+0000 mon.smithi050 (mon.0) 315 : audit [DBG] from='client.? 172.21.15.160:0/1246427981' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:15:56.401 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:15:56.401 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:15:57.627 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:57.628+0000 7f3cd64d8700 1 -- 172.21.15.160:0/1323577608 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3cd00f6510 msgr2=0x7f3cd00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:15:57.627 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:57.628+0000 7f3cd64d8700 1 --2- 172.21.15.160:0/1323577608 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3cd00f6510 0x7f3cd00f6930 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 rx=0x7f3cc0009fb0 tx=0x7f3cc0009b30).stop 2023-12-25T20:15:57.628 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:57.628+0000 7f3cd64d8700 1 -- 172.21.15.160:0/1323577608 shutdown_connections 2023-12-25T20:15:57.628 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:57.628+0000 7f3cd64d8700 1 --2- 172.21.15.160:0/1323577608 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3cd00f6510 0x7f3cd00f6930 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:15:57.628 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:57.628+0000 7f3cd64d8700 1 -- 172.21.15.160:0/1323577608 >> 172.21.15.160:0/1323577608 conn(0x7f3cd00f1c70 msgr2=0x7f3cd00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:15:57.628 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:57.628+0000 7f3cd64d8700 1 -- 172.21.15.160:0/1323577608 shutdown_connections 2023-12-25T20:15:57.628 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:57.628+0000 7f3cd64d8700 1 -- 172.21.15.160:0/1323577608 wait complete. 2023-12-25T20:15:57.629 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:57.628+0000 7f3cd64d8700 1 Processor -- start 2023-12-25T20:15:57.629 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:57.629+0000 7f3cd64d8700 1 -- start start 2023-12-25T20:15:57.629 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:57.629+0000 7f3cd64d8700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3cd00f6510 0x7f3cd0106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:15:57.629 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:57.629+0000 7f3cd64d8700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cd0106c20 con 0x7f3cd00f6510 2023-12-25T20:15:57.629 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:57.629+0000 7f3ccffff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3cd00f6510 0x7f3cd0106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:15:57.630 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:57.629+0000 7f3ccffff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3cd00f6510 0x7f3cd0106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:35030/0 (socket says 172.21.15.160:35030) 2023-12-25T20:15:57.630 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:57.629+0000 7f3ccffff700 1 -- 172.21.15.160:0/3074682795 learned_addr learned my addr 172.21.15.160:0/3074682795 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:15:57.630 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:57.630+0000 7f3ccffff700 1 -- 172.21.15.160:0/3074682795 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cc0005040 con 0x7f3cd00f6510 2023-12-25T20:15:57.630 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:57.630+0000 7f3ccffff700 1 --2- 172.21.15.160:0/3074682795 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3cd00f6510 0x7f3cd0106730 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7f3cc0014000 tx=0x7f3cc000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:15:57.630 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:57.630+0000 7f3cccff9700 1 -- 172.21.15.160:0/3074682795 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3cc00095f0 con 0x7f3cd00f6510 2023-12-25T20:15:57.630 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:57.630+0000 7f3cd64d8700 1 -- 172.21.15.160:0/3074682795 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cd0106e20 con 0x7f3cd00f6510 2023-12-25T20:15:57.630 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:57.630+0000 7f3cccff9700 1 -- 172.21.15.160:0/3074682795 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3cc0007e70 con 0x7f3cd00f6510 2023-12-25T20:15:57.630 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:57.630+0000 7f3cccff9700 1 -- 172.21.15.160:0/3074682795 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3cc0014ea0 con 0x7f3cd00f6510 2023-12-25T20:15:57.630 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:57.630+0000 7f3cd64d8700 1 -- 172.21.15.160:0/3074682795 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cd01072b0 con 0x7f3cd00f6510 2023-12-25T20:15:57.631 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:57.631+0000 7f3cccff9700 1 -- 172.21.15.160:0/3074682795 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3cc001d4e0 con 0x7f3cd00f6510 2023-12-25T20:15:57.631 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:57.631+0000 7f3cccff9700 1 --2- 172.21.15.160:0/3074682795 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f3cb802ff60 0x7f3cb8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:15:57.631 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:57.631+0000 7f3cccff9700 1 -- 172.21.15.160:0/3074682795 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3cc0040960 con 0x7f3cd00f6510 2023-12-25T20:15:57.631 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:57.631+0000 7f3cd64d8700 1 -- 172.21.15.160:0/3074682795 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cd0045ad0 con 0x7f3cd00f6510 2023-12-25T20:15:57.631 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:57.631+0000 7f3ccf7fe700 1 --2- 172.21.15.160:0/3074682795 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f3cb802ff60 0x7f3cb8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:15:57.631 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:57.632+0000 7f3ccf7fe700 1 --2- 172.21.15.160:0/3074682795 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f3cb802ff60 0x7f3cb8032420 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f3cc40096a0 tx=0x7f3cc4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:15:57.636 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:57.636+0000 7f3cccff9700 1 -- 172.21.15.160:0/3074682795 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3cc000f8f0 con 0x7f3cd00f6510 2023-12-25T20:15:57.838 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:57.838+0000 7f3cd64d8700 1 -- 172.21.15.160:0/3074682795 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3cd00fc050 con 0x7f3cd00f6510 2023-12-25T20:15:57.838 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:57.839+0000 7f3cccff9700 1 -- 172.21.15.160:0/3074682795 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3cc0012330 con 0x7f3cd00f6510 2023-12-25T20:15:57.839 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:15:57.839 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:15:57.840 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:57.840+0000 7f3cd64d8700 1 -- 172.21.15.160:0/3074682795 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f3cb802ff60 msgr2=0x7f3cb8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:15:57.840 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:57.840+0000 7f3cd64d8700 1 --2- 172.21.15.160:0/3074682795 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f3cb802ff60 0x7f3cb8032420 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f3cc40096a0 tx=0x7f3cc4006b40).stop 2023-12-25T20:15:57.840 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:57.840+0000 7f3cd64d8700 1 -- 172.21.15.160:0/3074682795 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3cd00f6510 msgr2=0x7f3cd0106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:15:57.840 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:57.840+0000 7f3cd64d8700 1 --2- 172.21.15.160:0/3074682795 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3cd00f6510 0x7f3cd0106730 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7f3cc0014000 tx=0x7f3cc000b040).stop 2023-12-25T20:15:57.840 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:57.840+0000 7f3cd64d8700 1 -- 172.21.15.160:0/3074682795 shutdown_connections 2023-12-25T20:15:57.840 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:57.840+0000 7f3cd64d8700 1 --2- 172.21.15.160:0/3074682795 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f3cb802ff60 0x7f3cb8032420 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:15:57.841 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:57.840+0000 7f3cd64d8700 1 --2- 172.21.15.160:0/3074682795 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3cd00f6510 0x7f3cd0106730 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:15:57.841 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:57.840+0000 7f3cd64d8700 1 -- 172.21.15.160:0/3074682795 >> 172.21.15.160:0/3074682795 conn(0x7f3cd00f1c70 msgr2=0x7f3cd00f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:15:57.841 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:57.841+0000 7f3cd64d8700 1 -- 172.21.15.160:0/3074682795 shutdown_connections 2023-12-25T20:15:57.841 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:15:57.841+0000 7f3cd64d8700 1 -- 172.21.15.160:0/3074682795 wait complete. 2023-12-25T20:15:57.841 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:15:58.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:15:57 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:15:56.707341+0000 mgr.smithi050.nbzhun (mgr.14162) 269 : cluster [DBG] pgmap v255: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:15:58.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:15:57 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:15:57.839688+0000 mon.smithi050 (mon.0) 316 : audit [DBG] from='client.? 172.21.15.160:0/3074682795' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:15:59.220 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:15:59.221 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:16:00.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:15:59 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:15:58.707627+0000 mgr.smithi050.nbzhun (mgr.14162) 270 : cluster [DBG] pgmap v256: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:16:00.396 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:00.395+0000 7f23097c0700 1 -- 172.21.15.160:0/2668093586 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2304060280 msgr2=0x7f23040f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:16:00.396 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:00.395+0000 7f23097c0700 1 --2- 172.21.15.160:0/2668093586 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2304060280 0x7f23040f90f0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 rx=0x7f22f8009fb0 tx=0x7f22f8009b30).stop 2023-12-25T20:16:00.397 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:00.395+0000 7f23097c0700 1 -- 172.21.15.160:0/2668093586 shutdown_connections 2023-12-25T20:16:00.397 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:00.395+0000 7f23097c0700 1 --2- 172.21.15.160:0/2668093586 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2304060280 0x7f23040f90f0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:16:00.397 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:00.395+0000 7f23097c0700 1 -- 172.21.15.160:0/2668093586 >> 172.21.15.160:0/2668093586 conn(0x7f23040f0b70 msgr2=0x7f23040f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:16:00.397 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:00.395+0000 7f23097c0700 1 -- 172.21.15.160:0/2668093586 shutdown_connections 2023-12-25T20:16:00.397 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:00.395+0000 7f23097c0700 1 -- 172.21.15.160:0/2668093586 wait complete. 2023-12-25T20:16:00.397 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:00.396+0000 7f23097c0700 1 Processor -- start 2023-12-25T20:16:00.397 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:00.396+0000 7f23097c0700 1 -- start start 2023-12-25T20:16:00.397 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:00.396+0000 7f23097c0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2304060280 0x7f23041066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:16:00.397 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:00.396+0000 7f23097c0700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2304106bd0 con 0x7f2304060280 2023-12-25T20:16:00.397 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:00.396+0000 7f2302ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2304060280 0x7f23041066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:16:00.397 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:00.396+0000 7f2302ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2304060280 0x7f23041066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:35058/0 (socket says 172.21.15.160:35058) 2023-12-25T20:16:00.397 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:00.396+0000 7f2302ffd700 1 -- 172.21.15.160:0/396899973 learned_addr learned my addr 172.21.15.160:0/396899973 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:16:00.397 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:00.396+0000 7f2302ffd700 1 -- 172.21.15.160:0/396899973 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22f8005040 con 0x7f2304060280 2023-12-25T20:16:00.398 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:00.397+0000 7f2302ffd700 1 --2- 172.21.15.160:0/396899973 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2304060280 0x7f23041066e0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7f22f8009830 tx=0x7f22f800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:16:00.398 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:00.397+0000 7f22eb7fe700 1 -- 172.21.15.160:0/396899973 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f22f8009320 con 0x7f2304060280 2023-12-25T20:16:00.398 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:00.397+0000 7f23097c0700 1 -- 172.21.15.160:0/396899973 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2304106dd0 con 0x7f2304060280 2023-12-25T20:16:00.398 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:00.397+0000 7f22eb7fe700 1 -- 172.21.15.160:0/396899973 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f22f80136c0 con 0x7f2304060280 2023-12-25T20:16:00.398 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:00.397+0000 7f22eb7fe700 1 -- 172.21.15.160:0/396899973 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f22f801ee90 con 0x7f2304060280 2023-12-25T20:16:00.398 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:00.397+0000 7f23097c0700 1 -- 172.21.15.160:0/396899973 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2304107260 con 0x7f2304060280 2023-12-25T20:16:00.399 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:00.398+0000 7f22eb7fe700 1 -- 172.21.15.160:0/396899973 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f22f801e420 con 0x7f2304060280 2023-12-25T20:16:00.399 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:00.398+0000 7f23097c0700 1 -- 172.21.15.160:0/396899973 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2304045ad0 con 0x7f2304060280 2023-12-25T20:16:00.399 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:00.398+0000 7f22eb7fe700 1 --2- 172.21.15.160:0/396899973 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f22ec02ff10 0x7f22ec0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:16:00.399 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:00.398+0000 7f22eb7fe700 1 -- 172.21.15.160:0/396899973 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f22f8040f30 con 0x7f2304060280 2023-12-25T20:16:00.399 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:00.398+0000 7f23027fc700 1 --2- 172.21.15.160:0/396899973 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f22ec02ff10 0x7f22ec0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:16:00.399 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:00.398+0000 7f23027fc700 1 --2- 172.21.15.160:0/396899973 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f22ec02ff10 0x7f22ec0323d0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f22f40096a0 tx=0x7f22f4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:16:00.402 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:00.403+0000 7f22eb7fe700 1 -- 172.21.15.160:0/396899973 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f22f806d2b0 con 0x7f2304060280 2023-12-25T20:16:00.608 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:00.608+0000 7f23097c0700 1 -- 172.21.15.160:0/396899973 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2304108110 con 0x7f2304060280 2023-12-25T20:16:00.609 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:00.609+0000 7f22eb7fe700 1 -- 172.21.15.160:0/396899973 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f22f801c790 con 0x7f2304060280 2023-12-25T20:16:00.609 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:16:00.609 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:16:00.611 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:00.611+0000 7f23097c0700 1 -- 172.21.15.160:0/396899973 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f22ec02ff10 msgr2=0x7f22ec0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:16:00.611 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:00.611+0000 7f23097c0700 1 --2- 172.21.15.160:0/396899973 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f22ec02ff10 0x7f22ec0323d0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f22f40096a0 tx=0x7f22f4006b40).stop 2023-12-25T20:16:00.611 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:00.611+0000 7f23097c0700 1 -- 172.21.15.160:0/396899973 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2304060280 msgr2=0x7f23041066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:16:00.611 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:00.611+0000 7f23097c0700 1 --2- 172.21.15.160:0/396899973 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2304060280 0x7f23041066e0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7f22f8009830 tx=0x7f22f800b040).stop 2023-12-25T20:16:00.611 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:00.611+0000 7f23097c0700 1 -- 172.21.15.160:0/396899973 shutdown_connections 2023-12-25T20:16:00.611 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:00.611+0000 7f23097c0700 1 --2- 172.21.15.160:0/396899973 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f22ec02ff10 0x7f22ec0323d0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:16:00.611 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:00.611+0000 7f23097c0700 1 --2- 172.21.15.160:0/396899973 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2304060280 0x7f23041066e0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:16:00.611 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:00.611+0000 7f23097c0700 1 -- 172.21.15.160:0/396899973 >> 172.21.15.160:0/396899973 conn(0x7f23040f0b70 msgr2=0x7f23040f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:16:00.611 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:00.611+0000 7f23097c0700 1 -- 172.21.15.160:0/396899973 shutdown_connections 2023-12-25T20:16:00.612 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:00.611+0000 7f23097c0700 1 -- 172.21.15.160:0/396899973 wait complete. 2023-12-25T20:16:00.612 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:16:01.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:16:00 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:16:00.610172+0000 mon.smithi050 (mon.0) 317 : audit [DBG] from='client.? 172.21.15.160:0/396899973' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:16:02.033 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:16:02.033 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:16:02.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:16:01 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:16:00.707961+0000 mgr.smithi050.nbzhun (mgr.14162) 271 : cluster [DBG] pgmap v257: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:16:03.210 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:03.209+0000 7f2180e79700 1 -- 172.21.15.160:0/3514156124 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f217c0f5410 msgr2=0x7f217c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:16:03.210 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:03.209+0000 7f2180e79700 1 --2- 172.21.15.160:0/3514156124 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f217c0f5410 0x7f217c0f5830 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 rx=0x7f2164009fb0 tx=0x7f2164009b30).stop 2023-12-25T20:16:03.210 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:03.209+0000 7f2180e79700 1 -- 172.21.15.160:0/3514156124 shutdown_connections 2023-12-25T20:16:03.210 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:03.209+0000 7f2180e79700 1 --2- 172.21.15.160:0/3514156124 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f217c0f5410 0x7f217c0f5830 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:16:03.211 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:03.209+0000 7f2180e79700 1 -- 172.21.15.160:0/3514156124 >> 172.21.15.160:0/3514156124 conn(0x7f217c0f0b70 msgr2=0x7f217c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:16:03.211 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:03.209+0000 7f2180e79700 1 -- 172.21.15.160:0/3514156124 shutdown_connections 2023-12-25T20:16:03.211 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:03.209+0000 7f2180e79700 1 -- 172.21.15.160:0/3514156124 wait complete. 2023-12-25T20:16:03.211 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:03.210+0000 7f2180e79700 1 Processor -- start 2023-12-25T20:16:03.211 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:03.210+0000 7f2180e79700 1 -- start start 2023-12-25T20:16:03.211 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:03.210+0000 7f2180e79700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f217c0f5410 0x7f217c105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:16:03.211 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:03.210+0000 7f2180e79700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f217c105b20 con 0x7f217c0f5410 2023-12-25T20:16:03.211 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:03.210+0000 7f217a59c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f217c0f5410 0x7f217c105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:16:03.211 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:03.210+0000 7f217a59c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f217c0f5410 0x7f217c105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:35082/0 (socket says 172.21.15.160:35082) 2023-12-25T20:16:03.211 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:03.210+0000 7f217a59c700 1 -- 172.21.15.160:0/491739368 learned_addr learned my addr 172.21.15.160:0/491739368 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:16:03.213 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:03.211+0000 7f217a59c700 1 -- 172.21.15.160:0/491739368 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2164005040 con 0x7f217c0f5410 2023-12-25T20:16:03.213 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:03.211+0000 7f217a59c700 1 --2- 172.21.15.160:0/491739368 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f217c0f5410 0x7f217c105630 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7f2164013510 tx=0x7f216400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:16:03.213 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:03.211+0000 7f2172ffd700 1 -- 172.21.15.160:0/491739368 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2164009320 con 0x7f217c0f5410 2023-12-25T20:16:03.213 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:03.211+0000 7f2180e79700 1 -- 172.21.15.160:0/491739368 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f217c105d20 con 0x7f217c0f5410 2023-12-25T20:16:03.213 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:03.211+0000 7f2172ffd700 1 -- 172.21.15.160:0/491739368 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2164013f00 con 0x7f217c0f5410 2023-12-25T20:16:03.213 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:03.211+0000 7f2172ffd700 1 -- 172.21.15.160:0/491739368 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2164014d90 con 0x7f217c0f5410 2023-12-25T20:16:03.213 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:03.211+0000 7f2180e79700 1 -- 172.21.15.160:0/491739368 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f217c1061b0 con 0x7f217c0f5410 2023-12-25T20:16:03.213 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:03.212+0000 7f2180e79700 1 -- 172.21.15.160:0/491739368 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f217c045ad0 con 0x7f217c0f5410 2023-12-25T20:16:03.213 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:03.212+0000 7f2172ffd700 1 -- 172.21.15.160:0/491739368 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f21640095f0 con 0x7f217c0f5410 2023-12-25T20:16:03.213 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:03.212+0000 7f2172ffd700 1 --2- 172.21.15.160:0/491739368 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f216802ff60 0x7f2168032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:16:03.213 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:03.212+0000 7f2172ffd700 1 -- 172.21.15.160:0/491739368 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f21640410c0 con 0x7f217c0f5410 2023-12-25T20:16:03.214 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:03.212+0000 7f2179d9b700 1 --2- 172.21.15.160:0/491739368 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f216802ff60 0x7f2168032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:16:03.214 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:03.213+0000 7f2179d9b700 1 --2- 172.21.15.160:0/491739368 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f216802ff60 0x7f2168032420 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f216c0096a0 tx=0x7f216c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:16:03.216 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:03.217+0000 7f2172ffd700 1 -- 172.21.15.160:0/491739368 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2164014420 con 0x7f217c0f5410 2023-12-25T20:16:03.423 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:03.423+0000 7f2180e79700 1 -- 172.21.15.160:0/491739368 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f217c0fae60 con 0x7f217c0f5410 2023-12-25T20:16:03.423 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:03.424+0000 7f2172ffd700 1 -- 172.21.15.160:0/491739368 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f216401d7f0 con 0x7f217c0f5410 2023-12-25T20:16:03.423 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:16:03.424 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:16:03.425 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:03.425+0000 7f2180e79700 1 -- 172.21.15.160:0/491739368 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f216802ff60 msgr2=0x7f2168032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:16:03.425 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:03.425+0000 7f2180e79700 1 --2- 172.21.15.160:0/491739368 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f216802ff60 0x7f2168032420 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f216c0096a0 tx=0x7f216c006b40).stop 2023-12-25T20:16:03.425 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:03.425+0000 7f2180e79700 1 -- 172.21.15.160:0/491739368 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f217c0f5410 msgr2=0x7f217c105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:16:03.425 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:03.425+0000 7f2180e79700 1 --2- 172.21.15.160:0/491739368 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f217c0f5410 0x7f217c105630 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7f2164013510 tx=0x7f216400b040).stop 2023-12-25T20:16:03.425 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:03.425+0000 7f2180e79700 1 -- 172.21.15.160:0/491739368 shutdown_connections 2023-12-25T20:16:03.425 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:03.425+0000 7f2180e79700 1 --2- 172.21.15.160:0/491739368 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f216802ff60 0x7f2168032420 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:16:03.425 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:03.425+0000 7f2180e79700 1 --2- 172.21.15.160:0/491739368 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f217c0f5410 0x7f217c105630 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:16:03.425 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:03.425+0000 7f2180e79700 1 -- 172.21.15.160:0/491739368 >> 172.21.15.160:0/491739368 conn(0x7f217c0f0b70 msgr2=0x7f217c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:16:03.426 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:03.425+0000 7f2180e79700 1 -- 172.21.15.160:0/491739368 shutdown_connections 2023-12-25T20:16:03.426 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:03.425+0000 7f2180e79700 1 -- 172.21.15.160:0/491739368 wait complete. 2023-12-25T20:16:03.426 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:16:04.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:16:03 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:16:02.708350+0000 mgr.smithi050.nbzhun (mgr.14162) 272 : cluster [DBG] pgmap v258: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:16:04.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:16:03 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:16:03.424342+0000 mon.smithi050 (mon.0) 318 : audit [DBG] from='client.? 172.21.15.160:0/491739368' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:16:04.812 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:16:04.813 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:16:05.993 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:05.992+0000 7fe61835e700 1 -- 172.21.15.160:0/954964663 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe6100f6510 msgr2=0x7fe6100f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:16:05.994 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:05.992+0000 7fe61835e700 1 --2- 172.21.15.160:0/954964663 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe6100f6510 0x7fe6100f6930 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 rx=0x7fe600009fb0 tx=0x7fe600009b30).stop 2023-12-25T20:16:05.994 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:05.992+0000 7fe61835e700 1 -- 172.21.15.160:0/954964663 shutdown_connections 2023-12-25T20:16:05.994 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:05.992+0000 7fe61835e700 1 --2- 172.21.15.160:0/954964663 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe6100f6510 0x7fe6100f6930 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:16:05.994 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:05.992+0000 7fe61835e700 1 -- 172.21.15.160:0/954964663 >> 172.21.15.160:0/954964663 conn(0x7fe6100f1c70 msgr2=0x7fe6100f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:16:05.994 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:05.992+0000 7fe61835e700 1 -- 172.21.15.160:0/954964663 shutdown_connections 2023-12-25T20:16:05.994 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:05.992+0000 7fe61835e700 1 -- 172.21.15.160:0/954964663 wait complete. 2023-12-25T20:16:05.994 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:05.992+0000 7fe61835e700 1 Processor -- start 2023-12-25T20:16:05.994 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:05.992+0000 7fe61835e700 1 -- start start 2023-12-25T20:16:05.994 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:05.993+0000 7fe61835e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe6100f6510 0x7fe610106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:16:05.994 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:05.993+0000 7fe61835e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe610106c20 con 0x7fe6100f6510 2023-12-25T20:16:05.994 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:05.993+0000 7fe6160fa700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe6100f6510 0x7fe610106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:16:05.994 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:05.993+0000 7fe6160fa700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe6100f6510 0x7fe610106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:35102/0 (socket says 172.21.15.160:35102) 2023-12-25T20:16:05.994 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:05.993+0000 7fe6160fa700 1 -- 172.21.15.160:0/3302503518 learned_addr learned my addr 172.21.15.160:0/3302503518 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:16:05.996 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:05.993+0000 7fe6160fa700 1 -- 172.21.15.160:0/3302503518 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe600005040 con 0x7fe6100f6510 2023-12-25T20:16:05.996 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:05.993+0000 7fe6160fa700 1 --2- 172.21.15.160:0/3302503518 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe6100f6510 0x7fe610106730 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7fe600013510 tx=0x7fe60000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:16:05.996 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:05.994+0000 7fe606ffd700 1 -- 172.21.15.160:0/3302503518 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe600009320 con 0x7fe6100f6510 2023-12-25T20:16:05.996 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:05.994+0000 7fe61835e700 1 -- 172.21.15.160:0/3302503518 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe610106e20 con 0x7fe6100f6510 2023-12-25T20:16:05.996 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:05.994+0000 7fe606ffd700 1 -- 172.21.15.160:0/3302503518 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe600013f00 con 0x7fe6100f6510 2023-12-25T20:16:05.996 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:05.994+0000 7fe606ffd700 1 -- 172.21.15.160:0/3302503518 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe600014d90 con 0x7fe6100f6510 2023-12-25T20:16:05.996 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:05.994+0000 7fe61835e700 1 -- 172.21.15.160:0/3302503518 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6101072b0 con 0x7fe6100f6510 2023-12-25T20:16:05.996 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:05.994+0000 7fe606ffd700 1 -- 172.21.15.160:0/3302503518 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe600014420 con 0x7fe6100f6510 2023-12-25T20:16:05.996 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:05.994+0000 7fe61835e700 1 -- 172.21.15.160:0/3302503518 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe610045ad0 con 0x7fe6100f6510 2023-12-25T20:16:05.996 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:05.995+0000 7fe606ffd700 1 --2- 172.21.15.160:0/3302503518 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fe5fc02ffb0 0x7fe5fc032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:16:05.996 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:05.995+0000 7fe606ffd700 1 -- 172.21.15.160:0/3302503518 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe600041090 con 0x7fe6100f6510 2023-12-25T20:16:05.996 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:05.995+0000 7fe6158f9700 1 --2- 172.21.15.160:0/3302503518 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fe5fc02ffb0 0x7fe5fc032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:16:05.997 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:05.995+0000 7fe6158f9700 1 --2- 172.21.15.160:0/3302503518 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fe5fc02ffb0 0x7fe5fc032470 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7fe60c0096a0 tx=0x7fe60c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:16:06.000 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:05.999+0000 7fe606ffd700 1 -- 172.21.15.160:0/3302503518 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe60006d2b0 con 0x7fe6100f6510 2023-12-25T20:16:06.205 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:06.205+0000 7fe61835e700 1 -- 172.21.15.160:0/3302503518 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe6100fbf60 con 0x7fe6100f6510 2023-12-25T20:16:06.206 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:06.205+0000 7fe606ffd700 1 -- 172.21.15.160:0/3302503518 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe6000095f0 con 0x7fe6100f6510 2023-12-25T20:16:06.206 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:16:06.206 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:16:06.208 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:06.207+0000 7fe61835e700 1 -- 172.21.15.160:0/3302503518 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fe5fc02ffb0 msgr2=0x7fe5fc032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:16:06.208 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:06.207+0000 7fe61835e700 1 --2- 172.21.15.160:0/3302503518 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fe5fc02ffb0 0x7fe5fc032470 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7fe60c0096a0 tx=0x7fe60c006b40).stop 2023-12-25T20:16:06.208 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:06.207+0000 7fe61835e700 1 -- 172.21.15.160:0/3302503518 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe6100f6510 msgr2=0x7fe610106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:16:06.208 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:06.207+0000 7fe61835e700 1 --2- 172.21.15.160:0/3302503518 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe6100f6510 0x7fe610106730 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7fe600013510 tx=0x7fe60000b040).stop 2023-12-25T20:16:06.208 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:06.207+0000 7fe61835e700 1 -- 172.21.15.160:0/3302503518 shutdown_connections 2023-12-25T20:16:06.208 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:06.207+0000 7fe61835e700 1 --2- 172.21.15.160:0/3302503518 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fe5fc02ffb0 0x7fe5fc032470 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:16:06.208 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:06.207+0000 7fe61835e700 1 --2- 172.21.15.160:0/3302503518 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe6100f6510 0x7fe610106730 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:16:06.208 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:06.207+0000 7fe61835e700 1 -- 172.21.15.160:0/3302503518 >> 172.21.15.160:0/3302503518 conn(0x7fe6100f1c70 msgr2=0x7fe6100f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:16:06.208 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:06.207+0000 7fe61835e700 1 -- 172.21.15.160:0/3302503518 shutdown_connections 2023-12-25T20:16:06.208 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:06.207+0000 7fe61835e700 1 -- 172.21.15.160:0/3302503518 wait complete. 2023-12-25T20:16:06.209 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:16:06.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:16:05 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:16:04.708673+0000 mgr.smithi050.nbzhun (mgr.14162) 273 : cluster [DBG] pgmap v259: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:16:07.218 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:16:06 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:16:06.207169+0000 mon.smithi050 (mon.0) 319 : audit [DBG] from='client.? 172.21.15.160:0/3302503518' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:16:07.611 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:16:07.612 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:16:08.468 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:16:07 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:16:06.708934+0000 mgr.smithi050.nbzhun (mgr.14162) 274 : cluster [DBG] pgmap v260: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:16:08.799 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:08.798+0000 7fca9912b700 1 -- 172.21.15.160:0/4121781594 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fca94060140 msgr2=0x7fca940f8fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:16:08.800 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:08.798+0000 7fca9912b700 1 --2- 172.21.15.160:0/4121781594 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fca94060140 0x7fca940f8fc0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 rx=0x7fca84009fb0 tx=0x7fca84009b30).stop 2023-12-25T20:16:08.800 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:08.798+0000 7fca9912b700 1 -- 172.21.15.160:0/4121781594 shutdown_connections 2023-12-25T20:16:08.800 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:08.798+0000 7fca9912b700 1 --2- 172.21.15.160:0/4121781594 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fca94060140 0x7fca940f8fc0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:16:08.800 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:08.798+0000 7fca9912b700 1 -- 172.21.15.160:0/4121781594 >> 172.21.15.160:0/4121781594 conn(0x7fca940f09a0 msgr2=0x7fca940f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:16:08.800 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:08.798+0000 7fca9912b700 1 -- 172.21.15.160:0/4121781594 shutdown_connections 2023-12-25T20:16:08.800 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:08.798+0000 7fca9912b700 1 -- 172.21.15.160:0/4121781594 wait complete. 2023-12-25T20:16:08.800 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:08.798+0000 7fca9912b700 1 Processor -- start 2023-12-25T20:16:08.800 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:08.799+0000 7fca9912b700 1 -- start start 2023-12-25T20:16:08.800 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:08.799+0000 7fca9912b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fca94060140 0x7fca94106520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:16:08.800 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:08.799+0000 7fca9912b700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca94106a10 con 0x7fca94060140 2023-12-25T20:16:08.800 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:08.799+0000 7fca93fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fca94060140 0x7fca94106520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:16:08.800 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:08.799+0000 7fca93fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fca94060140 0x7fca94106520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:42296/0 (socket says 172.21.15.160:42296) 2023-12-25T20:16:08.800 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:08.799+0000 7fca93fff700 1 -- 172.21.15.160:0/2806943791 learned_addr learned my addr 172.21.15.160:0/2806943791 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:16:08.802 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:08.799+0000 7fca93fff700 1 -- 172.21.15.160:0/2806943791 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca84005040 con 0x7fca94060140 2023-12-25T20:16:08.802 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:08.799+0000 7fca93fff700 1 --2- 172.21.15.160:0/2806943791 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fca94060140 0x7fca94106520 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7fca84013510 tx=0x7fca8400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:16:08.802 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:08.800+0000 7fca90ff9700 1 -- 172.21.15.160:0/2806943791 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fca84009320 con 0x7fca94060140 2023-12-25T20:16:08.802 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:08.800+0000 7fca9912b700 1 -- 172.21.15.160:0/2806943791 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca94106c10 con 0x7fca94060140 2023-12-25T20:16:08.802 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:08.800+0000 7fca90ff9700 1 -- 172.21.15.160:0/2806943791 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fca84013f00 con 0x7fca94060140 2023-12-25T20:16:08.802 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:08.800+0000 7fca90ff9700 1 -- 172.21.15.160:0/2806943791 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fca84014d90 con 0x7fca94060140 2023-12-25T20:16:08.802 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:08.800+0000 7fca9912b700 1 -- 172.21.15.160:0/2806943791 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca941070a0 con 0x7fca94060140 2023-12-25T20:16:08.802 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:08.800+0000 7fca90ff9700 1 -- 172.21.15.160:0/2806943791 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fca840095f0 con 0x7fca94060140 2023-12-25T20:16:08.802 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:08.800+0000 7fca9912b700 1 -- 172.21.15.160:0/2806943791 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca94045ad0 con 0x7fca94060140 2023-12-25T20:16:08.803 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:08.801+0000 7fca90ff9700 1 --2- 172.21.15.160:0/2806943791 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fca7c02ff60 0x7fca7c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:16:08.803 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:08.801+0000 7fca90ff9700 1 -- 172.21.15.160:0/2806943791 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fca84041000 con 0x7fca94060140 2023-12-25T20:16:08.803 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:08.801+0000 7fca937fe700 1 --2- 172.21.15.160:0/2806943791 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fca7c02ff60 0x7fca7c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:16:08.803 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:08.801+0000 7fca937fe700 1 --2- 172.21.15.160:0/2806943791 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fca7c02ff60 0x7fca7c032420 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7fca880096a0 tx=0x7fca88006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:16:08.806 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:08.805+0000 7fca90ff9700 1 -- 172.21.15.160:0/2806943791 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fca84014420 con 0x7fca94060140 2023-12-25T20:16:09.010 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:09.010+0000 7fca9912b700 1 -- 172.21.15.160:0/2806943791 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fca94107f50 con 0x7fca94060140 2023-12-25T20:16:09.011 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:09.010+0000 7fca90ff9700 1 -- 172.21.15.160:0/2806943791 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fca84010230 con 0x7fca94060140 2023-12-25T20:16:09.011 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:16:09.011 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:16:09.012 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:09.012+0000 7fca9912b700 1 -- 172.21.15.160:0/2806943791 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fca7c02ff60 msgr2=0x7fca7c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:16:09.012 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:09.012+0000 7fca9912b700 1 --2- 172.21.15.160:0/2806943791 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fca7c02ff60 0x7fca7c032420 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7fca880096a0 tx=0x7fca88006b40).stop 2023-12-25T20:16:09.013 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:09.012+0000 7fca9912b700 1 -- 172.21.15.160:0/2806943791 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fca94060140 msgr2=0x7fca94106520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:16:09.013 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:09.012+0000 7fca9912b700 1 --2- 172.21.15.160:0/2806943791 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fca94060140 0x7fca94106520 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7fca84013510 tx=0x7fca8400b040).stop 2023-12-25T20:16:09.013 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:09.012+0000 7fca9912b700 1 -- 172.21.15.160:0/2806943791 shutdown_connections 2023-12-25T20:16:09.013 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:09.012+0000 7fca9912b700 1 --2- 172.21.15.160:0/2806943791 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fca7c02ff60 0x7fca7c032420 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:16:09.013 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:09.012+0000 7fca9912b700 1 --2- 172.21.15.160:0/2806943791 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fca94060140 0x7fca94106520 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:16:09.013 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:09.012+0000 7fca9912b700 1 -- 172.21.15.160:0/2806943791 >> 172.21.15.160:0/2806943791 conn(0x7fca940f09a0 msgr2=0x7fca940f14d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:16:09.013 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:09.012+0000 7fca9912b700 1 -- 172.21.15.160:0/2806943791 shutdown_connections 2023-12-25T20:16:09.013 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:09.012+0000 7fca9912b700 1 -- 172.21.15.160:0/2806943791 wait complete. 2023-12-25T20:16:09.013 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:16:10.388 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:16:10.389 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:16:10.468 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:16:09 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:16:08.709256+0000 mgr.smithi050.nbzhun (mgr.14162) 275 : cluster [DBG] pgmap v261: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:16:10.469 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:16:09 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:16:09.012045+0000 mon.smithi050 (mon.0) 320 : audit [DBG] from='client.? 172.21.15.160:0/2806943791' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:16:11.622 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:11.621+0000 7fca9277c700 1 -- 172.21.15.160:0/1055142154 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fca8c0f5410 msgr2=0x7fca8c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:16:11.622 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:11.621+0000 7fca9277c700 1 --2- 172.21.15.160:0/1055142154 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fca8c0f5410 0x7fca8c0f5830 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 rx=0x7fca80009fb0 tx=0x7fca80009b30).stop 2023-12-25T20:16:11.622 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:11.621+0000 7fca9277c700 1 -- 172.21.15.160:0/1055142154 shutdown_connections 2023-12-25T20:16:11.622 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:11.621+0000 7fca9277c700 1 --2- 172.21.15.160:0/1055142154 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fca8c0f5410 0x7fca8c0f5830 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:16:11.622 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:11.621+0000 7fca9277c700 1 -- 172.21.15.160:0/1055142154 >> 172.21.15.160:0/1055142154 conn(0x7fca8c0f0b70 msgr2=0x7fca8c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:16:11.622 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:11.621+0000 7fca9277c700 1 -- 172.21.15.160:0/1055142154 shutdown_connections 2023-12-25T20:16:11.622 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:11.621+0000 7fca9277c700 1 -- 172.21.15.160:0/1055142154 wait complete. 2023-12-25T20:16:11.623 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:11.621+0000 7fca9277c700 1 Processor -- start 2023-12-25T20:16:11.623 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:11.622+0000 7fca9277c700 1 -- start start 2023-12-25T20:16:11.624 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:11.622+0000 7fca9277c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fca8c0f5410 0x7fca8c105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:16:11.625 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:11.622+0000 7fca9277c700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca8c105b20 con 0x7fca8c0f5410 2023-12-25T20:16:11.625 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:11.622+0000 7fca8bfff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fca8c0f5410 0x7fca8c105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:16:11.625 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:11.622+0000 7fca8bfff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fca8c0f5410 0x7fca8c105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:42300/0 (socket says 172.21.15.160:42300) 2023-12-25T20:16:11.625 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:11.622+0000 7fca8bfff700 1 -- 172.21.15.160:0/4129767447 learned_addr learned my addr 172.21.15.160:0/4129767447 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:16:11.625 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:11.622+0000 7fca8bfff700 1 -- 172.21.15.160:0/4129767447 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca80005040 con 0x7fca8c0f5410 2023-12-25T20:16:11.625 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:11.622+0000 7fca8bfff700 1 --2- 172.21.15.160:0/4129767447 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fca8c0f5410 0x7fca8c105630 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7fca80013510 tx=0x7fca8000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:16:11.625 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:11.622+0000 7fca88ff9700 1 -- 172.21.15.160:0/4129767447 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fca80009320 con 0x7fca8c0f5410 2023-12-25T20:16:11.625 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:11.623+0000 7fca88ff9700 1 -- 172.21.15.160:0/4129767447 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fca80013f00 con 0x7fca8c0f5410 2023-12-25T20:16:11.625 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:11.623+0000 7fca9277c700 1 -- 172.21.15.160:0/4129767447 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca8c105d20 con 0x7fca8c0f5410 2023-12-25T20:16:11.625 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:11.623+0000 7fca88ff9700 1 -- 172.21.15.160:0/4129767447 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fca80014d90 con 0x7fca8c0f5410 2023-12-25T20:16:11.625 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:11.623+0000 7fca9277c700 1 -- 172.21.15.160:0/4129767447 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca8c1061b0 con 0x7fca8c0f5410 2023-12-25T20:16:11.625 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:11.623+0000 7fca88ff9700 1 -- 172.21.15.160:0/4129767447 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fca80014420 con 0x7fca8c0f5410 2023-12-25T20:16:11.625 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:11.623+0000 7fca9277c700 1 -- 172.21.15.160:0/4129767447 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca8c045ad0 con 0x7fca8c0f5410 2023-12-25T20:16:11.626 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:11.624+0000 7fca88ff9700 1 --2- 172.21.15.160:0/4129767447 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fca7402ffb0 0x7fca74032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:16:11.626 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:11.624+0000 7fca88ff9700 1 -- 172.21.15.160:0/4129767447 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fca80040fd0 con 0x7fca8c0f5410 2023-12-25T20:16:11.626 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:11.624+0000 7fca8b7fe700 1 --2- 172.21.15.160:0/4129767447 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fca7402ffb0 0x7fca74032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:16:11.626 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:11.624+0000 7fca8b7fe700 1 --2- 172.21.15.160:0/4129767447 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fca7402ffb0 0x7fca74032470 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7fca7c0096a0 tx=0x7fca7c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:16:11.629 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:11.628+0000 7fca88ff9700 1 -- 172.21.15.160:0/4129767447 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fca8006d2b0 con 0x7fca8c0f5410 2023-12-25T20:16:11.833 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:11.833+0000 7fca9277c700 1 -- 172.21.15.160:0/4129767447 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fca8c0fae60 con 0x7fca8c0f5410 2023-12-25T20:16:11.834 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:11.833+0000 7fca88ff9700 1 -- 172.21.15.160:0/4129767447 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fca800095f0 con 0x7fca8c0f5410 2023-12-25T20:16:11.834 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:16:11.834 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:16:11.836 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:11.835+0000 7fca9277c700 1 -- 172.21.15.160:0/4129767447 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fca7402ffb0 msgr2=0x7fca74032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:16:11.836 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:11.835+0000 7fca9277c700 1 --2- 172.21.15.160:0/4129767447 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fca7402ffb0 0x7fca74032470 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7fca7c0096a0 tx=0x7fca7c006b40).stop 2023-12-25T20:16:11.836 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:11.835+0000 7fca9277c700 1 -- 172.21.15.160:0/4129767447 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fca8c0f5410 msgr2=0x7fca8c105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:16:11.836 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:11.835+0000 7fca9277c700 1 --2- 172.21.15.160:0/4129767447 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fca8c0f5410 0x7fca8c105630 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7fca80013510 tx=0x7fca8000b040).stop 2023-12-25T20:16:11.836 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:11.835+0000 7fca9277c700 1 -- 172.21.15.160:0/4129767447 shutdown_connections 2023-12-25T20:16:11.836 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:11.835+0000 7fca9277c700 1 --2- 172.21.15.160:0/4129767447 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fca7402ffb0 0x7fca74032470 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:16:11.836 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:11.835+0000 7fca9277c700 1 --2- 172.21.15.160:0/4129767447 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fca8c0f5410 0x7fca8c105630 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:16:11.836 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:11.835+0000 7fca9277c700 1 -- 172.21.15.160:0/4129767447 >> 172.21.15.160:0/4129767447 conn(0x7fca8c0f0b70 msgr2=0x7fca8c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:16:11.836 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:11.835+0000 7fca9277c700 1 -- 172.21.15.160:0/4129767447 shutdown_connections 2023-12-25T20:16:11.836 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:11.835+0000 7fca9277c700 1 -- 172.21.15.160:0/4129767447 wait complete. 2023-12-25T20:16:11.836 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:16:12.468 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:16:11 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:16:10.709603+0000 mgr.smithi050.nbzhun (mgr.14162) 276 : cluster [DBG] pgmap v262: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:16:12.469 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:16:11 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:16:11.835250+0000 mon.smithi050 (mon.0) 321 : audit [DBG] from='client.? 172.21.15.160:0/4129767447' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:16:13.223 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:16:13.223 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:16:14.386 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:14.384+0000 7f1e9ec15700 1 -- 172.21.15.160:0/2967686849 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1e980f6510 msgr2=0x7f1e980f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:16:14.387 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:14.384+0000 7f1e9ec15700 1 --2- 172.21.15.160:0/2967686849 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1e980f6510 0x7f1e980f6930 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 rx=0x7f1e8c009fb0 tx=0x7f1e8c009b30).stop 2023-12-25T20:16:14.387 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:14.384+0000 7f1e9ec15700 1 -- 172.21.15.160:0/2967686849 shutdown_connections 2023-12-25T20:16:14.387 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:14.384+0000 7f1e9ec15700 1 --2- 172.21.15.160:0/2967686849 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1e980f6510 0x7f1e980f6930 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:16:14.387 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:14.384+0000 7f1e9ec15700 1 -- 172.21.15.160:0/2967686849 >> 172.21.15.160:0/2967686849 conn(0x7f1e980f1c70 msgr2=0x7f1e980f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:16:14.387 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:14.384+0000 7f1e9ec15700 1 -- 172.21.15.160:0/2967686849 shutdown_connections 2023-12-25T20:16:14.387 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:14.384+0000 7f1e9ec15700 1 -- 172.21.15.160:0/2967686849 wait complete. 2023-12-25T20:16:14.387 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:14.385+0000 7f1e9ec15700 1 Processor -- start 2023-12-25T20:16:14.387 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:14.385+0000 7f1e9ec15700 1 -- start start 2023-12-25T20:16:14.387 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:14.385+0000 7f1e9ec15700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1e980f6510 0x7f1e9806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:16:14.387 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:14.385+0000 7f1e9ec15700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e98069e20 con 0x7f1e980f6510 2023-12-25T20:16:14.388 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:14.385+0000 7f1e9c9b1700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1e980f6510 0x7f1e9806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:16:14.388 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:14.385+0000 7f1e9c9b1700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1e980f6510 0x7f1e9806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:42322/0 (socket says 172.21.15.160:42322) 2023-12-25T20:16:14.388 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:14.385+0000 7f1e9c9b1700 1 -- 172.21.15.160:0/288049320 learned_addr learned my addr 172.21.15.160:0/288049320 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:16:14.388 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:14.386+0000 7f1e9c9b1700 1 -- 172.21.15.160:0/288049320 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e8c005040 con 0x7f1e980f6510 2023-12-25T20:16:14.388 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:14.386+0000 7f1e9c9b1700 1 --2- 172.21.15.160:0/288049320 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1e980f6510 0x7f1e9806b9a0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7f1e8c013510 tx=0x7f1e8c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:16:14.388 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:14.386+0000 7f1e957fa700 1 -- 172.21.15.160:0/288049320 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1e8c009320 con 0x7f1e980f6510 2023-12-25T20:16:14.388 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:14.386+0000 7f1e9ec15700 1 -- 172.21.15.160:0/288049320 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e98069fc0 con 0x7f1e980f6510 2023-12-25T20:16:14.388 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:14.386+0000 7f1e957fa700 1 -- 172.21.15.160:0/288049320 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1e8c013f00 con 0x7f1e980f6510 2023-12-25T20:16:14.388 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:14.386+0000 7f1e957fa700 1 -- 172.21.15.160:0/288049320 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1e8c014d90 con 0x7f1e980f6510 2023-12-25T20:16:14.388 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:14.386+0000 7f1e9ec15700 1 -- 172.21.15.160:0/288049320 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e9806a450 con 0x7f1e980f6510 2023-12-25T20:16:14.389 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:14.387+0000 7f1e957fa700 1 -- 172.21.15.160:0/288049320 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1e8c0095f0 con 0x7f1e980f6510 2023-12-25T20:16:14.389 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:14.387+0000 7f1e9ec15700 1 -- 172.21.15.160:0/288049320 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e98059100 con 0x7f1e980f6510 2023-12-25T20:16:14.389 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:14.387+0000 7f1e957fa700 1 --2- 172.21.15.160:0/288049320 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f1e8802ffb0 0x7f1e88032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:16:14.390 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:14.387+0000 7f1e957fa700 1 -- 172.21.15.160:0/288049320 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1e8c0410a0 con 0x7f1e980f6510 2023-12-25T20:16:14.390 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:14.387+0000 7f1e97fff700 1 --2- 172.21.15.160:0/288049320 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f1e8802ffb0 0x7f1e88032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:16:14.390 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:14.387+0000 7f1e97fff700 1 --2- 172.21.15.160:0/288049320 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f1e8802ffb0 0x7f1e88032470 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f1e840096a0 tx=0x7f1e84006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:16:14.392 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:14.391+0000 7f1e957fa700 1 -- 172.21.15.160:0/288049320 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1e8c06d2b0 con 0x7f1e980f6510 2023-12-25T20:16:14.468 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:16:13 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:16:12.710041+0000 mgr.smithi050.nbzhun (mgr.14162) 277 : cluster [DBG] pgmap v263: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:16:14.601 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:14.600+0000 7f1e9ec15700 1 -- 172.21.15.160:0/288049320 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1e980fbf60 con 0x7f1e980f6510 2023-12-25T20:16:14.601 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:14.601+0000 7f1e957fa700 1 -- 172.21.15.160:0/288049320 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1e8c014420 con 0x7f1e980f6510 2023-12-25T20:16:14.602 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:16:14.602 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:16:14.603 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:14.602+0000 7f1e9ec15700 1 -- 172.21.15.160:0/288049320 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f1e8802ffb0 msgr2=0x7f1e88032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:16:14.603 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:14.602+0000 7f1e9ec15700 1 --2- 172.21.15.160:0/288049320 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f1e8802ffb0 0x7f1e88032470 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f1e840096a0 tx=0x7f1e84006b40).stop 2023-12-25T20:16:14.603 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:14.602+0000 7f1e9ec15700 1 -- 172.21.15.160:0/288049320 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1e980f6510 msgr2=0x7f1e9806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:16:14.603 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:14.602+0000 7f1e9ec15700 1 --2- 172.21.15.160:0/288049320 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1e980f6510 0x7f1e9806b9a0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7f1e8c013510 tx=0x7f1e8c00b040).stop 2023-12-25T20:16:14.603 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:14.602+0000 7f1e9ec15700 1 -- 172.21.15.160:0/288049320 shutdown_connections 2023-12-25T20:16:14.603 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:14.602+0000 7f1e9ec15700 1 --2- 172.21.15.160:0/288049320 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f1e8802ffb0 0x7f1e88032470 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:16:14.604 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:14.603+0000 7f1e9ec15700 1 --2- 172.21.15.160:0/288049320 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1e980f6510 0x7f1e9806b9a0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:16:14.604 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:14.603+0000 7f1e9ec15700 1 -- 172.21.15.160:0/288049320 >> 172.21.15.160:0/288049320 conn(0x7f1e980f1c70 msgr2=0x7f1e980f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:16:14.604 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:14.603+0000 7f1e9ec15700 1 -- 172.21.15.160:0/288049320 shutdown_connections 2023-12-25T20:16:14.604 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:14.603+0000 7f1e9ec15700 1 -- 172.21.15.160:0/288049320 wait complete. 2023-12-25T20:16:14.604 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:16:15.468 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:16:14 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:16:14.602714+0000 mon.smithi050 (mon.0) 322 : audit [DBG] from='client.? 172.21.15.160:0/288049320' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:16:15.469 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:16:14 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:16:14.771940+0000 mon.smithi050 (mon.0) 323 : audit [INF] from='mgr.14162 172.21.15.50:0/3178367813' entity='mgr.smithi050.nbzhun' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi050.nbzhun/mirror_snapshot_schedule"}]: dispatch 2023-12-25T20:16:15.469 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:16:14 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:16:14.772253+0000 mon.smithi050 (mon.0) 324 : audit [INF] from='mgr.14162 172.21.15.50:0/3178367813' entity='mgr.smithi050.nbzhun' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi050.nbzhun/trash_purge_schedule"}]: dispatch 2023-12-25T20:16:15.982 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:16:15.983 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:16:16.468 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:16:15 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:16:14.710377+0000 mgr.smithi050.nbzhun (mgr.14162) 278 : cluster [DBG] pgmap v264: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:16:17.181 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:17.180+0000 7f5960c31700 1 -- 172.21.15.160:0/826994978 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f595c0f78e0 msgr2=0x7f595c0f7d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:16:17.182 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:17.180+0000 7f5960c31700 1 --2- 172.21.15.160:0/826994978 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f595c0f78e0 0x7f595c0f7d00 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 rx=0x7f594c009fb0 tx=0x7f594c009b30).stop 2023-12-25T20:16:17.182 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:17.180+0000 7f5960c31700 1 -- 172.21.15.160:0/826994978 shutdown_connections 2023-12-25T20:16:17.182 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:17.180+0000 7f5960c31700 1 --2- 172.21.15.160:0/826994978 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f595c0f78e0 0x7f595c0f7d00 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:16:17.182 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:17.180+0000 7f5960c31700 1 -- 172.21.15.160:0/826994978 >> 172.21.15.160:0/826994978 conn(0x7f595c0f0120 msgr2=0x7f595c0f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:16:17.182 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:17.180+0000 7f5960c31700 1 -- 172.21.15.160:0/826994978 shutdown_connections 2023-12-25T20:16:17.182 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:17.180+0000 7f5960c31700 1 -- 172.21.15.160:0/826994978 wait complete. 2023-12-25T20:16:17.182 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:17.180+0000 7f5960c31700 1 Processor -- start 2023-12-25T20:16:17.182 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:17.181+0000 7f5960c31700 1 -- start start 2023-12-25T20:16:17.182 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:17.181+0000 7f5960c31700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f595c0f78e0 0x7f595c0f71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:16:17.182 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:17.181+0000 7f5960c31700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f595c0f5630 con 0x7f595c0f78e0 2023-12-25T20:16:17.184 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:17.181+0000 7f595b7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f595c0f78e0 0x7f595c0f71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:16:17.184 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:17.181+0000 7f595b7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f595c0f78e0 0x7f595c0f71b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:42336/0 (socket says 172.21.15.160:42336) 2023-12-25T20:16:17.184 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:17.181+0000 7f595b7fe700 1 -- 172.21.15.160:0/2731024712 learned_addr learned my addr 172.21.15.160:0/2731024712 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:16:17.184 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:17.182+0000 7f595b7fe700 1 -- 172.21.15.160:0/2731024712 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f594c005040 con 0x7f595c0f78e0 2023-12-25T20:16:17.184 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:17.182+0000 7f595b7fe700 1 --2- 172.21.15.160:0/2731024712 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f595c0f78e0 0x7f595c0f71b0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7f594c013510 tx=0x7f594c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:16:17.184 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:17.182+0000 7f5943fff700 1 -- 172.21.15.160:0/2731024712 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f594c009320 con 0x7f595c0f78e0 2023-12-25T20:16:17.184 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:17.182+0000 7f5960c31700 1 -- 172.21.15.160:0/2731024712 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f595c0f57d0 con 0x7f595c0f78e0 2023-12-25T20:16:17.184 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:17.182+0000 7f5943fff700 1 -- 172.21.15.160:0/2731024712 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f594c013f00 con 0x7f595c0f78e0 2023-12-25T20:16:17.185 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:17.182+0000 7f5943fff700 1 -- 172.21.15.160:0/2731024712 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f594c014dc0 con 0x7f595c0f78e0 2023-12-25T20:16:17.185 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:17.182+0000 7f5960c31700 1 -- 172.21.15.160:0/2731024712 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f595c0f5c60 con 0x7f595c0f78e0 2023-12-25T20:16:17.185 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:17.183+0000 7f5943fff700 1 -- 172.21.15.160:0/2731024712 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f594c01d3d0 con 0x7f595c0f78e0 2023-12-25T20:16:17.185 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:17.183+0000 7f5960c31700 1 -- 172.21.15.160:0/2731024712 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f595c059100 con 0x7f595c0f78e0 2023-12-25T20:16:17.185 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:17.183+0000 7f5943fff700 1 --2- 172.21.15.160:0/2731024712 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f594402ff60 0x7f5944032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:16:17.185 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:17.183+0000 7f5943fff700 1 -- 172.21.15.160:0/2731024712 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f594c041020 con 0x7f595c0f78e0 2023-12-25T20:16:17.185 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:17.183+0000 7f595affd700 1 --2- 172.21.15.160:0/2731024712 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f594402ff60 0x7f5944032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:16:17.185 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:17.184+0000 7f595affd700 1 --2- 172.21.15.160:0/2731024712 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f594402ff60 0x7f5944032420 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f59500096a0 tx=0x7f5950006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:16:17.189 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:17.188+0000 7f5943fff700 1 -- 172.21.15.160:0/2731024712 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f594c0126f0 con 0x7f595c0f78e0 2023-12-25T20:16:17.393 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:17.392+0000 7f5960c31700 1 -- 172.21.15.160:0/2731024712 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f595c0fbce0 con 0x7f595c0f78e0 2023-12-25T20:16:17.394 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:17.393+0000 7f5943fff700 1 -- 172.21.15.160:0/2731024712 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f594c012330 con 0x7f595c0f78e0 2023-12-25T20:16:17.394 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:16:17.394 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:16:17.395 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:17.395+0000 7f5960c31700 1 -- 172.21.15.160:0/2731024712 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f594402ff60 msgr2=0x7f5944032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:16:17.396 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:17.395+0000 7f5960c31700 1 --2- 172.21.15.160:0/2731024712 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f594402ff60 0x7f5944032420 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f59500096a0 tx=0x7f5950006b40).stop 2023-12-25T20:16:17.396 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:17.395+0000 7f5960c31700 1 -- 172.21.15.160:0/2731024712 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f595c0f78e0 msgr2=0x7f595c0f71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:16:17.396 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:17.395+0000 7f5960c31700 1 --2- 172.21.15.160:0/2731024712 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f595c0f78e0 0x7f595c0f71b0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7f594c013510 tx=0x7f594c00b040).stop 2023-12-25T20:16:17.396 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:17.395+0000 7f5960c31700 1 -- 172.21.15.160:0/2731024712 shutdown_connections 2023-12-25T20:16:17.396 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:17.395+0000 7f5960c31700 1 --2- 172.21.15.160:0/2731024712 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f594402ff60 0x7f5944032420 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:16:17.396 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:17.395+0000 7f5960c31700 1 --2- 172.21.15.160:0/2731024712 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f595c0f78e0 0x7f595c0f71b0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:16:17.396 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:17.395+0000 7f5960c31700 1 -- 172.21.15.160:0/2731024712 >> 172.21.15.160:0/2731024712 conn(0x7f595c0f0120 msgr2=0x7f595c0f0bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:16:17.396 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:17.395+0000 7f5960c31700 1 -- 172.21.15.160:0/2731024712 shutdown_connections 2023-12-25T20:16:17.396 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:17.395+0000 7f5960c31700 1 -- 172.21.15.160:0/2731024712 wait complete. 2023-12-25T20:16:17.397 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:16:18.468 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:16:17 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:16:16.710825+0000 mgr.smithi050.nbzhun (mgr.14162) 279 : cluster [DBG] pgmap v265: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:16:18.469 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:16:17 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:16:17.394885+0000 mon.smithi050 (mon.0) 325 : audit [DBG] from='client.? 172.21.15.160:0/2731024712' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:16:18.800 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:16:18.801 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:16:20.042 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:20.041+0000 7f4a6ad95700 1 -- 172.21.15.160:0/1911556717 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4a640f6510 msgr2=0x7f4a640f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:16:20.042 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:20.041+0000 7f4a6ad95700 1 --2- 172.21.15.160:0/1911556717 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4a640f6510 0x7f4a640f6930 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 rx=0x7f4a58009fb0 tx=0x7f4a58009b30).stop 2023-12-25T20:16:20.042 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:20.041+0000 7f4a6ad95700 1 -- 172.21.15.160:0/1911556717 shutdown_connections 2023-12-25T20:16:20.042 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:20.041+0000 7f4a6ad95700 1 --2- 172.21.15.160:0/1911556717 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4a640f6510 0x7f4a640f6930 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:16:20.042 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:20.041+0000 7f4a6ad95700 1 -- 172.21.15.160:0/1911556717 >> 172.21.15.160:0/1911556717 conn(0x7f4a640f1c70 msgr2=0x7f4a640f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:16:20.042 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:20.041+0000 7f4a6ad95700 1 -- 172.21.15.160:0/1911556717 shutdown_connections 2023-12-25T20:16:20.043 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:20.041+0000 7f4a6ad95700 1 -- 172.21.15.160:0/1911556717 wait complete. 2023-12-25T20:16:20.043 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:20.042+0000 7f4a6ad95700 1 Processor -- start 2023-12-25T20:16:20.043 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:20.042+0000 7f4a6ad95700 1 -- start start 2023-12-25T20:16:20.044 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:20.042+0000 7f4a6ad95700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4a640f6510 0x7f4a6406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:16:20.044 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:20.042+0000 7f4a6ad95700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a64069e20 con 0x7f4a640f6510 2023-12-25T20:16:20.044 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:20.042+0000 7f4a68b31700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4a640f6510 0x7f4a6406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:16:20.044 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:20.042+0000 7f4a68b31700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4a640f6510 0x7f4a6406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:43348/0 (socket says 172.21.15.160:43348) 2023-12-25T20:16:20.044 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:20.042+0000 7f4a68b31700 1 -- 172.21.15.160:0/1618403155 learned_addr learned my addr 172.21.15.160:0/1618403155 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:16:20.044 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:20.042+0000 7f4a68b31700 1 -- 172.21.15.160:0/1618403155 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a58005040 con 0x7f4a640f6510 2023-12-25T20:16:20.044 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:20.043+0000 7f4a68b31700 1 --2- 172.21.15.160:0/1618403155 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4a640f6510 0x7f4a6406b9a0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7f4a58013510 tx=0x7f4a5800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:16:20.044 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:20.043+0000 7f4a617fa700 1 -- 172.21.15.160:0/1618403155 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4a58009320 con 0x7f4a640f6510 2023-12-25T20:16:20.044 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:20.043+0000 7f4a6ad95700 1 -- 172.21.15.160:0/1618403155 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a64069fc0 con 0x7f4a640f6510 2023-12-25T20:16:20.044 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:20.043+0000 7f4a617fa700 1 -- 172.21.15.160:0/1618403155 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4a58013f00 con 0x7f4a640f6510 2023-12-25T20:16:20.044 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:20.043+0000 7f4a617fa700 1 -- 172.21.15.160:0/1618403155 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4a58014d90 con 0x7f4a640f6510 2023-12-25T20:16:20.045 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:20.043+0000 7f4a6ad95700 1 -- 172.21.15.160:0/1618403155 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a6406a450 con 0x7f4a640f6510 2023-12-25T20:16:20.045 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:20.044+0000 7f4a617fa700 1 -- 172.21.15.160:0/1618403155 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4a580263a0 con 0x7f4a640f6510 2023-12-25T20:16:20.045 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:20.044+0000 7f4a6ad95700 1 -- 172.21.15.160:0/1618403155 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a64059100 con 0x7f4a640f6510 2023-12-25T20:16:20.046 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:20.044+0000 7f4a617fa700 1 --2- 172.21.15.160:0/1618403155 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f4a5402ff60 0x7f4a54032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:16:20.046 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:20.044+0000 7f4a617fa700 1 -- 172.21.15.160:0/1618403155 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4a580413a0 con 0x7f4a640f6510 2023-12-25T20:16:20.046 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:20.044+0000 7f4a63fff700 1 --2- 172.21.15.160:0/1618403155 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f4a5402ff60 0x7f4a54032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:16:20.046 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:20.044+0000 7f4a63fff700 1 --2- 172.21.15.160:0/1618403155 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f4a5402ff60 0x7f4a54032420 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f4a500096a0 tx=0x7f4a50006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:16:20.049 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:20.049+0000 7f4a617fa700 1 -- 172.21.15.160:0/1618403155 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4a5801c920 con 0x7f4a640f6510 2023-12-25T20:16:20.254 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:20.254+0000 7f4a6ad95700 1 -- 172.21.15.160:0/1618403155 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4a64045ad0 con 0x7f4a640f6510 2023-12-25T20:16:20.255 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:20.254+0000 7f4a617fa700 1 -- 172.21.15.160:0/1618403155 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4a5801c920 con 0x7f4a640f6510 2023-12-25T20:16:20.255 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:16:20.255 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:16:20.257 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:20.256+0000 7f4a6ad95700 1 -- 172.21.15.160:0/1618403155 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f4a5402ff60 msgr2=0x7f4a54032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:16:20.257 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:20.256+0000 7f4a6ad95700 1 --2- 172.21.15.160:0/1618403155 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f4a5402ff60 0x7f4a54032420 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f4a500096a0 tx=0x7f4a50006b40).stop 2023-12-25T20:16:20.257 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:20.256+0000 7f4a6ad95700 1 -- 172.21.15.160:0/1618403155 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4a640f6510 msgr2=0x7f4a6406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:16:20.257 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:20.256+0000 7f4a6ad95700 1 --2- 172.21.15.160:0/1618403155 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4a640f6510 0x7f4a6406b9a0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7f4a58013510 tx=0x7f4a5800b040).stop 2023-12-25T20:16:20.257 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:20.256+0000 7f4a6ad95700 1 -- 172.21.15.160:0/1618403155 shutdown_connections 2023-12-25T20:16:20.257 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:20.256+0000 7f4a6ad95700 1 --2- 172.21.15.160:0/1618403155 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f4a5402ff60 0x7f4a54032420 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:16:20.257 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:20.256+0000 7f4a6ad95700 1 --2- 172.21.15.160:0/1618403155 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4a640f6510 0x7f4a6406b9a0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:16:20.257 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:20.256+0000 7f4a6ad95700 1 -- 172.21.15.160:0/1618403155 >> 172.21.15.160:0/1618403155 conn(0x7f4a640f1c70 msgr2=0x7f4a640f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:16:20.257 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:20.256+0000 7f4a6ad95700 1 -- 172.21.15.160:0/1618403155 shutdown_connections 2023-12-25T20:16:20.257 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:20.256+0000 7f4a6ad95700 1 -- 172.21.15.160:0/1618403155 wait complete. 2023-12-25T20:16:20.258 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:16:20.468 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:16:19 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:16:18.711150+0000 mgr.smithi050.nbzhun (mgr.14162) 280 : cluster [DBG] pgmap v266: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:16:21.469 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:16:20 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:16:20.256202+0000 mon.smithi050 (mon.0) 326 : audit [DBG] from='client.? 172.21.15.160:0/1618403155' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:16:21.653 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:16:21.653 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:16:22.468 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:16:21 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:16:20.711465+0000 mgr.smithi050.nbzhun (mgr.14162) 281 : cluster [DBG] pgmap v267: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:16:22.864 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:22.863+0000 7fa55ebd8700 1 -- 172.21.15.160:0/3502377207 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa5580f6510 msgr2=0x7fa5580f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:16:22.864 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:22.863+0000 7fa55ebd8700 1 --2- 172.21.15.160:0/3502377207 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa5580f6510 0x7fa5580f6930 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 rx=0x7fa54c009fb0 tx=0x7fa54c009b30).stop 2023-12-25T20:16:22.864 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:22.863+0000 7fa55ebd8700 1 -- 172.21.15.160:0/3502377207 shutdown_connections 2023-12-25T20:16:22.864 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:22.863+0000 7fa55ebd8700 1 --2- 172.21.15.160:0/3502377207 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa5580f6510 0x7fa5580f6930 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:16:22.864 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:22.863+0000 7fa55ebd8700 1 -- 172.21.15.160:0/3502377207 >> 172.21.15.160:0/3502377207 conn(0x7fa5580f1c70 msgr2=0x7fa5580f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:16:22.865 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:22.863+0000 7fa55ebd8700 1 -- 172.21.15.160:0/3502377207 shutdown_connections 2023-12-25T20:16:22.865 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:22.863+0000 7fa55ebd8700 1 -- 172.21.15.160:0/3502377207 wait complete. 2023-12-25T20:16:22.865 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:22.863+0000 7fa55ebd8700 1 Processor -- start 2023-12-25T20:16:22.865 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:22.863+0000 7fa55ebd8700 1 -- start start 2023-12-25T20:16:22.865 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:22.864+0000 7fa55ebd8700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa5581067f0 0x7fa558106c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:16:22.865 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:22.864+0000 7fa55ebd8700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa558107100 con 0x7fa5581067f0 2023-12-25T20:16:22.866 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:22.864+0000 7fa55c974700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa5581067f0 0x7fa558106c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:16:22.867 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:22.864+0000 7fa55c974700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa5581067f0 0x7fa558106c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:43370/0 (socket says 172.21.15.160:43370) 2023-12-25T20:16:22.867 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:22.864+0000 7fa55c974700 1 -- 172.21.15.160:0/1930443944 learned_addr learned my addr 172.21.15.160:0/1930443944 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:16:22.867 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:22.864+0000 7fa55c974700 1 -- 172.21.15.160:0/1930443944 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa54c005040 con 0x7fa5581067f0 2023-12-25T20:16:22.867 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:22.865+0000 7fa55c974700 1 --2- 172.21.15.160:0/1930443944 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa5581067f0 0x7fa558106c10 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7fa54c009f80 tx=0x7fa54c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:16:22.867 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:22.865+0000 7fa5557fa700 1 -- 172.21.15.160:0/1930443944 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa54c0136c0 con 0x7fa5581067f0 2023-12-25T20:16:22.867 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:22.865+0000 7fa55ebd8700 1 -- 172.21.15.160:0/1930443944 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa558107300 con 0x7fa5581067f0 2023-12-25T20:16:22.867 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:22.865+0000 7fa55ebd8700 1 -- 172.21.15.160:0/1930443944 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa558107e50 con 0x7fa5581067f0 2023-12-25T20:16:22.867 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:22.865+0000 7fa5557fa700 1 -- 172.21.15.160:0/1930443944 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa54c0095f0 con 0x7fa5581067f0 2023-12-25T20:16:22.867 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:22.865+0000 7fa5557fa700 1 -- 172.21.15.160:0/1930443944 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa54c01d3a0 con 0x7fa5581067f0 2023-12-25T20:16:22.867 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:22.866+0000 7fa5557fa700 1 -- 172.21.15.160:0/1930443944 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa54c025060 con 0x7fa5581067f0 2023-12-25T20:16:22.867 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:22.866+0000 7fa55ebd8700 1 -- 172.21.15.160:0/1930443944 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa558059100 con 0x7fa5581067f0 2023-12-25T20:16:22.867 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:22.866+0000 7fa5557fa700 1 --2- 172.21.15.160:0/1930443944 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fa54802ff60 0x7fa548032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:16:22.867 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:22.866+0000 7fa5557fa700 1 -- 172.21.15.160:0/1930443944 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa54c040a80 con 0x7fa5581067f0 2023-12-25T20:16:22.867 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:22.866+0000 7fa557fff700 1 --2- 172.21.15.160:0/1930443944 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fa54802ff60 0x7fa548032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:16:22.868 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:22.867+0000 7fa557fff700 1 --2- 172.21.15.160:0/1930443944 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fa54802ff60 0x7fa548032420 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7fa5440096a0 tx=0x7fa544006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:16:22.871 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:22.871+0000 7fa5557fa700 1 -- 172.21.15.160:0/1930443944 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa54c01d500 con 0x7fa5581067f0 2023-12-25T20:16:23.076 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:23.076+0000 7fa55ebd8700 1 -- 172.21.15.160:0/1930443944 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa5581085b0 con 0x7fa5581067f0 2023-12-25T20:16:23.077 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:23.076+0000 7fa5557fa700 1 -- 172.21.15.160:0/1930443944 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa54c012330 con 0x7fa5581067f0 2023-12-25T20:16:23.077 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:16:23.077 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:16:23.079 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:23.078+0000 7fa55ebd8700 1 -- 172.21.15.160:0/1930443944 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fa54802ff60 msgr2=0x7fa548032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:16:23.079 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:23.078+0000 7fa55ebd8700 1 --2- 172.21.15.160:0/1930443944 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fa54802ff60 0x7fa548032420 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7fa5440096a0 tx=0x7fa544006b40).stop 2023-12-25T20:16:23.079 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:23.078+0000 7fa55ebd8700 1 -- 172.21.15.160:0/1930443944 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa5581067f0 msgr2=0x7fa558106c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:16:23.079 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:23.078+0000 7fa55ebd8700 1 --2- 172.21.15.160:0/1930443944 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa5581067f0 0x7fa558106c10 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7fa54c009f80 tx=0x7fa54c009b30).stop 2023-12-25T20:16:23.079 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:23.078+0000 7fa55ebd8700 1 -- 172.21.15.160:0/1930443944 shutdown_connections 2023-12-25T20:16:23.079 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:23.078+0000 7fa55ebd8700 1 --2- 172.21.15.160:0/1930443944 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7fa54802ff60 0x7fa548032420 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:16:23.079 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:23.078+0000 7fa55ebd8700 1 --2- 172.21.15.160:0/1930443944 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa5581067f0 0x7fa558106c10 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:16:23.079 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:23.078+0000 7fa55ebd8700 1 -- 172.21.15.160:0/1930443944 >> 172.21.15.160:0/1930443944 conn(0x7fa5580f1c70 msgr2=0x7fa5580f27b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:16:23.079 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:23.078+0000 7fa55ebd8700 1 -- 172.21.15.160:0/1930443944 shutdown_connections 2023-12-25T20:16:23.080 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:23.078+0000 7fa55ebd8700 1 -- 172.21.15.160:0/1930443944 wait complete. 2023-12-25T20:16:23.080 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:16:24.469 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:16:23 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:16:22.711859+0000 mgr.smithi050.nbzhun (mgr.14162) 282 : cluster [DBG] pgmap v268: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:16:24.469 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:16:23 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:16:23.078208+0000 mon.smithi050 (mon.0) 327 : audit [DBG] from='client.? 172.21.15.160:0/1930443944' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:16:24.475 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T20:16:24.476 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T20:16:25.604 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:25.603+0000 7f149a8e8700 1 -- 172.21.15.160:0/3925338089 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f14940f5140 msgr2=0x7f14940f5560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:16:25.604 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:25.603+0000 7f149a8e8700 1 --2- 172.21.15.160:0/3925338089 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f14940f5140 0x7f14940f5560 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 rx=0x7f1484009fb0 tx=0x7f1484009b30).stop 2023-12-25T20:16:25.605 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:25.603+0000 7f149a8e8700 1 -- 172.21.15.160:0/3925338089 shutdown_connections 2023-12-25T20:16:25.605 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:25.603+0000 7f149a8e8700 1 --2- 172.21.15.160:0/3925338089 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f14940f5140 0x7f14940f5560 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:16:25.605 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:25.603+0000 7f149a8e8700 1 -- 172.21.15.160:0/3925338089 >> 172.21.15.160:0/3925338089 conn(0x7f14940f08c0 msgr2=0x7f14940f2ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:16:25.605 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:25.603+0000 7f149a8e8700 1 -- 172.21.15.160:0/3925338089 shutdown_connections 2023-12-25T20:16:25.605 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:25.603+0000 7f149a8e8700 1 -- 172.21.15.160:0/3925338089 wait complete. 2023-12-25T20:16:25.605 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:25.603+0000 7f149a8e8700 1 Processor -- start 2023-12-25T20:16:25.605 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:25.603+0000 7f149a8e8700 1 -- start start 2023-12-25T20:16:25.605 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:25.604+0000 7f149a8e8700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f14940f5140 0x7f14940fd140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:16:25.605 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:25.604+0000 7f149a8e8700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14940fb5c0 con 0x7f14940f5140 2023-12-25T20:16:25.605 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:25.604+0000 7f14998e6700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f14940f5140 0x7f14940fd140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:16:25.605 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:25.604+0000 7f14998e6700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f14940f5140 0x7f14940fd140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.160:43386/0 (socket says 172.21.15.160:43386) 2023-12-25T20:16:25.605 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:25.604+0000 7f14998e6700 1 -- 172.21.15.160:0/946739051 learned_addr learned my addr 172.21.15.160:0/946739051 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-12-25T20:16:25.607 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:25.604+0000 7f14998e6700 1 -- 172.21.15.160:0/946739051 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1484005040 con 0x7f14940f5140 2023-12-25T20:16:25.607 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:25.604+0000 7f14998e6700 1 --2- 172.21.15.160:0/946739051 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f14940f5140 0x7f14940fd140 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7f1484013510 tx=0x7f148400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:16:25.607 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:25.604+0000 7f148a7fc700 1 -- 172.21.15.160:0/946739051 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1484009320 con 0x7f14940f5140 2023-12-25T20:16:25.607 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:25.604+0000 7f149a8e8700 1 -- 172.21.15.160:0/946739051 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14940fb760 con 0x7f14940f5140 2023-12-25T20:16:25.607 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:25.604+0000 7f148a7fc700 1 -- 172.21.15.160:0/946739051 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1484013f00 con 0x7f14940f5140 2023-12-25T20:16:25.607 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:25.604+0000 7f148a7fc700 1 -- 172.21.15.160:0/946739051 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1484014d90 con 0x7f14940f5140 2023-12-25T20:16:25.607 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:25.605+0000 7f149a8e8700 1 -- 172.21.15.160:0/946739051 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14940fbbf0 con 0x7f14940f5140 2023-12-25T20:16:25.608 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:25.605+0000 7f148a7fc700 1 -- 172.21.15.160:0/946739051 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f14840095f0 con 0x7f14940f5140 2023-12-25T20:16:25.608 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:25.605+0000 7f149a8e8700 1 -- 172.21.15.160:0/946739051 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1494059100 con 0x7f14940f5140 2023-12-25T20:16:25.608 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:25.605+0000 7f148a7fc700 1 --2- 172.21.15.160:0/946739051 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f148002ff60 0x7f1480032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T20:16:25.608 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:25.605+0000 7f148a7fc700 1 -- 172.21.15.160:0/946739051 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1484040fe0 con 0x7f14940f5140 2023-12-25T20:16:25.608 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:25.606+0000 7f14990e5700 1 --2- 172.21.15.160:0/946739051 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f148002ff60 0x7f1480032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T20:16:25.608 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:25.606+0000 7f14990e5700 1 --2- 172.21.15.160:0/946739051 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f148002ff60 0x7f1480032420 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7f14900096a0 tx=0x7f1490006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T20:16:25.611 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:25.610+0000 7f148a7fc700 1 -- 172.21.15.160:0/946739051 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f148401d7f0 con 0x7f14940f5140 2023-12-25T20:16:25.818 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:25.818+0000 7f149a8e8700 1 -- 172.21.15.160:0/946739051 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f14940fac00 con 0x7f14940f5140 2023-12-25T20:16:25.819 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:25.818+0000 7f148a7fc700 1 -- 172.21.15.160:0/946739051 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1484010230 con 0x7f14940f5140 2023-12-25T20:16:25.819 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-25T20:16:25.819 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"02c8f25c-a361-11ee-95a5-87774f69a715","modified":"2023-12-25T20:06:20.367705Z","created":"2023-12-25T20:06:20.367705Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi050","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T20:16:25.821 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:25.820+0000 7f149a8e8700 1 -- 172.21.15.160:0/946739051 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f148002ff60 msgr2=0x7f1480032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:16:25.821 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:25.820+0000 7f149a8e8700 1 --2- 172.21.15.160:0/946739051 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f148002ff60 0x7f1480032420 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7f14900096a0 tx=0x7f1490006b40).stop 2023-12-25T20:16:25.821 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:25.820+0000 7f149a8e8700 1 -- 172.21.15.160:0/946739051 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f14940f5140 msgr2=0x7f14940fd140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T20:16:25.821 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:25.820+0000 7f149a8e8700 1 --2- 172.21.15.160:0/946739051 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f14940f5140 0x7f14940fd140 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7f1484013510 tx=0x7f148400b040).stop 2023-12-25T20:16:25.821 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:25.820+0000 7f149a8e8700 1 -- 172.21.15.160:0/946739051 shutdown_connections 2023-12-25T20:16:25.821 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:25.820+0000 7f149a8e8700 1 --2- 172.21.15.160:0/946739051 >> [v2:172.21.15.50:6800/1043424882,v1:172.21.15.50:6801/1043424882] conn(0x7f148002ff60 0x7f1480032420 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:16:25.821 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:25.820+0000 7f149a8e8700 1 --2- 172.21.15.160:0/946739051 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f14940f5140 0x7f14940fd140 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T20:16:25.821 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:25.820+0000 7f149a8e8700 1 -- 172.21.15.160:0/946739051 >> 172.21.15.160:0/946739051 conn(0x7f14940f08c0 msgr2=0x7f149405fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T20:16:25.821 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:25.820+0000 7f149a8e8700 1 -- 172.21.15.160:0/946739051 shutdown_connections 2023-12-25T20:16:25.821 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-25T20:16:25.820+0000 7f149a8e8700 1 -- 172.21.15.160:0/946739051 wait complete. 2023-12-25T20:16:25.822 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-12-25T20:16:26.240 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_b14a0a106b8cffae0144ff9fb83d9ce1af1a4bd2/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds 2023-12-25T20:16:26.242 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2023-12-25T20:16:26.242 DEBUG:teuthology.orchestra.run.smithi050:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-12-25T20:16:26.268 DEBUG:teuthology.orchestra.run.smithi160:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-12-25T20:16:26.286 INFO:tasks.cephadm:Stopping all daemons... 2023-12-25T20:16:26.286 INFO:tasks.cephadm.mon.smithi050:Stopping mon.smithi050... 2023-12-25T20:16:26.286 DEBUG:teuthology.orchestra.run.smithi050:> sudo systemctl stop ceph-02c8f25c-a361-11ee-95a5-87774f69a715@mon.smithi050 2023-12-25T20:16:26.340 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:16:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: cluster 2023-12-25T20:16:24.712316+0000 mgr.smithi050.nbzhun (mgr.14162) 283 : cluster [DBG] pgmap v269: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T20:16:26.340 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:16:25 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: audit 2023-12-25T20:16:25.820164+0000 mon.smithi050 (mon.0) 328 : audit [DBG] from='client.? 172.21.15.160:0/946739051' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T20:16:26.718 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:16:26 smithi050 systemd[1]: Stopping Ceph mon.smithi050 for 02c8f25c-a361-11ee-95a5-87774f69a715... 2023-12-25T20:16:26.718 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:16:26 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:16:26.417+0000 7f06e2e93700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi050 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2023-12-25T20:16:26.719 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:16:26 smithi050 ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050[111770]: debug 2023-12-25T20:16:26.417+0000 7f06e2e93700 -1 mon.smithi050@0(leader) e1 *** Got Signal Terminated *** 2023-12-25T20:16:26.959 DEBUG:teuthology.orchestra.run.smithi050:> sudo pkill -f 'journalctl -f -n 0 -u ceph-02c8f25c-a361-11ee-95a5-87774f69a715@mon.smithi050.service' 2023-12-25T20:16:27.030 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:16:26 smithi050 podman[117625]: ceph-02c8f25c-a361-11ee-95a5-87774f69a715-mon.smithi050 2023-12-25T20:16:27.030 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:16:26 smithi050 systemd[1]: ceph-02c8f25c-a361-11ee-95a5-87774f69a715@mon.smithi050.service: Succeeded. 2023-12-25T20:16:27.030 INFO:journalctl@ceph.mon.smithi050.smithi050.stdout:Dec 25 20:16:26 smithi050 systemd[1]: Stopped Ceph mon.smithi050 for 02c8f25c-a361-11ee-95a5-87774f69a715. 2023-12-25T20:16:27.065 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-12-25T20:16:27.065 INFO:tasks.cephadm.mon.smithi050:Stopped mon.smithi050 2023-12-25T20:16:27.065 INFO:tasks.cephadm.mon.smithi160:Stopping mon.smithi160... 2023-12-25T20:16:27.065 DEBUG:teuthology.orchestra.run.smithi160:> sudo systemctl stop ceph-02c8f25c-a361-11ee-95a5-87774f69a715@mon.smithi160 2023-12-25T20:16:27.099 INFO:teuthology.orchestra.run.smithi160.stderr:Failed to stop ceph-02c8f25c-a361-11ee-95a5-87774f69a715@mon.smithi160.service: Unit ceph-02c8f25c-a361-11ee-95a5-87774f69a715@mon.smithi160.service not loaded. 2023-12-25T20:16:27.100 DEBUG:teuthology.orchestra.run:got remote process result: 5 2023-12-25T20:16:27.101 ERROR:tasks.cephadm:Failed to stop "ceph.mon.smithi160" Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_b14a0a106b8cffae0144ff9fb83d9ce1af1a4bd2/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_b14a0a106b8cffae0144ff9fb83d9ce1af1a4bd2/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_b14a0a106b8cffae0144ff9fb83d9ce1af1a4bd2/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi160 with status 5: 'sudo systemctl stop ceph-02c8f25c-a361-11ee-95a5-87774f69a715@mon.smithi160' 2023-12-25T20:16:27.102 INFO:tasks.cephadm:Archiving crash dumps... 2023-12-25T20:16:27.103 DEBUG:teuthology.misc:Transferring archived files from smithi050:/var/lib/ceph/02c8f25c-a361-11ee-95a5-87774f69a715/crash to /home/teuthworker/archive/yuriw-2023-12-25_17:28:36-rados-wip-yuri5-testing-2023-12-15-0747-pacific-distro-default-smithi/7500869/remote/ubuntu@smithi050.front.sepia.ceph.com/crash 2023-12-25T20:16:27.104 DEBUG:teuthology.orchestra.run.smithi050:> sudo tar c -f - -C /var/lib/ceph/02c8f25c-a361-11ee-95a5-87774f69a715/crash -- . 2023-12-25T20:16:27.146 DEBUG:teuthology.misc:Transferring archived files from smithi160:/var/lib/ceph/02c8f25c-a361-11ee-95a5-87774f69a715/crash to /home/teuthworker/archive/yuriw-2023-12-25_17:28:36-rados-wip-yuri5-testing-2023-12-15-0747-pacific-distro-default-smithi/7500869/remote/ubuntu@smithi160.front.sepia.ceph.com/crash 2023-12-25T20:16:27.147 DEBUG:teuthology.orchestra.run.smithi160:> sudo tar c -f - -C /var/lib/ceph/02c8f25c-a361-11ee-95a5-87774f69a715/crash -- . 2023-12-25T20:16:27.188 INFO:teuthology.orchestra.run.smithi160.stderr:tar: /var/lib/ceph/02c8f25c-a361-11ee-95a5-87774f69a715/crash: Cannot open: No such file or directory 2023-12-25T20:16:27.188 INFO:teuthology.orchestra.run.smithi160.stderr:tar: Error is not recoverable: exiting now 2023-12-25T20:16:27.189 INFO:tasks.cephadm:Checking cluster log for badness... 2023-12-25T20:16:27.189 DEBUG:teuthology.orchestra.run.smithi050:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/02c8f25c-a361-11ee-95a5-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2023-12-25T20:16:27.224 INFO:teuthology.orchestra.run.smithi050.stderr:grep: /var/log/ceph/02c8f25c-a361-11ee-95a5-87774f69a715/ceph.log: No such file or directory 2023-12-25T20:16:27.226 INFO:tasks.cephadm:Compressing logs... 2023-12-25T20:16:27.227 DEBUG:teuthology.orchestra.run.smithi050:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-12-25T20:16:27.269 DEBUG:teuthology.orchestra.run.smithi160:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-12-25T20:16:27.297 INFO:teuthology.orchestra.run.smithi160.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2023-12-25T20:16:27.297 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2023-12-25T20:16:27.298 INFO:teuthology.orchestra.run.smithi050.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2023-12-25T20:16:27.298 INFO:teuthology.orchestra.run.smithi050.stderr:: No such file or directory 2023-12-25T20:16:27.298 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/02c8f25c-a361-11ee-95a5-87774f69a715/ceph-volume.log 2023-12-25T20:16:27.299 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/02c8f25c-a361-11ee-95a5-87774f69a715/ceph-mon.smithi050.log 2023-12-25T20:16:27.299 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/02c8f25c-a361-11ee-95a5-87774f69a715/ceph-mgr.smithi050.nbzhun.log 2023-12-25T20:16:27.299 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cephadm.log: 96.2% -- replaced with /var/log/ceph/cephadm.log.gz 2023-12-25T20:16:27.299 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/02c8f25c-a361-11ee-95a5-87774f69a715/ceph-volume.log: /var/log/ceph/02c8f25c-a361-11ee-95a5-87774f69a715/ceph-mon.smithi050.log: /var/log/ceph/02c8f25c-a361-11ee-95a5-87774f69a715/ceph-mgr.smithi050.nbzhun.log: 90.8% -- replaced with /var/log/ceph/02c8f25c-a361-11ee-95a5-87774f69a715/ceph-volume.log.gz 2023-12-25T20:16:27.300 INFO:teuthology.orchestra.run.smithi160.stderr: 2023-12-25T20:16:27.300 INFO:teuthology.orchestra.run.smithi160.stderr:real 0m0.014s 2023-12-25T20:16:27.300 INFO:teuthology.orchestra.run.smithi160.stderr:user 0m0.012s 2023-12-25T20:16:27.300 INFO:teuthology.orchestra.run.smithi160.stderr:sys 0m0.010s 2023-12-25T20:16:27.304 INFO:teuthology.orchestra.run.smithi050.stderr: 92.9% -- replaced with /var/log/ceph/cephadm.log.gz 2023-12-25T20:16:27.379 INFO:teuthology.orchestra.run.smithi050.stderr: 92.7% -- replaced with /var/log/ceph/02c8f25c-a361-11ee-95a5-87774f69a715/ceph-mgr.smithi050.nbzhun.log.gz 2023-12-25T20:16:27.450 INFO:teuthology.orchestra.run.smithi050.stderr: 90.5% -- replaced with /var/log/ceph/02c8f25c-a361-11ee-95a5-87774f69a715/ceph-mon.smithi050.log.gz 2023-12-25T20:16:27.452 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-12-25T20:16:27.453 INFO:teuthology.orchestra.run.smithi050.stderr:real 0m0.166s 2023-12-25T20:16:27.453 INFO:teuthology.orchestra.run.smithi050.stderr:user 0m0.247s 2023-12-25T20:16:27.453 INFO:teuthology.orchestra.run.smithi050.stderr:sys 0m0.018s 2023-12-25T20:16:27.453 INFO:tasks.cephadm:Archiving logs... 2023-12-25T20:16:27.453 DEBUG:teuthology.misc:Transferring archived files from smithi050:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-12-25_17:28:36-rados-wip-yuri5-testing-2023-12-15-0747-pacific-distro-default-smithi/7500869/remote/ubuntu@smithi050.front.sepia.ceph.com/log 2023-12-25T20:16:27.455 DEBUG:teuthology.orchestra.run.smithi050:> sudo tar c -f - -C /var/log/ceph -- . 2023-12-25T20:16:27.567 DEBUG:teuthology.misc:Transferring archived files from smithi160:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-12-25_17:28:36-rados-wip-yuri5-testing-2023-12-15-0747-pacific-distro-default-smithi/7500869/remote/ubuntu@smithi160.front.sepia.ceph.com/log 2023-12-25T20:16:27.568 DEBUG:teuthology.orchestra.run.smithi160:> sudo tar c -f - -C /var/log/ceph -- . 2023-12-25T20:16:27.597 INFO:tasks.cephadm:Removing cluster... 2023-12-25T20:16:27.598 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 --force 2023-12-25T20:17:19.599 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 02c8f25c-a361-11ee-95a5-87774f69a715 --force 2023-12-25T20:17:19.909 INFO:tasks.cephadm:Removing cephadm ... 2023-12-25T20:17:19.910 DEBUG:teuthology.orchestra.run.smithi050:> rm -rf /home/ubuntu/cephtest/cephadm 2023-12-25T20:17:19.931 DEBUG:teuthology.orchestra.run.smithi160:> rm -rf /home/ubuntu/cephtest/cephadm 2023-12-25T20:17:19.949 INFO:tasks.cephadm:Teardown complete 2023-12-25T20:17:19.950 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_b14a0a106b8cffae0144ff9fb83d9ce1af1a4bd2/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_b14a0a106b8cffae0144ff9fb83d9ce1af1a4bd2/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_b14a0a106b8cffae0144ff9fb83d9ce1af1a4bd2/qa/tasks/cephadm.py", line 1463, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_b14a0a106b8cffae0144ff9fb83d9ce1af1a4bd2/qa/tasks/cephadm.py", line 1376, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_b14a0a106b8cffae0144ff9fb83d9ce1af1a4bd2/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi160 with status 5: 'sudo systemctl stop ceph-02c8f25c-a361-11ee-95a5-87774f69a715@mon.smithi160' 2023-12-25T20:17:20.071 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=a51700a31cba42bf9a8d06fc7041ba85 Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_b14a0a106b8cffae0144ff9fb83d9ce1af1a4bd2/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_b14a0a106b8cffae0144ff9fb83d9ce1af1a4bd2/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_b14a0a106b8cffae0144ff9fb83d9ce1af1a4bd2/qa/tasks/cephadm.py", line 1463, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_b14a0a106b8cffae0144ff9fb83d9ce1af1a4bd2/qa/tasks/cephadm.py", line 1376, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_b14a0a106b8cffae0144ff9fb83d9ce1af1a4bd2/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi160 with status 5: 'sudo systemctl stop ceph-02c8f25c-a361-11ee-95a5-87774f69a715@mon.smithi160' 2023-12-25T20:17:20.076 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2023-12-25T20:17:20.086 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-12-25T20:17:20.384 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-12-25T20:17:20.384 DEBUG:teuthology.orchestra.run.smithi050:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-12-25T20:17:20.399 DEBUG:teuthology.orchestra.run.smithi160:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-12-25T20:17:20.480 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-12-25T20:17:20.489 INFO:teuthology.task.clock:Checking final clock skew... 2023-12-25T20:17:20.489 DEBUG:teuthology.orchestra.run.smithi050:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-25T20:17:20.492 DEBUG:teuthology.orchestra.run.smithi160:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-25T20:17:20.512 INFO:teuthology.orchestra.run.smithi050.stderr:bash: ntpq: command not found 2023-12-25T20:17:20.513 INFO:teuthology.orchestra.run.smithi160.stderr:bash: ntpq: command not found 2023-12-25T20:17:20.514 INFO:teuthology.orchestra.run.smithi160.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-25T20:17:20.515 INFO:teuthology.orchestra.run.smithi160.stdout:=============================================================================== 2023-12-25T20:17:20.515 INFO:teuthology.orchestra.run.smithi160.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 23 +616us[ +614us] +/- 82ms 2023-12-25T20:17:20.515 INFO:teuthology.orchestra.run.smithi160.stdout:^* hv02.front.sepia.ceph.com 3 6 377 21 +399us[ +397us] +/- 71ms 2023-12-25T20:17:20.515 INFO:teuthology.orchestra.run.smithi160.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 21 -1132us[-1134us] +/- 89ms 2023-12-25T20:17:20.515 INFO:teuthology.orchestra.run.smithi160.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-12-25T20:17:20.517 INFO:teuthology.orchestra.run.smithi050.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-25T20:17:20.518 INFO:teuthology.orchestra.run.smithi050.stdout:=============================================================================== 2023-12-25T20:17:20.518 INFO:teuthology.orchestra.run.smithi050.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 20 +604us[ +604us] +/- 82ms 2023-12-25T20:17:20.518 INFO:teuthology.orchestra.run.smithi050.stdout:^* hv02.front.sepia.ceph.com 3 6 377 21 +400us[ +416us] +/- 71ms 2023-12-25T20:17:20.518 INFO:teuthology.orchestra.run.smithi050.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 20 -1159us[-1159us] +/- 89ms 2023-12-25T20:17:20.518 INFO:teuthology.orchestra.run.smithi050.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-12-25T20:17:20.519 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-12-25T20:17:20.943 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-12-25T20:17:20.944 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-12-25T20:17:20.952 DEBUG:teuthology.orchestra.run.smithi050:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-12-25T20:17:21.052 DEBUG:teuthology.orchestra.run.smithi160:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-12-25T20:17:21.152 DEBUG:teuthology.orchestra.run.smithi050:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-25T20:17:21.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-25T20:17:21.188 DEBUG:teuthology.orchestra.run.smithi160:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-25T20:17:21.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-25T20:17:21.221 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-12-25T20:17:21.231 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-12-25T20:17:21.240 INFO:teuthology.task.internal:Duration was 1407.002779 seconds 2023-12-25T20:17:21.240 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-12-25T20:17:21.250 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-12-25T20:17:21.250 DEBUG:teuthology.orchestra.run.smithi050:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-12-25T20:17:21.253 DEBUG:teuthology.orchestra.run.smithi160:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-12-25T20:17:21.299 INFO:teuthology.orchestra.run.smithi050.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-25T20:17:21.308 INFO:teuthology.orchestra.run.smithi160.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-25T20:17:21.728 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-12-25T20:17:21.728 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi050.front.sepia.ceph.com 2023-12-25T20:17:21.729 DEBUG:teuthology.orchestra.run.smithi050:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-12-25T20:17:21.753 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi160.front.sepia.ceph.com 2023-12-25T20:17:21.753 DEBUG:teuthology.orchestra.run.smithi160:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-12-25T20:17:21.776 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-12-25T20:17:21.776 DEBUG:teuthology.orchestra.run.smithi050:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-12-25T20:17:21.797 DEBUG:teuthology.orchestra.run.smithi160:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-12-25T20:17:21.863 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-12-25T20:17:21.863 DEBUG:teuthology.orchestra.run.smithi050:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-12-25T20:17:21.865 DEBUG:teuthology.orchestra.run.smithi160:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-12-25T20:17:21.952 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-12-25T20:17:21.963 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-12-25T20:17:21.963 DEBUG:teuthology.orchestra.run.smithi050:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-12-25T20:17:21.993 DEBUG:teuthology.orchestra.run.smithi160:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-12-25T20:17:22.021 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-12-25T20:17:22.033 DEBUG:teuthology.orchestra.run.smithi050:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-12-25T20:17:22.036 DEBUG:teuthology.orchestra.run.smithi160:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-12-25T20:17:22.061 INFO:teuthology.orchestra.run.smithi050.stdout:kernel.core_pattern = core 2023-12-25T20:17:22.088 INFO:teuthology.orchestra.run.smithi160.stdout:kernel.core_pattern = core 2023-12-25T20:17:22.141 DEBUG:teuthology.orchestra.run.smithi050:> test -e /home/ubuntu/cephtest/archive/coredump 2023-12-25T20:17:22.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-25T20:17:22.159 DEBUG:teuthology.orchestra.run.smithi160:> test -e /home/ubuntu/cephtest/archive/coredump 2023-12-25T20:17:22.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-25T20:17:22.199 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-12-25T20:17:22.212 INFO:teuthology.task.internal:Transferring archived files... 2023-12-25T20:17:22.212 DEBUG:teuthology.misc:Transferring archived files from smithi050:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-12-25_17:28:36-rados-wip-yuri5-testing-2023-12-15-0747-pacific-distro-default-smithi/7500869/remote/smithi050 2023-12-25T20:17:22.213 DEBUG:teuthology.orchestra.run.smithi050:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-12-25T20:17:22.254 DEBUG:teuthology.misc:Transferring archived files from smithi160:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-12-25_17:28:36-rados-wip-yuri5-testing-2023-12-15-0747-pacific-distro-default-smithi/7500869/remote/smithi160 2023-12-25T20:17:22.255 DEBUG:teuthology.orchestra.run.smithi160:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-12-25T20:17:22.295 INFO:teuthology.task.internal:Removing archive directory... 2023-12-25T20:17:22.295 DEBUG:teuthology.orchestra.run.smithi050:> rm -rf -- /home/ubuntu/cephtest/archive 2023-12-25T20:17:22.298 DEBUG:teuthology.orchestra.run.smithi160:> rm -rf -- /home/ubuntu/cephtest/archive 2023-12-25T20:17:22.346 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-12-25T20:17:22.359 INFO:teuthology.task.internal:Not uploading archives. 2023-12-25T20:17:22.359 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-12-25T20:17:22.369 INFO:teuthology.task.internal:Tidying up after the test... 2023-12-25T20:17:22.369 DEBUG:teuthology.orchestra.run.smithi050:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-12-25T20:17:22.372 DEBUG:teuthology.orchestra.run.smithi160:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-12-25T20:17:22.389 INFO:teuthology.orchestra.run.smithi050.stdout: 525359 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Dec 25 20:17 /home/ubuntu/cephtest 2023-12-25T20:17:22.405 INFO:teuthology.orchestra.run.smithi160.stdout: 525366 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Dec 25 20:17 /home/ubuntu/cephtest 2023-12-25T20:17:22.406 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-12-25T20:17:22.417 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-12-25T20:17:22.437 INFO:teuthology.nuke:Checking targets against current locks 2023-12-25T20:17:22.466 DEBUG:teuthology.nuke:shortname: smithi050 2023-12-25T20:17:22.466 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-25T20:17:22.491 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi050.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-25_17:28:36-rados-wip-yuri5-testing-2023-12-15-0747-pacific-distro-default-smithi/7500869', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-25 19:45:06.249870', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDUisXIgHIseMJVWCQr2pJaStGQPrO9ixhqevO3+QgZvApgL+eVstcZ9+0GeOLVR5I3NJcbS+YnnDVLmLTWmD+U='} 2023-12-25T20:17:22.495 DEBUG:teuthology.nuke:shortname: smithi160 2023-12-25T20:17:22.496 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-25T20:17:22.522 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi160.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-25_17:28:36-rados-wip-yuri5-testing-2023-12-15-0747-pacific-distro-default-smithi/7500869', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-25 19:45:06.248307', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMh/duOsT8rOcQLFOpEZX5ZqrEYscseh2K7m3kgPRUTPE+IcOg6z9u92kDmVc8jMA2wITbTMMP0bfNGrH9JJZyc='} 2023-12-25T20:17:22.549 INFO:teuthology.orchestra.console.smithi050:Power off 2023-12-25T20:17:22.549 DEBUG:teuthology.orchestra.console.smithi050:pexpect command: ipmitool -H smithi050.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-12-25T20:17:22.577 DEBUG:teuthology.orchestra.console.smithi050:power off output: Chassis Power Control: Down/Off 2023-12-25T20:17:22.577 DEBUG:teuthology.orchestra.console.smithi050:pexpect command: ipmitool -H smithi050.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-25T20:17:22.594 INFO:teuthology.orchestra.console.smithi160:Power off 2023-12-25T20:17:22.595 DEBUG:teuthology.orchestra.console.smithi160:pexpect command: ipmitool -H smithi160.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-12-25T20:17:22.614 DEBUG:teuthology.orchestra.console.smithi050:check power output: Chassis Power is on 2023-12-25T20:17:22.622 DEBUG:teuthology.orchestra.console.smithi160:power off output: Chassis Power Control: Down/Off 2023-12-25T20:17:22.622 DEBUG:teuthology.orchestra.console.smithi160:pexpect command: ipmitool -H smithi160.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-25T20:17:22.659 DEBUG:teuthology.orchestra.console.smithi160:check power output: Chassis Power is on 2023-12-25T20:17:26.615 DEBUG:teuthology.orchestra.console.smithi050:pexpect command: ipmitool -H smithi050.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-25T20:17:26.660 DEBUG:teuthology.orchestra.console.smithi160:pexpect command: ipmitool -H smithi160.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-25T20:17:26.727 DEBUG:teuthology.orchestra.console.smithi050:check power output: Chassis Power is on 2023-12-25T20:17:26.773 DEBUG:teuthology.orchestra.console.smithi160:check power output: Chassis Power is on 2023-12-25T20:17:30.729 DEBUG:teuthology.orchestra.console.smithi050:pexpect command: ipmitool -H smithi050.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-25T20:17:30.774 DEBUG:teuthology.orchestra.console.smithi160:pexpect command: ipmitool -H smithi160.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-25T20:17:30.840 DEBUG:teuthology.orchestra.console.smithi050:check power output: Chassis Power is off 2023-12-25T20:17:30.885 DEBUG:teuthology.orchestra.console.smithi160:check power output: Chassis Power is on 2023-12-25T20:17:30.942 INFO:teuthology.orchestra.console.smithi050:Power off completed 2023-12-25T20:17:34.889 DEBUG:teuthology.orchestra.console.smithi160:pexpect command: ipmitool -H smithi160.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-25T20:17:34.999 DEBUG:teuthology.orchestra.console.smithi160:check power output: Chassis Power is off 2023-12-25T20:17:35.100 INFO:teuthology.orchestra.console.smithi160:Power off completed 2023-12-25T20:17:35.208 INFO:teuthology.run:Summary data: description: rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} duration: 1407.0027792453766 failure_reason: 'Command failed on smithi160 with status 5: ''sudo systemctl stop ceph-02c8f25c-a361-11ee-95a5-87774f69a715@mon.smithi160''' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=a51700a31cba42bf9a8d06fc7041ba85 status: fail success: false 2023-12-25T20:17:35.208 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-25T20:17:35.299 INFO:teuthology.run:FAIL